What weightless inference costs

The four metric families that tell you whether weightless decomposition is worth it, how each is measured, and which need a GPU.

Weightless inference makes a trade: it ships and holds far less weight data, and pays for it by reconstructing each projection inside a kernel at every step. Whether that trade is worth it comes down to four questions, each with a metric you can measure directly from this repo.

Compression: how much smaller is the theta bundle than the original weights?
Reconstruction fidelity: how closely does diag(s)·W0 + U·Vᵀ match the original weight?
Output quality: does the reconstructed model still behave like the original?
Serving performance: what does in-kernel reconstruction cost in throughput and latency?

The first two are analytic: they need no GPU and are deterministic, so the numbers on the compression page reproduce exactly. The last two need a running model, so they’re measured with max generate and max serve on real hardware, and the numbers carry the device they were measured on.

Compression

The headline metric is the ratio of the theta bundle (W0 + s + U + V across every layer and family) to the original projection weights it replaces. Lower is better. It’s set by the LoRA rank and the number of shared blocks G: rank raises fidelity and size together, so the metric is always reported against the rank that produced it. See compression for the measured rank sweep.

Reconstruction fidelity

Before any token is generated, you can ask how good the decomposition is on its own terms. relative_frobenius_error(original, reconstructed) reports ‖W − (diag(s)·W0 + U·Vᵀ)‖_F / ‖W‖_F for a single projection. It’s the upstream predictor of quality: if a layer reconstructs poorly, its outputs drift. Because it’s analytic, it’s the cheapest signal for choosing a rank before spending GPU time. See core.decompose for the helpers.

Output quality

Fidelity per weight doesn’t guarantee model behavior, because small per-layer errors compound over an autoregressive decode. Two end-to-end metrics measure the model as a whole against native Gemma 3:

Greedy-decode match runs the same prompts through both models at temperature 0 and reports the fraction of matching output. It’s a direct behavioral check.
Perplexity delta compares each model’s logprob for the same tokens and reports the relative gap. It’s a finer-grained signal than exact-match, sensitive to distribution shift the greedy token hides.

See quality for the measured deltas.

Serving performance

In-kernel reconstruction replaces one stored-weight matmul with a small set of operations that rebuild the weight on the fly, so the cost shows up in three standard serving numbers:

Decode throughput (tokens per second) during generation.
Time to first token (TTFT), which folds in prefill.
Time per output token (TPOT), the steady-state decode cost.

The interesting variable is the reconstruction path. The custom-op path wins at low rank and the GEMM path wins at higher rank, so throughput is always reported alongside the rank and the path that produced it. See throughput for the measured numbers per device.

Reproducing the numbers

Every metric on these pages comes from the test harness in tests/:

# Analytic metrics (no GPU, deterministic):
pixi run pytest tests/test_benchmark.py -v -k "ThetaBundleSize" -s
pixi run pytest tests/test_decompose.py -v -s

# Serve-based metrics (need a GPU and an HF token with the Gemma 3 license):
pixi run pytest tests/test_benchmark.py -v -m serve -k Throughput -s
pixi run pytest tests/test_quality.py -v -m serve -s

Each metrics page names the device and configuration its numbers were measured on, so a number is never separated from the conditions that produced it.