Quality

The (G, rank) coherence frontier for weightless Gemma 3 on B200 — which configurations still generate usable text, and at what footprint.

Compression and throughput only matter if the model still works, so quality is the gating axis. It has two levels: whether the model generates coherent text at all (a hard pass or fail), and, for configs that pass, how close it stays to native Gemma 3 (greedy-decode match and perplexity). The variable that decides both is the pair (G, rank): G (shared_blocks) sets how many layers share one W0, and rank sets the size of the per-layer LoRA delta. Smaller G and lower rank mean more compression but a coarser approximation.

The coherence frontier

To find which configurations are usable, weightless Gemma 3 1B generated from a fixed prompt (The capital of France is) across a grid of G and rank on the B200, classifying each output as coherent or garbage. Footprint is the analytic theta size as a fraction of the dense projection weights; a value above 100% means no compression (a distinct W0 per layer plus LoRA overhead).

Verdict: no coherent, compressed operating point on this model

Only G=26 produces usable text, and every configuration that shares a W0 across layers (G of 13 or fewer) generates garbage at every rank tested, up to rank 64. This is a hard negative result: the analytic decomposition has no operating point that is both coherent and compressed for Gemma 3 1B.

The reason is structural, not a bug. G=26 on a 26-layer model puts each layer in its own band, so its W0 is that layer’s own weight, the scale is near 1, and the LoRA delta is near zero. That is effectively passthrough: it stores the original weight per layer (footprint above 100%) and therefore stays coherent. Compression only appears when layers share a W0 (G < 26), and there the per-layer residual weight - s·W0 has high effective rank, so a rank-4 to rank-64 delta cannot reconstruct it. Sharing is exactly what breaks the model, and more rank does not buy it back within the range that keeps the bundle small.

Throughput is roughly flat across the grid (136 to 208 tok/s) because the graph is the same regardless of G and rank; coherence, not speed, is what changes. So for this model the weightless trade collapses: the only coherent config gives no compression and still runs about 2x slower than vanilla MAX.

Why sharing fails: reconstruction error

The coherence result is explained directly by how far the decomposition lands from the real weights. Measured on the actual Gemma 3 1B weights (relative Frobenius error of diag(s)·W0 + U·Vᵀ against each true weight, averaged over all layers and families):

At G=1 the reconstruction is 95% off at rank 4 and still 45% off at rank 256, so the “reconstructed” weight keeps almost none of the original and generation is garbage. Three things drive it:

• The band-mean W0 is a poor shared block. Gemma’s per-layer weights do not cluster around their family mean, so the residual weight - s·W0 that the LoRA delta must absorb carries most of the weight’s energy. Sharing itself is the problem: even mild sharing (G=13, 50% footprint) is 66% off at rank 4.
• The residual is genuinely high-rank, dominated by the large MLP matrices (up_proj is always the worst family, 4096x1152). Rank 256 is full rank for the attention projections yet still leaves G=1 at 45%, because 256 is tiny against 4096.
• G=26 is exactly zero because each layer’s W0 is its own weight, s is 1, and the residual vanishes. It reconstructs perfectly precisely because it is passthrough, not compression.

This is not a bug in the decomposition, and it is not fixable by raising rank (the table already runs to rank 256) or by choosing a slightly better mean, since the gap to a usable error (roughly under 10%) is 8x to 20x. A usable weightless Gemma 3 1B would need a fundamentally better shared representation or a training step that fits the shared blocks and deltas to a distillation objective, rather than the one-shot analytic decomposition this framework does today. Reproduce the error table with pixi run python over the real safetensors and core.decompose.relative_frobenius_error.

Reproducing

for G in 1 2 13 26; do for r in 4 32 64; do
  WEIGHTLESS_BLOCKS=$G WEIGHTLESS_RANK=$r pixi run python run_max.py generate \
    --model google/gemma-3-1b-it --custom-architectures weightless_fw \
    --prompt "The capital of France is" --max-new-tokens 16
done; done

For the closer parity metrics on a coherent config:

pixi run pytest tests/test_quality.py -v -m serve -s

Parity at a coherent config

• Greedy-decode match runs prompts through the weightless and native models at temperature 0 and reports the fraction of matching output.
• Perplexity delta compares each model’s logprob for the same tokens and reports the relative gap.

Both are only meaningful for a (G, rank) that passes the coherence gate above, and both are reported against the configuration that produced them, because a higher rank or larger G closes the gap at the cost of a larger theta bundle.