Banded sharing

Trade a little more storage for a closer fit by giving contiguous layer bands their own shared block.

A single shared block per family (G=1) is the smallest possible theta bundle, but it forces every layer in the family to approximate its weight from the same W0. Layers that differ a lot from the family mean then carry a larger low-rank residual.

Banded sharing (G>1) partitions the decoder’s layers into G contiguous bands, each with its own W0 per family. Because a band’s mean is closer to the layers inside it, the per-layer residual shrinks, at the cost of storing G shared blocks instead of one. This is the Relaxed Recursive Transformer sharing scheme.

How layers map to bands

Bands are contiguous and as equal in size as possible; earlier bands take the remainder. For a 26-layer model:

The model’s band_index() method produces this layer-to-band map, and W0 injection uses the correct band’s block for each layer.

Key layout

The band appears only in the W0 key. The per-layer s, U, and V keys are identical regardless of G, because a band is a property of the shared block alone:

G = 1:   W0.q_proj
G > 1:   W0.0.q_proj   W0.1.q_proj   ...

Choosing G

shared_blocks is read from the model config, and WEIGHTLESS_BLOCKS overrides it at runtime for both the decomposition adapter and the model so the two stay in sync. Start at G=1; raise it only if reconstruction error at your target rank is too high. See runtime configuration.