Runtime configuration

Environment variables and config fields that control rank, banded sharing, and the reconstruction path.

Weightless behavior is set two ways: config fields read from the model’s config.json (with backward-compatible defaults so unmodified Gemma 3 repos load), and environment variables that override them at runtime. Environment variables win, and they are applied to both the decomposition adapter and the model so the two stay consistent.

Environment variables

Variable	Default	Effect
`WEIGHTLESS_RANK`	config `lora_rank` (32)	LoRA rank `r` of the `U, V` delta. Higher rank fits the weight more closely but ships more bits.
`WEIGHTLESS_BLOCKS`	config `shared_blocks` (1)	Number of contiguous layer bands `G`, each with its own `W0`. See banded sharing.
`WEIGHTLESS_GEMM`	unset (auto)	Force the reconstruction path: `1`/`true` for GEMM, `0`/`false` for the custom op. Unset is device-aware: GEMM on non-Metal accelerators (NVIDIA GPUs) at any rank, and the rank crossover (GEMM when `rank >= 8`) on Metal.
`WEIGHTLESS_QKV_FUSE`	`1`	Fuse the Q/K/V projections in the GEMM path. Set `0` to disable. No effect on the custom-op path.
`WEIGHTLESS_FUSED_MLP`	`1`	Fuse gate + up + activation + down into one `generated_mlp` custom op. Custom-op path only. Set `0` to disable.
`WEIGHTLESS_FUSE_LORA`	`0`	Fold the LoRA `U, V` delta into a single effective-weight matmul. GEMM path only.
`WEIGHTLESS_PRECOMPUTE`	`0`	Precompute the effective weight inside the graph. GEMM path only; experimental tuning knob.
`WEIGHTLESS_PRECOMPUTE_OUTSIDE`	`0`	Precompute the effective weight at load time, materializing the full per-layer weight. GEMM path only; experimental tuning knob.
`WEIGHTLESS_MEF_CACHE_DIR`	`~/.cache/weightless_fw/compiled_graphs/`	Directory for cached compiled graphs. Set to an empty string to disable caching.

For example, serve at rank 16 with the GEMM path forced on:

WEIGHTLESS_RANK=16 WEIGHTLESS_GEMM=1 \
  pixi run python run_max.py serve \
    --model google/gemma-3-1b-it \
    --custom-architectures weightless_fw \
    --port 3100

Config fields

These are read from config.json by WeightlessGemma3Config and inherited alongside every standard Gemma 3 field:

Field	Default	Description
`lora_rank`	`32`	Rank of the per-(layer, family) LoRA delta.
`tie_families`	`"all"`	`"all"` makes every projection weightless; `"attn"` restricts it to `q/k/v/o` and leaves the MLP dense.
`shared_blocks`	`1`	Number of layer bands `G`.
`tied_projections`	all seven families	Families reconstructed by the kernel; set to the attention families when `tie_families="attn"`.