Throughput

Decode throughput, TTFT, and TPOT for the weightless model on an NVIDIA B200, measured per LoRA rank and reconstruction path.

Throughput measures what in-kernel reconstruction costs at serving time. Each number is decode throughput (tokens per second), time to first token (TTFT), or time per output token (TPOT), reported against the device, the LoRA rank, and the reconstruction path that produced it. The custom-op path and the GEMM path are math-identical, so any difference here is pure performance.

Measured on NVIDIA B200

All four numbers below are single-run max generate measurements on one B200, generating 128 tokens from a fixed prompt. Reproduce with the script committed at tests/ or directly:

WEIGHTLESS_RANK=<r> WEIGHTLESS_GEMM=<0|1> pixi run python run_max.py generate \
  --model google/gemma-3-1b-it --custom-architectures weightless_fw \
  --prompt "<prompt>" --max-new-tokens 128

Decode throughput

The full matrix, with latency:

The GEMM path wins on NVIDIA GPUs at every rank

On the B200, the GEMM path is faster than the custom op at both ranks: 48% faster at rank 4 (213.9 versus 144.6) and more than 10x faster at rank 32 (166.1 versus 15.0). The custom op collapses as rank grows because it reconstructs each output element with a per-block tree reduction, a launch-bound pattern that leaves the B200’s matrix units idle. The GEMM path instead lowers to MAX’s stock device GEMM kernels, which are built for exactly this hardware.

Device-aware auto-select

The original auto-select heuristic picked the custom op below rank 8, a crossover measured on Apple Metal. On the B200 that left roughly a third of decode throughput on the table at rank 4, so the default is now device-aware: on a non-Metal accelerator the unset default is the GEMM path at any rank, while Metal keeps its rank crossover.

With WEIGHTLESS_GEMM unset on the B200, the auto-select picks the GEMM path (the runtime’s accelerator API reports the name cuda) and delivers GEMM-path throughput at both ranks:

No manual WEIGHTLESS_GEMM=1 is needed on NVIDIA GPUs anymore. To force the custom op for comparison, set WEIGHTLESS_GEMM=0.

Versus vanilla MAX

The comparison that matters is against stock, non-weightless Gemma 3. Measured on the same B200 at rank 4, median of 3 runs of 128 tokens each:

Weightless decodes about 2x slower than vanilla dense on the B200. This is expected: for each projection the weightless GEMM path runs the full W0 matmul (the same size as the dense weight) plus two LoRA matmuls. At batch size 1 decode is launch-bound, so the extra dispatches, not the extra arithmetic, dominate.

So on this hardware and model there is currently no measured weightless configuration that is both coherent and compressed: G=26 is coherent but uncompressed, and G=1 at rank 4 is compressed but incoherent. Finding a (G, rank) that preserves quality while sharing W0 is the open question, and it must be answered on the quality axis, not throughput alone. Native dense at 458 tok/s is the bar any usable weightless config has to justify itself against.