Gyro TQ: How uncalibrated sub-2-bit weight quantization works

1. Why standard low-bit quantization breaks on outliers

Large language model weights are mostly small numbers clustered near zero, punctuated by steep outlier channels 10 to 20 times larger than their neighbors. When a standard block quantizer assigns one scale factor to a block, the largest outlier stretches the rounding grid across the entire group. Gyro TQ solves this geometrically: before quantizing a single bit, it rotates every 128-weight block so outlier spikes spread evenly across all 128 coordinates while preserving exact dot products.

Existing sub-3-bit formats try to protect outlier channels by running a calibration pass over hundreds of sample prompts (an importance matrix, or imatrix) or by solving per-layer Hessians. That calibration step takes hours on large models, ties the quantized file to the domain of the calibration text, and still leaves bulky sub-block scale headers or multi-megabyte lookup tables that stall GPU memory bandwidth during token generation. Because the Hadamard rotation in Figure 1 is orthogonal and self-inverse, Gyro TQ eliminates calibration data completely.

2. The three-step Gyro TQ pipeline

Every Gyro TQ tensor row is partitioned into independent blocks of QK_GYRO = 128 contiguous weights. Encoding and decoding each 128-weight block uses three deterministic stages:

Step 1: Block-Hadamard rotation (T = H_128 D_s)

Each 128-weight block is multiplied by a deterministic pseudorandom sign vector s of +1 and -1 entries, followed by the normalized 128-by-128 Walsh-Hadamard transform H_128. The transform turns spiky coordinates into a tight bell-shaped distribution while preserving exact dot products: (T w) · (T x) = w · x.

Step 2: 15-bit tail-biting circular trellis

Rather than rounding each weight independently, an encoder searches a 32,768-state trellis over a circular bitstream where the ending state wraps back to match the starting state. At decode time, the kernel needs no search at all. It simply slides a 16-bit window by K bits per step and masks to 15 bits (s_t & 0x7FFF).

Step 3: 32-bit golden-ratio hash into a shared 8 KiB codebook

Each 15-bit trellis state is multiplied modulo 2³² by the golden-ratio constant 0x9E3779B1. The top 22 bits of that product split into two 11-bit indices (idx0 and idx1, each 0..2047) that address two 2-D weight pairs inside a single 8,192-byte lookup table shared across all nine Gyro TQ formats.

3. The nine Gyro TQ formats (GGUF types 144 to 152)

All nine formats share the same 128-weight block size, the same 2-byte FP16 scale header d, the same 15-bit state mask, and the same 8 KiB codebook. They differ only in how many new bits K advance the sliding window per trellis step and whether each step emits a 4-weight quad (32 steps per block, using both idx0 and idx1) or a 2-weight pair (64 steps per block, using idx0 only):

Provenance: Format table from SPEC.md §1.2. Effective bits per weight equals (block_bytes * 8) / 128, which includes the 2-byte FP16 block scale d (0.125 bpw overhead). Notice that TQ2_T and TQP4 both occupy 34 bytes per block (2.125 bpw): TQ2_T uses byte-aligned 8-bit steps over 32 quads for maximum decode speed, whereas TQP4 uses 4-bit steps over 64 pairs for finer trellis granularity.

4. Validation: WASM reference kernels and rotation ablations

Two experimental checks verify the Gyro TQ design: bit-exact conformance of the reference WebAssembly kernels against the specification, and an ablation on a 0.8B tool-calling router showing why Hadamard rotation is essential when quantizing weights without a calibration set.

4.1 Bit-exact WebAssembly kernel verification

The reference WebAssembly implementation compiles the C99 reference decoder and fast Walsh-Hadamard transform into a 20,856-byte binary (gyro_tq.wasm) that runs identically in Node.js, Cloudflare Workers, and browsers. All five verification suites pass with zero discrepancies:

Provenance: Measured by running node wasm/test_wasm.mjs against wasm/gyro_tq.wasm (compiled from ref/gyro_tq_ref.c and wasm/gyro_tq_wasm.c).

4.2 Why rotation matters: AWQ + Hadamard ablation on a 0.8B tool router

To isolate the effect of block-Hadamard rotation on sub-4-bit weights, we evaluated a fine-tuned Qwen3.5-0.8B router across a 111-turn tool-routing benchmark (49 tool-call turns, 62 plain-response turns, 24 candidate tools). Naive 4-bit round-to-nearest (RTN) quantization without rotation degrades overall routing accuracy by 6.3 percentage points and introduces 7 false tool-to-none hand-offs. Adding a 128-wide block-Hadamard rotation with activation-aware scaling recovers full FP16 overall accuracy (95.5%, matching 106/111 turns) while cutting weight memory by 3.37x:

Provenance: Measured on NVIDIA L4 GPU in benchmarks/qwen08b_quant_eval/results.json. Notice the last row: when uniform scalar rounding is pushed down to 3 bits (W3-g128), even AWQ + Hadamard drops to 73.0% because independent scalar grids lose too much volume efficiency below 3 bits. That breakdown is why Gyro TQ replaces independent scalar rounding with a 15-bit tail-biting trellis over 2-D codebook pairs.

4.3 Damped AWQ + Hadamard calibration: solving post-rotation variance flattening

When optional calibration activations are available, feeding a standard diagonal importance vector (imatrix) into the trellis after a 128-wide block-Hadamard rotation fails. Because every entry of H_128 has magnitude 1/sqrt(128), the rotated diagonal second moment inside each 128-weight block collapses to the block average. On a heavy-tailed 32 x 768 LLM layer with Pareto channel scales and 16 outlier channels up to 15x, block-Hadamard rotation flattens within-block importance variation from a coefficient of variation of CV = 8.12 down to 0.054.

Gyro TQ resolves this by applying a damped AWQ channel pre-scale s_j = (E[x_j^2])^alpha with exponent alpha in [0.25, 0.30] on the calibration activations X_cal before the block-Hadamard rotation, paired with a per-row calibration-loss selector (min(cal_loss)). Across TQK6 through TQP5, damped AWQ + Hadamard beats both uncalibrated Hadamard and unrotated imatrix:

Provenance: Measured on disjoint 512-sample calibration and 512-sample validation splits in step3_alpha_sweep.json (SHA256SUMS 68a87b1b...). At TQP5 (alpha = 0.25), validation activation MSE drops to 2.19, which is 76.5% lower than unrotated imatrix, 85.6% lower than uncalibrated Hadamard, and 93.1% lower than uncalibrated unrotated quantization.

4.4 Vector store compression on Flickr8k-1k (EmbeddingGemma 2)

Gyro TQ also compresses dense embedding indexes for local retrieval. On the 1,000-image Flickr8k-1k benchmark encoded with EmbeddingGemma 2 (768d float32 baseline at 3,072 B/vec, and Matryoshka 256d), pairing a 128-wide block-Hadamard rotation with TQ2_T shrinks vectors by 15.1x (to 204 B/vec at 768d) and 45.2x (to 68 B/vec at 256d) while preserving or exceeding float32 retrieval accuracy:

Provenance: Measured in gyro-tq-local-embed/2026-10-25/results.json (SHA256SUMS 56e8480e...) with disjoint COCO-10k calibration anchors. At 768d + Hadamard-128 + TQ2_T (hadamard128_uncal), cosine similarity reaches 0.9634 and dot-product MSE drops to 0.00045 (5.1x lower than unrotated 0.00231). Code inspection of src/tq_ref.c:362 explains why rotation is essential before importance weighting on embeddings: without Hadamard, w_importance inflates the Viterbi scale-search anchor sigma = sqrt(swxx / sw) by 2x to 3x at TQK4 and underflows non-peak coordinates, whereas Hadamard-128 equalizes block energy and cuts dot-product MSE by 50% to 80%.

4.5 Sub-millisecond 768x6 decision router head in WebAssembly

For lightweight local routing, we quantized a frozen 768x6 decision router head (recipe 20cd2cf5, 18,432 B in float32) and benchmarked full WebAssembly dequantization plus the 6x768 dot product in V8. A complete router evaluation takes 13.84 µs (0.014 ms) per query inside the standalone 9,271-byte WebAssembly module:

Provenance: Measured in V8 WebAssembly and C11 reference harnesses in gyro-tq-local-embed/2026-10-25/results.json. Full WASM dequantization plus the 6x768 dot product runs in 13.84 µs (0.014 ms) per query with zero additional tool-to-none errors across all three quantized formats.

5. Real-world memory cliffs: what fits in consumer RAM and VRAM

Token generation in local LLMs is memory-bandwidth bound: every generated token must stream all active model weights from RAM or VRAM into the compute units. Crossing a hardware memory boundary forces weights to spill over PCIe or SSD swap and drops throughput by an order of magnitude. By compressing weights to 1.125 to 2.125 bpw with uncalibrated model files, Gyro TQ shifts frontier model classes down by one full hardware tier:

Provenance: Weight footprints are analytical estimates from bits_per_weight * parameter_count / 8 using exact Gyro TQ block sizes from SPEC.md §1.2 (excluding unquantized embedding/norm tensors and runtime KV cache).

6. What is in the open release, and what is not there yet

We check new formats so we can separate what works today from what still needs engineering:

7. Words used on this page

Block-Hadamard rotation (H_128 D_s)

A fast orthogonal mixing transform applied to 128 weights at a time using only additions, subtractions, and a 1/sqrt(128) scale. It spreads any single outlier spike evenly across all 128 positions without changing dot products.

BPW (bits per weight)

The average number of storage bits consumed by each neural network parameter, including block scale headers. For a 128-weight Gyro TQ block of B bytes, bpw = (B * 8) / 128.

Calibration dataset / imatrix

A sample text corpus run through a model during conventional quantization to measure which weights matter most. Gyro TQ requires no calibration dataset and produces uncalibrated model files directly from the weights.

Codebook (CODEBOOK[4096])

The fixed 8,192-byte lookup table of 2,048 two-dimensional coordinate pairs (4,096 FP16 numbers) shared by every Gyro TQ format. Its 8 KiB footprint stays hot in L1 cache or GPU shared memory.

Multiplicative hash (0x9E3779B1)

A single 32-bit unsigned integer multiplication derived from the golden ratio that scatters sequential 15-bit trellis states evenly across pairs of 11-bit codebook indices (idx0 and idx1).

Tail-biting circular trellis

A sliding-window bitstream where the final step wraps around to the first byte of the block so the starting and ending 15-bit states match, avoiding boundary bit waste.