How Strata runs a 125-billion-parameter model on a single 12 GB to 24 GB graphics card

Tier 1: GPU VRAM (12 to 24 GB) holds the always-hot core plus an adaptive expert cache

The ~3.5 GB of dense weights and the 0.8 GB MTP draft layer never leave VRAM. Whatever VRAM remains after the KV cache is packed with the most frequently routed experts (~4,500 of 24,576 experts on a 12 GB RTX 5070 at 4K context; 22,107 MiB of 23,028 MiB total VRAM used on our 24 GB Modal A10 run). Every extra gigabyte of VRAM holds roughly 700 additional experts. Because conversation topics shift, Strata updates this cache on the fly: every 4 decode rounds, it swaps up to 96 newly hot experts into the slots of the least-used ones without pausing generation, raising the VRAM expert hit rate on a 12 GB card from 50% (static profile) to 72%.

Tier 2: Pinned host RAM (32 to 64 GB) holds all 24,576 routed experts

At startup, Strata locks the expert arena into page-locked host RAM (35.9 GiB host RAM used on our Modal A10 run with IQ2_XS, or 23 GB for the half-expert Coder variant that fits on a 32 GB PC). When a machine has less RAM than the full expert arena, Strata's --resident-experts mode copies only the experts that are not already sitting in GPU VRAM (~16 to 18 GB in RAM when paired with a 24 GB GPU), doing zero SSD reads during generation.

Tier 3: NVMe SSD (--ple-io direct) serves the 28.8 GB n-gram table without touching RAM cache

The 51-billion-parameter PLE table is indexed purely by the last three token IDs, so the engine knows which 16 rows (<= 23 KB) a token needs before the layer starts. Strata issues 16 unbuffered O_DIRECT reads one step ahead straight from the NVMe drive (14.4K+ rows/s vs 2.1K rows/s with mmap on a 77.8K-token prompt), leaving system RAM and the operating system's page cache undisturbed for the expert arena.

Mapped host doorbell and a single captured CUDA graph

All 48 layers of a verify window are captured as one CUDA graph per window size. At each layer, the GPU runs the attention or DeltaNet mixer and the router, then writes the 10 selected expert IDs into a tiny 4-byte stream-ordered "doorbell" in mapped pinned host memory. The CPU spins on that doorbell with zero CUDA driver synchronization overhead and immediately partitions the layer's 10 experts.

Dynamic PCIe and CPU co-execution (--pcie-frac auto-refinement)

While the GPU computes the experts that are already cached in VRAM, the missed experts are split between the PCIe bus and the CPU cores. Strata measures the wall-clock speed of the CPU's AVX-512 / AVX2 expert thread pool (R GB/s, such as 32 to 37 µs per expert on an RTX 3060 host) against the probed PCIe link speed (L GB/s) and sets the PCIe DMA share to:

The GPU's DMA copy engine pulls that fraction across PCIe while the CPU computes the remaining missed experts in place in system RAM at the exact same instant, writing the finished vectors back into mapped memory for the GPU to sum. At 4K context with Q2_0 on a 12 GB RTX 5070 and 6-core Ryzen 5 7600, one 48-layer verify window takes 34.3 ms: 14.6 ms on the GPU and 13.4 ms on the CPU, balanced to within a millisecond while the CPU streams expert weights from DDR5 at 41 GB/s. Auto-refining the PCIe share from 0.55 to 0.20 on an RTX 3060 (IQ3_XXS) lifted decode throughput by +30.0% (5/5 paired runs). For short prompts (< 1,024 tokens, STRATA_PREFILL_CPU_SHARE=auto), the CPU also takes rare prompt experts, cutting 512 to 1,000-token prompt time by 22% to 26% on RTX 5070 (Q2_0), 19% to 28% on RTX 3060 (IQ3_XXS), 25% to 35% on Tesla P100, and up to 41% to 49% on PCIe 3.0 x8 setups (KL = 0.0005 to 0.0007).

Verify-window route tail skip (STRATA_ROUTE_TAIL_SKIP=7)

When verifying a window of up to 4 tokens at once, an expert that is missing from VRAM and is ranked at the very tail of the router's top-10 (rank 7, 8, or 9, carrying the smallest router weights) by every token in the window contributes almost nothing to the sum. Strata skips copying or computing those unanimous tail misses altogether (while keeping every VRAM-resident expert and any expert ranked 0..6 by at least one token). Across interleaved benchmarks on fresh engines, route tail skip speeds up decoding by +15.6% on RTX 3060 (IQ3_XXS), +12.9% on RTX A4000 (Q2_0), and +14.6% on Tesla P100 (5/5 paired runs each), while keeping 97.7% to 98.2% top-1 token agreement and a mean teacher-forced KL divergence of 0.0030 to 0.0058. (When 90% or more of all experts already fit in VRAM, Strata automatically disables tail skip.)

Exact speculative decoding with the model's own MTP layer and prompt lookup

Qwen3.8-Flash-Next has a built-in one-layer Multi-Token Prediction (MTP) head that standard GGUF conversions strip out. Strata fetches just those MTP tensors, packs their experts to 2-bit (~0.8 GB in VRAM with a vocabulary subset covering 106,299 English, code, and CJK token IDs), and drafts up to 3 tokens ahead whenever draft confidence is at least 50%. When a reply edits code or quotes earlier context, prompt lookup drafts up to 5 tokens straight from the prompt (+6% to +11% faster on code edits). All 48 layers then verify the last accepted token plus the drafts in one pass, accepting 2.4 to 3.2 tokens per pass on average (3.23 to 3.28 tokens/window at 4K context). That raises 4K decode speed by 1.6x to 1.8x (47–57 tok/s plain to 82–95 tok/s with MTP), with greedy output that is 100% token-for-token identical to non-speculative decoding. For temperature-sampled generation, opt-in coupled Gumbel-max drafting (STRATA_SPEC_COUPLED=1, STRATA_SPEC_GUMBEL=1) keys exponential noise by token ID so the draft and target models agree more often, lifting draft acceptance from 52.8% to 59.9% (+9% output tok/s on Radeon 8060S).

Constant-memory DeltaNet layers plus streaming and K8V4 KV cache out to 262K tokens

In Qwen3.8-Flash-Next, 36 of the 48 layers use Gated DeltaNet, a linear-attention mixer whose recurrent state has a fixed size regardless of prompt length (O(1) per token). Only 12 layers use Qwen Sparse Attention (2 KV heads attending to at most ~2,048 selected positions). Even so, at 262,144 (262K) tokens an 8-bit KV cache in VRAM would shrink the expert cache from ~4,500 slots down to ~1,589 slots on a 12 GB card. Above 64K context, Strata turns on KV streaming (--kv-resident 32768): it keeps the active 32,768 KV positions in VRAM and streams older KV blocks from pinned host RAM (~13.7 KB of RAM per token, or 1.7 GB at 128K context). Freeing that VRAM lifts 262K cached experts on an RTX 5070 (Q2_0) from 1,589 to 3,872 slots and raises decode speed from 50.9 to 62.6 tokens/s with identical attention values. Pairing that with hybrid K8V4 KV quantization (--kv k8v4, keeping keys in 8-bit and values in Hadamard-rotated 4-bit) cuts KV memory by another 23%, boosting 198K-context decode on an RTX 3090 (Coder) from 85 to 99 tokens/s (+780 expert slots on an 8 GB RTX 2060 SUPER at 128K) with the same needle-in-a-haystack retrieval score.

More experts inside the same 12 GB to 24 GB VRAM budget

AgentionAI's upstream-0.2.4-agention fork of Strata adds native support for Gyro TQ's eight trellis-coded tensor types (TQK4 at 1.125 bits per weight through TQP7 at 3.625 bits per weight). Compressing expert or dense layers with block-Hadamard rotation and trellis lookup shrinks each expert's byte footprint so the GPU expert cache holds a larger share of the 24,576 experts in VRAM, while our damped AWQ pre-scaling (α ∈ [0.25, 0.30]) before Hadamard preserves outlier-channel precision (-83.1% to -85.6% validation MSE versus uncalibrated Hadamard).

Single-card economics for 35B to 125B models

A 35B-A3B Mixture-of-Experts model compressed to TQ2_T (2.125 bpw) drops from 19.76 GiB (Q4_K_M) to 10.49 GiB, fitting its entire expert arena plus a 6 GiB KV cache inside a single 24 GB card ($0.80/hr Modal L4 vs $1.95/hr 48 GB L40S). And when you step up to the 125B Qwen3.8-Flash-Next checkpoint (512 experts per layer plus a 51B PLE table), Strata's three-tier engine runs it at 77 to 88.8 tokens/s on a single 24 GB A10 or 78.6 to 94 tokens/s on a 12 GB desktop RTX 5070.

Engine license vs model weight licenses

Strata itself is open source under the MIT License. The models it runs carry their own licenses: Qwen3.8-Flash-Next (and derivative GGUF quantizations such as ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF) is governed by the Qwen Community License 1.0, whose Clause 2 requires a separate commercial agreement from Qwen if you offer the model or its outputs to third parties in a hosted "Model as a Service" or "AI Work Assistant" product. Our Modal A10 run used only public and synthetic prompts for internal benchmarking.

Single-user speed vs parallel batching without --batch-mtp

Strata is tuned first for one fast interactive session (88.8 tok/s short chat and 80.5 tok/s agent turns on one A10). In our concurrency sweep with plain "parallel": 8 slots (without --batch-mtp), total throughput across 8 simultaneous requests rose by only +30% (77.4 tok/s at concurrency 1 to 101.0 tok/s total / 14.3 tok/s per stream at concurrency 8) because plain parallel slots lose per-stream MTP speculative speedups.

System RAM size and speed still matter

To keep all 24,576 experts pinned in RAM without SSD paging during generation, a desktop needs 64 GB of RAM (48 GB for Q2_0 / IQ2_XS, or 32 GB for the 256-expert Coder variant or when paired with a 24 GB GPU in --resident-experts mode). Because the CPU streams missed Q2_0 experts from RAM at ~41 GB/s, dual-channel DDR5 is roughly twice as fast on the CPU half of a round as DDR4.