llama-modchip v2.4.0 Fits a 284B MoE Model Into 64GB RAM Without a GPU
llama-modchip v2.4.0 shrinks DeepSeek-V4-Flash 284B to 56.56 GiB and runs it on a 16-core Zen 5 CPU with 44.36 tok/s prefill and 9.06–9.95 tok/s lossless decode, no GPU required.
Memory Reduction and Quantization
The standard UD-IQ1_S build of DeepSeek-V4-Flash occupies 76.87 GiB on disk and exceeds the usable memory of a 64GB Linux machine, causing page-cache thrashing or OOM kills. llama-modchip v2.4.0 transcodes the checkpoint down to 56.56 GiB, an effective 1.71 bits per weight. The 284.33B-parameter model (13.17B active) then locks into RAM together with a 256K-token KV cache, with peak VmRSS reported at 57.61 GiB. The 256 routed MoE experts are mainly IQ1_S at 1.5625 bpw, while MLA attention projections use a Fractal-64 format at 3.75 bpw. Embeddings, dense layers, and shared experts stay in Q4_K/Q5_K/F32. By default, all six routed experts per layer are evaluated without pruning; logit cosine similarity against the unpatched runtime on the same checkpoint is exactly 1.00000.
CPU Kernel Patches
The project replaces single-column quantized matrix-vector loops during prompt prefill with a multi-column tiling schedule across ZMM AVX-512BW registers and L1D, L2, and L3 caches. On a 16-core AMD Zen 5 CPU (Ryzen 9 9955HX), prefill throughput increases from 25.0 to 44.36 tokens per second, a 77.1% gain. The unpatched UD-IQ1_S checkpoint incurs OOM or swap during prefill on 64GB and decodes at 5.91 tokens per second. Register-stationary MQA8 FlashAttention is used, and non-temporal loads (_MM_HINT_NTA) stream cold MoE expert weights so they do not evict hot KV cache lines from L3. Lossless Top-6 decoding reaches 9.06–9.95 tokens per second; optional Top-5 and Top-4 configurations raise decode speed to 10.74 tokens per second with small perplexity deltas.
Speculative Drafting and Context Restore
A 1.64 MB Zobrist n-gram drafter named Stockfish-PVS v2 works with /dev/shm KV slot snapshots. Warm restore times for 40-tool Claude Code prompts range from 13.1 to 32.6 ms. With speculative decoding, throughput for the 284B model climbs to 11.5–14.7 tokens per second. The patch also includes an O(1) speculative rollback fix for deepseek4 recurrent/MLA states.
Other Models and Hardware
For smaller devices, the 30.2B/3.0B-active Nemotron-3.5-30B-A3B is compressed to 14.35 GiB at 3.75 bpw and runs at 25.0–60.4 tokens per second on a 16-core Zen 5 CPU. A Vulkan hybrid setup using a Ryzen 9 7945HX3D and Radeon RX 9070 XT 16GB achieves 80.8 tokens per second, while a Threadripper PRO 7985WX reaches 111.6 tokens per second. The Fractal-64 and MXFP4 variants pass the project’s 100-point quality gate. The project supports x86_64 (AVX-512/AVX2) and ARM64 (NEON/SVE/Apple Metal) processors. A Windows batch file detects CPU cores, RAM, and GPU VRAM to compute a hybrid --n-cpu-moe split automatically.