Mustafa 
A 744-billion-parameter model on a machine with 25 GB of RAM sounds like a typo. It isn’t. Colibri, an open-source engine written in pure C by Vincenzo Fornaro, runs GLM-5.2 — a frontier-class Mixture-of-Experts model — on consumer hardware by streaming most of the model from disk instead of holding it in memory.
It has been all over my feeds, usually framed as “the end of expensive GPUs.” That framing is wrong, but the project is genuinely important for anyone running AI at home. Here is what it actually does, the real numbers, and what it changes about how I’d spec a homelab AI box.
Why a 744B model can fit at all
The trick is the architecture of the model, not magic in the engine. GLM-5.2 is a Mixture-of-Experts (MoE) model. Instead of one giant block of weights that every token passes through, it has a smaller “dense” core plus tens of thousands of small specialist sub-networks called experts. For each token, a router picks a handful of experts and ignores the rest.
So although the model has 744B parameters in total, only around 40B are active for any single token. The question stops being “can I fit 744B parameters in memory?” and becomes “can I get the right 40B into memory fast enough?”
Colibri answers that by treating storage, RAM and VRAM as one memory hierarchy:
- The dense core stays in RAM. About 17B parameters, quantized to int4, take roughly 9.9 GB.
- The routed experts live on disk. Around 19,000 experts of about 19 MB each, about 372 GB of weights in total.
- Experts are loaded on demand into an LRU cache, with prefetching and “pinning” of experts the model keeps coming back to. The OS page cache does a lot of the heavy lifting too.
The key design promise, in the project’s own words, is “no SLA on speed, and a hard guarantee on semantics.” Less fast memory makes it slower, but it never swaps in a smaller model or skips experts to keep up. You get the real model’s output, just later.
The honest numbers
This is where most coverage gets vague, so here are the throughput figures the project publishes for GLM-5.2 at int4:
| Hardware | Decode speed |
|---|---|
| 25 GB RAM dev box, cold cache | 0.05–0.1 tokens/sec |
| Single RTX 5070 Ti | ~1.07 tokens/sec |
| 128 GB RAM desktop, CPU only, warm cache | ~1.8 tokens/sec |
| 6× RTX 5090, everything resident | 5.8–6.8 tokens/sec |
For context, cloud H100 deployments of models like this are reported at somewhere in the 30–50 tokens/sec range. At 0.1 tokens/sec, a 500-word answer takes over an hour and a half. At ~1.8 tokens/sec it takes about six minutes, which is slow but workable for an overnight batch job.
I haven’t benchmarked the full model on my own machines yet, so treat those as the project’s figures, not mine. When I do, I’ll publish the results here with the exact config.
What the requirements really are
The README lists 16 GB of RAM as the minimum and 24 GB as comfortable, with no GPU required, on Linux, macOS or Windows. On paper, that includes my 16 GB Mac Mini. In practice, the spec that matters most isn’t in that line:
- Disk speed is the new VRAM. Every token triggers random reads of ~19 MB expert files. On a fast NVMe that’s tolerable; on a SATA SSD it hurts; on spinning disks or a NAS share it’s unusable. My Synology is a great place to store the weights and a terrible place to run them from.
- Spare RAM is cache. Anything above the ~10 GB dense core becomes expert cache. That’s a big part of why the 128 GB CPU-only desktop runs an order of magnitude faster than the 25 GB box: fewer trips to disk.
- You need ~372 GB free on the fast drive, plus room for the download.
- SSD wear is less scary than it sounds. Expert traffic is reads, which barely wear flash. The writes are mostly the KV cache.
Does this hurt NVIDIA?
The hot take is that if a 744B model runs on a laptop, nobody needs datacenter GPUs. Look at the table again: the fastest configuration uses six RTX 5090s and still trails a single cloud accelerator. GPUs didn’t stop mattering. What changed is where the bottleneck sits for people who can tolerate slow output.
The shift I actually expect is in how homelabbers spend money:
- Before: “How much VRAM can I afford?” decided which models you could run at all.
- With MoE streaming: almost any model runs. VRAM, RAM and NVMe bandwidth decide how fast. A used workstation with 128 GB of RAM and a Gen4 NVMe starts to look like a better frontier-model box than a single mid-range GPU.
That’s good news for storage and memory vendors, neutral-to-good for NVIDIA (people still want the fast path), and very good news for anyone who cares about privacy. A frontier model that never sends a prompt off your network is now a hardware question, not a policy one.
Who should try it today
Colibri is not a daily driver for interactive coding or chat — use a local 8B–32B model or a cloud API for that. It is worth trying if you want to:
- Run overnight batch jobs on sensitive documents with a frontier-class model, entirely offline.
- Understand how MoE models really work, from a codebase small enough to read in an afternoon.
- Future-proof a hardware purchase. If you’re buying a homelab AI box this year, weight RAM capacity and NVMe speed more heavily than you would have a year ago.
If you’re still working out what your current hardware can run, start with my LLM hardware checker. It covers the dense models most people run day to day. The build is two commands, straight from the repo:
git clone https://github.com/JustVugg/colibri && cd colibri/c
./setup.sh # needs gcc or clang with OpenMPThen point it at the converted weights on your fastest drive:
COLI_MODEL=/nvme/glm52_i4 ./coli chatWhatever you think of the hype, it’s the clearest sign yet that “frontier model” and “runs at home” are no longer mutually exclusive. They’re just a speed trade-off.