LLM Tokens per Second Calculator: Any GPU, Any Model (2026)

Choose a model and your machine. Below the calculator are my own measured llama.cpp runs and the setups people ask about most, worked out.

How close is the estimate? See my measured runs below.

Measured on my own hardware

I ran the same three model files on my own machines with llama.cpp: Q4_K_M, 8,192 tokens of context, 256 tokens generated three times at temperature 0, median shown. Next to each result is what the calculator above says for the same machine.

• Ten of the twelve results fall inside the calculator’s range. That is a check on four machines, not proof for every card.
• The two misses: on the 8 GB RTX 2060 Super I forced every layer of gemma-4-12b onto the card, while the calculator assumes a few layers spill into system RAM, so the real result beat the estimate. On the Xeon, Qwen3.5-9B ran faster than estimated; the calculator is cautious here.
• Be fair to the CPU rows: I lowered the calculator’s CPU factor after my first runs on the NUC and the Xeon, so those rows are not an independent check. The Mac and RTX 2060 Super rows are.
• Builds used: llama.cpp b11567 on the Mac (12 October 2026); b11561 from the official Docker image on the NUC; the official b12064 Windows builds on the RTX 2060 Super (CUDA 12.4) and the Xeon (CPU). The NUC, 2060 Super and Xeon were run on 3 November 2026. The NUC and 2060 Super came within a few percent of my first runs in October; the Xeon was 25 to 40% faster than in October, on a newer build.

Common setups, worked out

The combinations people search for most, from the same calculator. Every figure here is an estimate. Use “Try your settings” to load a row into the calculator and change it.

Why two cards are not twice as fast: llama.cpp puts some layers on one card and the rest on the other, and runs them one after the other. A pair of 24 GB cards lets a 70B model sit fully in video memory, which is the big win over spilling into system RAM, but each token still waits for one card at a time. The calculator therefore uses one card’s memory bandwidth for dual-card setups.

What these numbers mean in practice

How the estimates are worked out

• Everything is an estimate except the “Measured on my own hardware” table.
• The formula: making each token means reading the model’s weights from memory once, so speed is roughly memory bandwidth divided by the file size, times an efficiency factor (0.6 for graphics cards, 0.7 for Apple silicon, 0.4 for CPUs). The result is shown as a range of plus or minus 25%.
• Models that do not fit: the part on the card and the part in system RAM are timed separately and added up. That is why a 70B model on one 24 GB card falls to about 1 tok/s.
• Mixture-of-experts models (names like 30B-A3B) are estimated from the active part only, because only those weights are read for each token.
• File sizes are real: they come from the actual GGUF files on Hugging Face and are refreshed every day.
• Your speed will differ with drivers, the llama.cpp version, context length, and how many layers really end up on the GPU.

Questions people ask

How many tokens per second does an RTX 4090 do?

At Q4_K_M with 8,192 tokens of context, the calculator estimates about 90–155 tok/s for an 8B model (Llama 3.1 8B) and 25–40 tok/s for a 32B model (Qwen2.5 32B). Llama 3.3 70B at Q4_K_M does not fit in 24 GB, so it drops to about 1 tok/s. These are estimates, not measured runs.

Is an RTX 3090 fast enough for a 70B model?

Not on its own at Q4_K_M. The 42.5 GB file does not fit in 24 GB, so about half the layers run from system RAM and the estimate falls to about 1 tok/s. Two RTX 3090s hold the whole file and are estimated at 10–17 tok/s.

Do two GPUs double tokens per second?

No. With llama.cpp’s normal layer split, the cards take turns, so a second card mainly lets a bigger model fit in video memory. The speed gain comes from not spilling into system RAM, not from adding the two cards’ bandwidth together.

RTX 4090 or Apple M3 Max for local LLMs?

For models that fit in 24 GB, the 4090 is faster because it has more than twice the memory bandwidth. For a 70B model at Q4_K_M, an M3 Max with 64 GB holds the whole file and is estimated at 5–8 tok/s, against about 1 tok/s on a single 4090 that has to spill into system RAM.

How many tokens per second is fast enough for chat?

Above about 10 tok/s text arrives at reading speed, above 20 tok/s it feels instant, and below 5 tok/s it feels slow. Agents and long outputs, where the model is called again and again, need 30 tok/s or more to feel responsive.

Why is my LLM slow even with a good GPU?

Usually because part of the model is running from system RAM. Check video memory use while it generates, and check the GPU layers setting (--n-gpu-layers in llama.cpp, the GPU offload slider in LM Studio). A larger quant also slows things down: a Q8_0 file is about twice the size of Q4_K_M, so expect roughly half the speed. A very long context adds to the memory needed and can push layers off the card.

Can I run an LLM on CPU only?

Yes. On my Intel NUC (i7-8559U, DDR4, no graphics card used) llama.cpp measured 5.6 tok/s on Qwen3.5-4B and 3.1 tok/s on Qwen3.5-9B at Q4_K_M. A desktop with DDR5 is estimated at 5–8 tok/s on an 8B model and about 3–4 tok/s on 14B. Workable for occasional questions, slow for daily chat.

Related guides

• ✅ Will It Run? Check any model against your GPU or Mac
• 🧮 Calculate the VRAM your model needs
• 💾 VRAM requirements by model size
• 💰 Best GPU for local LLMs by budget
• 🧮 Quantization explained

Last updated: 3 November 2026. Rebuilt as a calculator, with my own measured runs added.