Skip to main content
Local AI

LLM Tokens/Sec Benchmarks 2026: RTX 4090 vs 3090, 7B-70B Q4 (llama.cpp)

llama.cpp tokens/sec benchmarks for 2026: RTX 4090 vs 3090 across 7B, 13B, 34B and 70B at Q4 — throughput and which GPU you actually need to run local LLMs at usable speed.

2026 updateNewer models (Qwen 3, Llama 4 Scout, DeepSeek R1, Gemma 3) run at speeds close to their size class here — a 32B runs like the 32B numbers below, an MoE like Llama 4 Scout runs faster than its total size suggests. To check your exact GPU against the current model list, use the LLM Hardware Checker.

How fast will Llama 3.1 70B run on a used RTX 3090? Is an M3 Max actually worth the money? These are the questions answered by real tokens-per-second benchmarks — the only metric that tells you whether your local LLM will feel instant or painful. Below are the numbers we’ve measured, plus a breakdown of what they mean in practice.

Quick answer: RTX 4090 leads at 135 tok/s on 7B models, RTX 3090 hits 95 tok/s, RTX 3060 12GB does 45 tok/s. On 70B models (Q2 quant), 4090 reaches 18 tok/s and 3090 hits 10 tok/s. Apple M3 Max 64GB runs 70B but only at 5 tok/s. CPU-only is painful — under 10 tok/s on anything.
Tokens per second benchmark bar chart comparing consumer GPUs for 7B to 70B LLM models
Real-world llama.cpp throughput — Q4_K_M quantization, single batch, warm model.

Detailed tokens-per-second table

What this table covers — and what it doesn’t. The figures below are for a single GPU running Q4_K_M (Q2_K for 70B on 24 GB cards) under llama.cpp at 2048 context, single-batch. Speeds vary with your CPU, RAM speed, driver and context length, so treat them as indicative rather than exact.

Not covered here: multi-GPU / dual-card setups (scaling depends heavily on layer split and PCIe bandwidth), and model sizes between the columns shown (e.g. 24B and 32B) — a 24B behaves close to the 34B column, a 32B slightly slower than it. For a size or quantization not listed, the VRAM calculator will tell you whether it fits your card, and the quantization guide explains which file to download.
GPU7B13B34B70B (Q2)
RTX 4090 24GB135 tok/s78 tok/s42 tok/s18 tok/s
RTX 3090 24GB95 tok/s55 tok/s28 tok/s10 tok/s
RTX 4070 Super 12GB75 tok/s40 tok/sOOMOOM
RTX 4060 Ti 16GB55 tok/s30 tok/s8 tok/sOOM
RTX 3060 12GB45 tok/s22 tok/sOOMOOM
Apple M3 Max 64GB40 tok/s22 tok/s11 tok/s5 tok/s
CPU only (DDR5)6-10 tok/s3-5 tok/s1-2 tok/s<1 tok/s

Get one of these for your homelab

Affiliate links — using these helps support the testing and benchmarks on this site at no extra cost to you.

Try a GPU Droplet
DigitalOcean — pay-by-hour H100/A100 $200 credit →

Don’t have one of these? Use the LLM hardware calculator to see what your current PC can already run.

What these numbers mean in practice

🟢 40+ tok/s — feels instant

Faster than you can read. Great for coding autocomplete, quick chat, and streaming responses. This is the bar you want for daily use.

🔵 20–40 tok/s — comfortable

Responses keep pace with your reading. Perfect for deep-thinking work where quality matters more than raw speed.

🟡 10–20 tok/s — usable

You notice the wait but it’s fine for one-off queries. Typical for running 70B on a single 24GB GPU.

🔴 Under 10 tok/s — painful

Fine for batch processing. Torturous for interactive chat. Consider a different GPU or a smaller model.

Benchmark methodology

Related guides

Last updated: 2026-04-22.

Also from us: abujiggy — smart travel from Dubai & the UAE  ·  FPL360 — Fantasy Premier League tools