Local AI

Local LLM Tokens/Sec Benchmarks 2026: RTX 4090 vs 3090 vs Mac (7B–70B)

Real tokens-per-second benchmarks for local LLMs across RTX 4090, 3090, and Apple Silicon — 7B to 70B at Q4/Q8. See which GPU actually gives you usable speed in 2026.

2026 updateNewer models (Qwen 3, Llama 4 Scout, DeepSeek R1, Gemma 3) run at speeds close to their size class here — a 32B runs like the 32B numbers below, an MoE like Llama 4 Scout runs faster than its total size suggests. To check your exact GPU against the current model list, use the LLM Hardware Checker.

How fast will Llama 3.1 70B run on a used RTX 3090? Is an M3 Max actually worth the money? These are the questions answered by real tokens-per-second benchmarks — the only metric that tells you whether your local LLM will feel instant or painful. Below are the numbers we’ve measured, plus a breakdown of what they mean in practice.

Quick answer: RTX 4090 leads at 135 tok/s on 7B models, RTX 3090 hits 95 tok/s, RTX 3060 12GB does 45 tok/s. On 70B models (Q2 quant), 4090 reaches 18 tok/s and 3090 hits 10 tok/s. Apple M3 Max 64GB runs 70B but only at 5 tok/s. CPU-only is painful — under 10 tok/s on anything.

Tokens per second benchmark bar chart comparing consumer GPUs for 7B to 70B LLM models — Real-world llama.cpp throughput — Q4_K_M quantization, single batch, warm model.

Detailed tokens-per-second table

GPU	7B	13B	34B	70B (Q2)
RTX 4090 24GB	135 tok/s	78 tok/s	42 tok/s	18 tok/s
RTX 3090 24GB	95 tok/s	55 tok/s	28 tok/s	10 tok/s
RTX 4070 Super 12GB	75 tok/s	40 tok/s	OOM	OOM
RTX 4060 Ti 16GB	55 tok/s	30 tok/s	8 tok/s	OOM
RTX 3060 12GB	45 tok/s	22 tok/s	OOM	OOM
Apple M3 Max 64GB	40 tok/s	22 tok/s	11 tok/s	5 tok/s
CPU only (DDR5)	6-10 tok/s	3-5 tok/s	1-2 tok/s	<1 tok/s

Get one of these for your homelab

Affiliate links — using these helps support the testing and benchmarks on this site at no extra cost to you.

RTX 4090 24GB
~135 t/s on 7B Check price → RTX 3090 24GB
~95 t/s on 7B, used market Check price → RTX 4070 Super 12GB
~75 t/s on 7B, sweet spot Check price → RTX 4060 Ti 16GB
Best $/GB-VRAM in 2026 Check price → Mac Studio M3 Max
64GB+ for 70B models Check price → Try a GPU Droplet
DigitalOcean — pay-by-hour H100/A100 $200 credit →

Don’t have one of these? Use the LLM hardware calculator to see what your current PC can already run.

What these numbers mean in practice

🟢 40+ tok/s — feels instant

Faster than you can read. Great for coding autocomplete, quick chat, and streaming responses. This is the bar you want for daily use.

🔵 20–40 tok/s — comfortable

Responses keep pace with your reading. Perfect for deep-thinking work where quality matters more than raw speed.

🟡 10–20 tok/s — usable

You notice the wait but it’s fine for one-off queries. Typical for running 70B on a single 24GB GPU.

🔴 Under 10 tok/s — painful

Fine for batch processing. Torturous for interactive chat. Consider a different GPU or a smaller model.

Benchmark methodology

Backend: llama.cpp b3520 (CUDA/Metal/Vulkan builds as appropriate)
Quantization: Q4_K_M for main numbers, Q2_K for 70B on 24GB cards
Context length: 2048 tokens
Single batch: numbers reflect solo use, not server-style batching
Warm model: first request excluded (cold start adds 2-5 seconds)
OS: Ubuntu 24.04 on Linux rigs, macOS 14 for Apple Silicon

Related guides

Last updated: 2026-04-22.