NVIDIA’s new small model is a 6.18 GB download. On my 8 GB graphics card it used 3.3 GB of video memory and ran at 73 tokens a second.
The short answer
Yes, comfortably. With llama.cpp, the Q4_K_M build of Nemotron 3.5 Bolt E4B ran at 73.5 tokens a second on an RTX 2060 Super, 28.9 on a Mac mini M4 and 5.0 on a 2018 Intel NUC with no GPU. The surprise is memory: the file is 6.18 GB, but llama.cpp put only 3.3 GB of it on the graphics card. The rest, two large lookup tables, stayed in system RAM. File size is a poor guide to VRAM for this model.
What you’ll learn
- Real generation and prompt speeds on three ordinary machines, two weeks after release
- How it compares with Qwen3.5 4B and 9B on the same hardware and test
- Why a 6 GB model needed only about 3 GB of VRAM, and how to check that on your own card
- What I did not test
Where this comes from
these are my own measurements from 7 November 2026, using the same llama.cpp test I used to check my local LLM calculator. Raw results and logs are kept with the calculator’s other benchmark data. Model details come from NVIDIA’s Hugging Face page.
What Nemotron 3.5 Bolt is
NVIDIA published Nemotron 3.5 Bolt E4B on Hugging Face on 23 October 2026. It is a small model post-trained from Google’s Gemma 4 E4B, aimed at local voice assistants, tool use and embedded devices. NVIDIA describes it as 8B total parameters with 4B “effective”, with up to 64K tokens of context, under the OpenMDW 1.1 licence. The repository name ends in “4GB”, which tells you the memory budget NVIDIA had in mind.
The official download is a single Q4_K_M GGUF file of 6,180,446,208 bytes. llama.cpp reports it as 7.52B parameters, 5.74 GiB, using the gemma4 architecture with 42 layers.
That raised the obvious question for anyone with a modest GPU: how does a 6 GB file fit a “4GB” model?
How I tested
The same test I use for every model in my tokens-per-second benchmarks: llama.cpp’s llama-server, 8,192 tokens of context, a 32-token warm-up, then three generations of 256 tokens at temperature 0. I quote the median. Each machine used the same llama.cpp build as its earlier runs, so the comparison with Qwen3.5 is like for like.
Test setup
- RTX 2060 Super
- 8 GB card in a Core i9-10900K desktop, Windows 11, llama.cpp b12064 (CUDA 12.4),
-ngl 99 - Mac mini M4
- macOS, llama.cpp b11567 (Metal), all layers on the GPU
- Intel NUC8i7BEH
- Core i7-8559U, CPU only, llama.cpp b11561 (official Docker image)
- Model
- Nemotron-3.5-Bolt-E4B-Q4_K_M.gguf, SHA-256 checked against Hugging Face
- Test
- 8,192 context, 3 × 256 tokens, temperature 0, median
Before trusting any speed, I checked that the model gave a sensible answer on each machine. A model that loads but outputs nonsense can still post a great tokens-per-second figure.
The results
Generation speed
tokens per second, median of three runs, higher is betterThe three runs on each machine were within 2% of each other (72.7 to 74.3 on the 2060 Super, 28.81 to 28.88 on the Mac, 5.03 to 5.04 on the NUC).
Prompt processing, which decides how long you wait before the first word on a long input, was 606 tokens a second on the 2060 Super, 214 on the Mac and 30 on the NUC.
How it compares with Qwen3.5 on the same machines
Here is the same test with two models I had already benchmarked. Note the file sizes: Nemotron’s download is bigger than Qwen3.5 9B’s.
| Machine | Qwen3.5 4B (2.74 GB) | Nemotron 3.5 Bolt (6.18 GB) | Qwen3.5 9B (5.68 GB) |
|---|---|---|---|
| RTX 2060 Super | 88.1 | 73.5 | 54.5 |
| Mac mini M4 | 29.7 | 28.9 | 17.7 |
| Intel NUC, CPU only | 5.63 | 5.03 | 3.07 |
Nemotron runs at roughly the speed of a 4B model (83% to 97% of Qwen3.5 4B), not a 9B one, even though its file is larger than the 9B’s. That matches NVIDIA’s “4B effective” description. I did not compare answer quality; NVIDIA’s card has its own benchmark table against Gemma 4 E4B and Qwen3.5 4B.
Where the other 3 GB went
On the 2060 Super, I re-ran the server with verbose logging (-v). Every layer was offloaded (“offloaded 43/43 layers to GPU”), yet the model buffers were split like this:
| Where | Size | What it holds |
|---|---|---|
| System RAM (memory-mapped) | 2,856 MiB | per_layer_token_embd, a per-layer embedding table |
| System RAM (memory-mapped) | 680 MiB | token_embd, the normal token embedding table |
| GPU | 3,023 MiB | the 42 transformer layers |
| GPU | 168 MiB | KV cache at 8,192 context |
| GPU | 136 MiB | compute buffer |
llama.cpp’s own estimate before loading was: “projected to use 3327 MiB of device memory vs. 7152 MiB of free device memory”.
So about 3.5 GB of the file never touches the graphics card. The log line that explains it:
tensor 'token_embd.weight' (q8_0) (and 1 others) cannot be used with preferred buffer type CUDA_Host, using CPU instead
The two embedding tables are lookups: for each token, the model reads the rows for that token rather than multiplying through the whole table. llama.cpp keeps them in system memory by default, with no extra flags from me. Gemma 4’s “E” models carry an unusually large per-layer embedding table (almost 3 GB here), which is why the gap between file size and VRAM is so big for this family.
The split is not partial offload. The 42 layers were all on the card; only the embedding tables stayed behind.
3,327 MiB on the card for a 5,894 MiB file, at 8K context, including the KV cache.
The 3.5 GB of embeddings have to live somewhere, so budget that much free RAM alongside the card.
Within 3% to 17% of Qwen3.5 4B on all three machines, despite a file 2.3 times the size.
Why the NUC wasn’t slowed by the big file
On a CPU-only machine, speed usually tracks how many bytes have to be read from RAM for each token, which is roughly the file size. On that basis, a 6.18 GB file should be slower than Qwen3.5 9B’s 5.68 GB on the NUC. It was 64% faster (5.03 against 3.07 tokens a second).
My interpretation, which I have not measured directly: because the embedding tables are lookups, each token reads only a tiny slice of those 3.5 GB. The weights that are read in full every token are the 3,023 MiB of layers, about the size of a 4B model file, which is about where the NUC’s speed landed. Treat this as a reasonable explanation rather than a proven one.
Will it run on your machine?
Likely fine
- An 8 GB NVIDIA card: 3.3 GB used, with plenty left for longer context
- An Apple Silicon Mac: 28.9 tok/s on a base M4
- A recent laptop or mini PC on CPU alone, if 5 tok/s is enough for you
Check first
- A 4 GB card: 3,327 MiB at 8K context fits on paper, but your desktop already uses some VRAM. I don’t own a 4 GB card to confirm
- Long context: the KV cache was 168 MiB at 8K. It will grow towards 64K; I didn’t measure how much
- Low system RAM: the 3.5 GB of embeddings sit in RAM, not on the card
To see the split on your own setup, start llama-server with -v and look for the load_tensors lines: CPU_Mapped model buffer size is what stayed in RAM and CUDA0 model buffer size is what went to the GPU. The common_params_fit_impl: projected to use line gives the total VRAM before anything is allocated.
- Three machines, one quant. I tested only NVIDIA’s Q4_K_M file. Other quants will differ.
- 8K context. NVIDIA supports up to 64K. Memory and speed at long context are not covered here.
- No speculative decoding. NVIDIA offers a separate “DFlash” drafter model that can speed up generation. I ran without it.
- Speed, not quality. I checked that answers were coherent, nothing more.
- The VRAM split is from Windows with CUDA. The Mac shares memory between CPU and GPU, so the split doesn’t apply there, and the NUC has no GPU in use.
- One explanation is an interpretation, as marked above.
Takeaway
Nemotron 3.5 Bolt is a genuinely easy model to run locally: 4B-class speed, and less than 3.5 GB of VRAM on an 8 GB card. The broader lesson is about sizing. For Gemma “E” models and their derivatives, the GGUF file size overstates the VRAM you need by close to double, and any rule of thumb that says “the file has to fit on your GPU” will turn people away who could run it fine. My own Will it run? calculator used to make exactly that assumption. I’ve changed it: for these models it now shows the split as VRAM plus system RAM, and its speed estimates match the three machines above.
What GPU would you try it on?
Your turn
Would you run a 4B-class model like Nemotron Bolt day to day?
I’m planning more “new model, measured” posts and want to know what hardware to prioritise.
Keep reading
Will it run? Local LLM calculator
I tested my local LLM calculator on four machinesAI helped me run the benchmarks and draft this article. The measurements, logs and hardware are my own.