Originally published at mrsaynothing.dev.

Walk into any local-LLM thread and the arguments are about GPUs. VRAM benchmarks, 24 GB cards, CUDA versus ROCm, whether the 3060 is still the people's card. Meanwhile the number that actually decides whether your model runs is sitting in the other slot, unbenchmarkable and unmarketed: how much RAM the machine has.

VRAM sells the dream. RAM decides whether the model boots at all — and how much context survives when it does.

I tested this on the box in front of me while writing this post: a Ryzen desktop with 32 GB of RAM and a GeForce RTX 3060 with 12 GB of VRAM. Pulled llama3.1:8b — the download page says 4.9 GB. Watch what it actually reserved:

That is the whole argument in one screen. A "4.9 GB model" reserved 7.0 GB before answering a single prompt — a 43% context tax — and kept 7,963 of 12,288 MiB of VRAM to itself. The weights were never the budget. The context was.

Where the extra gigabytes come from

The llama.cpp memory notes spell out the arithmetic that download pages omit: total memory = model weights + KV cache + compute buffer. Only the first term is constant. The KV cache grows linearly with context length, and the compute buffer grows with batch and graph shape. Ollama wraps llama.cpp, so the same law applies — which is why ollama ps reported 7.0 GB for a 4.9 GB tag at a 32,000-token window, all of it resident on the GPU.

A quantized model isn't a compromise. It's an admission that memory was always the real budget.

This is also why the GGUF quantization ladder exists at all. Q4 isn't a religion; it's what makes the memory arithmetic land inside hardware people own. And it's why two "identical" 8B setups behave nothing alike: same model, different context length, different machines entirely.

Mr Say Nothing

Sep 13

GGUF Quantization: Which Level Should You Use?

#ai #gguf #quantization #localllm

5 min read

The table nobody runs before buying

Same three model classes, both memory types, one 12 GB card and 32 GB of RAM — the configuration thousands of developers actually have:

Model class (Q4)	Download	Loaded + 32k ctx	On 12 GB VRAM	On 32 GB RAM
7–8B (`llama3.1:8b`)	4.9 GB	7.0 GB (measured)	100% GPU, ~8 GiB used	barely noticed
13–14B (`qwen2.5:14b`)	9.0 GB	~12 GB	offload begins	fine
27–32B (`gemma3:27b`)	17 GB	~20+ GB	CPU pulls the weight	the only reason it runs

Read the last two rows again. On VRAM alone, a 14B model at a real context window is already a split-offload job, and a 27B is impossible. On 32 GB of system RAM, both are merely slow. That difference — between impossible and slow — is the entire practical difference between RAM and VRAM. If it fits in VRAM, it's fast. If it fits in RAM, it works. If it fits in neither, you're swapping to an NVMe drive and time loses meaning.

Offload sits on PCIe, and the Ollama FAQ is straight about the cost: layers that don't fit on the GPU run on the CPU, and throughput drops hard as the GPU share shrinks. Nobody chooses that trade-off consciously. It happens silently, one layer at a time, and the symptom is just "local models feel overrated."

The honest ledger

Things that broke or surprised me while writing this piece, in order:

The 43% number itself. I expected the 8B model to occupy "about its file size." It reserved 7.0 GB against a 4.9 GB download. If that surprised me, it surprises everyone who reads spec sheets instead of ps output.
The screenshot session. My first capture grabbed the wrong window. My second worked, which is the one above. Receipts are a workflow, not a vibe — and a pull → capture → ollama rm cycle keeps the disk honest.
What didn't break: the GPU never spilled. 8B at 32k context on 12 GB is genuinely comfortable. The card is fine. The discourse around the card is what's miscalibrated.

None of this means GPUs don't matter — the 100% GPU line in that capture is why generation felt instant. It means the GPU is the second question. The runner choice comes after you know what fits, not before.

Mr Say Nothing

Sep 18

llama.cpp vs Ollama: Which Should You Run in 2026?

#ai #llamacpp #ollama #localllm

5 min read

The rule of thumb worth keeping

RAM you need = model file + KV cache for your real context window + 4 GB for being a computer. For 7–8B at Q4, 16 GB is comfortable. For 14B–32B, 32 GB stops being a luxury and starts being the point. Buy VRAM for the speed you want at the context you use; buy RAM for everything you'll ever load.

Check ollama ps once and the spec-sheet religion quietly ends.

So, two questions. When you spec your next dev box, are you buying VRAM for the benchmarks you'll post — or RAM for the models you'll actually run? And honestly: how many of the models you pulled at 2 a.m. did you delete before breakfast? I did, tonight, mid-post. Tell me I'm not the only one in the comments — and tell me which side of the RAM/VRAM split your build sits on.

More field notes at mrsaynothing.dev · code at GitHub · say hi: contact@mrsaynothing.dev

Even mid-sprint to a secret flight, the Navy’s tech chief has a pitch for investors

Petlibro’s new AI-powered feeder is a game changer for multi-cat homes

AI safety conversations have gotten unbelievable

Prices go up in 7 days. Get your Disrupt ticket now.

Vals, backed by Andreessen Horowitz, is looking to become the gold standard for AI benchmarking

India forces caller-ID apps to feed spam reports to telcos

How to build a real time voice AI agent in the browser

How to find the invisible product features users won’t ask for

Comparing AI agent sandbox platforms: E2B, Modal, Daytona, and more

How to use Vercel eve: The Next.js framework for AI agents

Understanding user stories in UX design: A practical guide

How to implement JWT authentication in NestJS

The End of Traditional UX

Architecture of WarchOS: A Custom Arch-Based Setup with Hyprland, a Dynamic CPU Scheduling Daemon, and an Automated Wine Prefix Manager

A Kafka Streams topology starts in a sandbox, not on your cluster

Nobody talks about RAM. Every local-LLM regret is a RAM problem.

[Boost]

Building FoxyInvoice — Chapter 2: Product thinking — pick a real problem, price it, position it

11 Best V0 Alternatives 2025: Real Reviews & Pricing

8 AI Developer Tools for Faster & Smarter Development 👨‍💻🚀

The 5-Minute Kubernetes Cluster Health Check

Livewire 4: The Future of PHP Components

Laracon 2025 Keynote

Laravel Wayfinder