How much memory does a local LLM need? RAM, VRAM and context

A model's file size is not its total memory requirement. Context, intermediate data and the runtime need space alongside the weights. “An 8 GB model fits into any 8 GB of memory” is therefore a poor basis for choosing a computer.

Start with three details: the exact model and quantization, the context length you need and the number of concurrent requests. Without them, “how much RAM does AI need?” is too broad to answer usefully.

What uses memory

The KV cache stores intermediate attention data so it does not all have to be recalculated for every generated token. Its strategy affects memory use. Hugging Face explanation.

• Hugging Face explanation

The practical trade-off is simple: the same model may need considerably more memory with a long context than a short one.

Estimate weight memory

A first approximation is:

weight bytes ≈ parameter count × bits per parameter / 8

The following is arithmetic for hypothetical models, not measurements of specific files or computer requirements. It uses decimal GB: one billion bytes. B denotes billion parameters.

Real quantization formats include metadata and may store different model components at different precisions. The table shows the scale, but does not replace the actual file's model card or a trial run.

To convert bytes to GiB, divide by 1,073,741,824. Keep GB and GiB distinct when comparing sizes.

RAM versus VRAM

RAM is system memory; VRAM is memory on a discrete GPU. Model placement depends on the hardware and inference engine. In unified-memory systems, the CPU and GPU share a resource that also has to accommodate the operating system.

You cannot simply add 16 GB of RAM to 8 GB of VRAM and treat the result as universally equivalent to a 24 GB graphics card. Work splitting and transfer overhead depend on the particular setup.

In Ollama, `ollama ps` shows how a loaded model is placed between CPU and GPU. Interpreting the output.

• Interpreting the output

How context affects memory

For a conventional full KV cache, a rough estimate is:

cache bytes ≈ 2 × layers × KV heads × head dimension
              × tokens × bytes per element × sequences

The factor of two accounts for keys and values. This is a simplified model for applicable architectures, not a universal LLM calculator. Sliding attention, compression and other implementations change the calculation. Cache strategies.

• Cache strategies

Compare configurations at the same configured context. If one model receives a short function and another an entire repository, their memory difference alone establishes very little.

Ollama lets you change context length; increasing it needs additional memory. Check current placement and context with `ollama ps`. Context configuration.

• Context configuration

Test your computer

Record the model, exact tag, quantization, engine and engine version. Run the same task with a short input and a longer one. Save:

1. Configured context and concurrent request count.
2. Memory use before and during execution.
3. CPU/GPU placement.
4. Time to the first response and total duration.
5. Result quality against predefined requirements.

Repeat the run to distinguish model loading from steady operation. If memory runs out, reduce one parameter first instead of changing the model, context and all settings together.

What to try with 16 or 32 GB

There is no hardware-independent list of guaranteed configurations. With limited memory, start with a small quantized model and a short task, as in the Ollama guide. Leave room for the system and applications.

• Ollama guide

If the model loads but runs uncomfortably slowly, inspect placement and context. Successful loading does not mean a configuration is practical for everyday work.