This page helps you estimate the size of a local llama model: how much RAM, VRAM, and disk space you need to run a given parameter count (7B, 13B, 70B) at a chosen precision (4-bit, 8-bit, 16-bit). You enter the model’s parameter count and the precision, and the calculator returns the approximate memory required for loading the model, plus a rough total system memory recommendation.
Formula
The core formula is straightforward: Memory (GB) = Parameters (billions) × Bytes per parameter × Overhead factor. For a model with P billion parameters at B bits per weight, the raw weight storage is P × B / 8 gigabytes. A common overhead factor of 1.1 (10%) accounts for the runtime graph, optimizer states, and small buffers. The KV cache (key-value cache) for attention layers is estimated separately as roughly 0.5 GB per billion parameters for a 4096-token context window; this scales with context length.
Worked Example
Using the defaults: a 7B model at 4-bit precision.
- Raw weights: 7 × 4 / 8 = 3.5 GB
- With 10% overhead: 3.5 × 1.1 = 3.85 GB
- KV cache estimate: 7 × 0.5 = 3.5 GB
- Total: 3.85 + 3.5 = 7.35 GB
The calculator rounds to 7.4 GB total. This matches the typical figure cited for running a 7B model in 4-bit: around 6–8 GB of RAM/VRAM.
What “Size” Really Means for a Local Llama Model
When people ask “size of local llama model,” they usually mean one of three things: the number of parameters (7B, 13B, 70B), the file size on disk (the GGUF or safetensors file), or the runtime memory footprint. This calculator addresses the third, which is the practical bottleneck. The disk file is roughly the same as the raw weight size (before overhead) – a 7B Q4 file is about 3.5–4 GB. The parameter count is fixed by the model architecture; you cannot change it without retraining.
How Precision Affects Size
Precision is the biggest lever you control. A 7B model in 16-bit floats takes ~14 GB; in 8-bit it takes ~7 GB; in 4-bit it takes ~3.5 GB. Lower precision (4-bit) is the default for most consumer GPUs because it fits in 8–12 GB VRAM. The trade-off is a small quality loss, often imperceptible for chat but measurable on benchmarks. Quantization methods like GGUF Q4_K_M or GPTQ 4-bit are standard; they store weights as integers with a scale factor, not as true 4-bit floats.
Choosing the Right Size for Your Hardware
Match the model size to your available memory:
- 8 GB VRAM (e.g., RTX 3060, 3070): 7B at 4-bit is comfortable; 13B at 4-bit is tight and may spill to RAM.
- 12 GB VRAM (e.g., RTX 3060 12GB, 4070): 13B at 4-bit works well; 7B at 8-bit is also fine.
- 16 GB VRAM (e.g., RTX 4080, 4090 laptop): 13B at 8-bit or 30B at 4-bit (with some offloading).
- 24 GB VRAM (e.g., RTX 3090, 4090): 30B at 4-bit comfortably; 70B at 4-bit requires offloading to system RAM.
- CPU-only (32 GB RAM): 7B at 4-bit runs slowly but works; 13B at 4-bit is usable with a good CPU.
Always leave some headroom – the operating system and other programs take 1–2 GB. If the total exceeds your VRAM, the inference software (llama.cpp, Ollama) will offload layers to system RAM, which slows generation dramatically.
Common Mistakes and Misconceptions
The most common error is confusing disk size with runtime memory. A 7B Q4 file is ~3.8 GB on disk, but loading it needs ~7 GB because of the KV cache and overhead. Another mistake is ignoring the KV cache: it grows linearly with context length and batch size. A 70B model with a 32k context can have a KV cache of 16+ GB alone. Also, “7B” is not a size in bytes; it is the number of parameters. The actual bytes depend on precision.
A frequent misconception: “I have 16 GB RAM, so I can run a 13B model.” True, but it will be slow if it runs on CPU. For GPU inference, you need the model to fit in VRAM, not system RAM. If you have a 6 GB GPU and 32 GB RAM, a 13B Q4 model will run, but each token might take seconds rather than milliseconds.
Practical Tips for Running Local Models
- Use llama.cpp or Ollama – both handle offloading and quantization automatically.
- Start with a 7B Q4 model – it is the sweet spot for most people.
- If you run out of memory, reduce the context length first; it directly shrinks the KV cache.
- Close other applications that use GPU memory (browsers with hardware acceleration).
- For CPU-only, use a model with fewer parameters and a small context; expect 1–5 tokens per second.
Frequently Asked Questions
How many GB is a 7B model? In 4-bit, about 3.5–4 GB on disk and roughly 7 GB in memory with a 4k context. In 8-bit, double those numbers. In 16-bit, about 14 GB on disk and 15+ GB in memory.
Can I run a 70B model on 32 GB RAM? Yes, but only in 4-bit and with heavy CPU offloading. The model weights alone are ~35 GB, so you need at least 48 GB RAM to be comfortable. Expect very slow generation (a few tokens per second).
What does Q4_K_M mean? It is a quantization method in GGUF format. Q4 means 4-bit weights, K refers to a specific quantization scheme, and M is the variant (medium). It offers a good balance between size and quality.
Does VRAM matter more than RAM? For speed, yes. If the model fits entirely in VRAM, generation is fast (20–100 tokens/s). If it spills to system RAM, speed drops to 1–5 tokens/s. RAM matters for capacity, but VRAM matters for performance.
Is a larger model always better? Not necessarily. A 13B model at 4-bit often outperforms a 7B model at 8-bit on reasoning tasks, but the 7B may be faster and easier to run. Quality also depends on the training data and fine-tuning, not just parameter count.
How do I check my GPU’s VRAM? On Windows, open Task Manager > Performance > GPU. On Linux, run nvidia-smi in the terminal. The memory shown is total VRAM; subtract ~1 GB for the desktop environment.
Related
Formula used
in params | Parameters (billions) | B | 7 | 1 | 1000 sel prec | Precision | 4-bit Q4=4, 8-bit Q8=8, 16-bit FP16=16 out vram | VRAM / RAM for weights | GB | params * prec / 8 * 1.1 | 1 out kv | KV cache (context 4k) | GB | params * 0.5 | 1 out total | Total memory needed | GB | params * prec / 8 * 1.1 + params * 0.5 | 1
Each line is one input (in, sel, list) or one result (out: label, unit, formula, decimals). How the calculators work
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.
- LDL to HDL Ratio: What It Means & How to Calculate It2.00
- Kappa to Lambda Ratio: What It Means & How to Use It
- Jasmine Rice to Water Ratio: Perfect Every Time1.50 cups
- How to Find Decibels from Intensity
- How to Do S/Z Ratio: A Step-by-Step Guide1.25
- How to Convert to Decibels (dB) Easily