help-calc

08 Numbers and Units

Llama.cpp vs Llama: Key Differences

Card 08-0268 · Updated · Standard formula, shown below

Minimum RAM needed 5.5 GB
Model size (B params)
7
Input
Quantization bits
4
Input
RAM for overhead (GB)
2
Input
You enter
Model size (B params), Quantization bits, RAM for overhead (GB)
You get
Minimum RAM needed (GB)
Rounding
1 dp

Llama.cpp is a software library that runs Meta’s Llama language models locally on your own hardware, while Llama refers to the actual AI models themselves. In short: Llama is the brain, and llama.cpp is the machine that lets you run that brain on a CPU, GPU, or even a smartphone without sending data to a cloud server.

Formula

The calculator estimates the minimum RAM you need to run a quantized Llama model locally. The formula is:

RAM needed (GB) = (Model size in billions of parameters × Quantization bits) / 8 + Overhead

  • m – model size in billions of parameters (e.g., 7 for Llama 2 7B, 13 for 13B, 70 for 70B)
  • q – quantization bits (4, 5, 6, or 8; lower means smaller file but lower quality)
  • r – extra RAM for the operating system, tokenizer, and context window (typically 1–4 GB)

The division by 8 converts bits to bytes. A 7B model at 4-bit quantization uses about 3.5 GB just for weights; overhead adds a bit more for the running process.

Worked Example

Using the defaults above (7B params, 4-bit quantization, 2 GB overhead):

  1. Weights memory = 7 × 4 / 8 = 3.5 GB
  2. Add overhead = 3.5 + 2 = 5.5 GB

The calculator returns 5.5 GB minimum RAM. In practice, you’d want at least 8 GB total system RAM to leave room for the OS and other apps.

Background: What Each One Actually Is

Llama is a family of large language models released by Meta starting in February 2023. The original Llama came in 7B, 13B, 33B, and 65B parameter sizes. Later versions include Llama 2 (2023) and Llama 3 (2024), with sizes from 8B up to 405B. These models are open-weight, meaning anyone can download the trained weights and run them locally.

Llama.cpp, created by Georgi Gerganov in March 2023, is a C/C++ implementation that runs these models without requiring Python, PyTorch, or a powerful GPU. It was originally written to run the 4-bit quantized version of the original Llama on an Apple MacBook, and it quickly grew into the de facto standard for local LLM inference.

The key distinction: you cannot run “Llama” by itself—you need software like llama.cpp, Ollama, or Hugging Face’s transformers to load the weights and generate text. Llama.cpp is one of several runners, but it is the most popular for CPU-only machines and older GPUs.

How Llama.cpp Works

Llama.cpp converts the original PyTorch model files (which can be tens of gigabytes) into a GGUF format file. GGUF is a binary format that stores the weights in a quantized form—usually 4-bit or 5-bit integers instead of 32-bit floats. This reduces file size by roughly 75–80%.

The library then uses SIMD instructions (AVX2, AVX512, NEON) to accelerate matrix multiplication on CPUs. It can also offload layers to an NVIDIA GPU via CUDA, an AMD GPU via Vulkan, or Apple Silicon via Metal. The result is that a 7B model runs at 10–30 tokens per second on a decent laptop CPU, and much faster on a GPU.

You interact with it through a command-line interface, a simple web server, or a Python binding. It supports features like context extension, grammar-constrained generation, and LoRA adapters for fine-tuning.

Choosing Between Llama and Llama.cpp

This is not really a choice—you need both. The real decision is which model size and which runner software to use. Here is how to think about it:

  • If you have a modern NVIDIA GPU with 8+ GB VRAM: use llama.cpp with a 13B or 30B model at 4-bit quantization for good quality and speed.
  • If you only have a CPU: stick with 7B or 8B models at 4-bit quantization. Expect 5–15 tokens per second depending on your processor.
  • If you want the easiest setup: use Ollama or LM Studio, which wrap llama.cpp behind a friendly interface.
  • If you want to fine-tune or train: use the original Llama weights with PyTorch, not llama.cpp, which is inference-only.

The model size you can run is limited by your RAM/VRAM. Use the calculator above to check before downloading a multi-gigabyte file.

Common Problems and Misconceptions

A frequent misconception is that llama.cpp is a model—it is not. People say “I downloaded llama.cpp” when they actually downloaded a GGUF file of a Llama model. Another common issue is running out of memory: a 70B model at 4-bit needs about 35 GB, so it will not run on a 16 GB laptop. The library will crash or swap to disk, producing extremely slow output.

Another problem is confusing quantization levels. Q4_K_M and Q5_K_M are different quality/size trade-offs. Q4 is smaller but slightly less accurate; Q8 is nearly lossless but twice the size. For most casual use, Q4_K_M is the sweet spot.

Finally, people often expect cloud-level quality. A 7B quantized model is far less capable than GPT-4 or Claude. It is useful for coding autocomplete, summarization, and roleplay, but it will hallucinate more and follow instructions less reliably.

Costs and Hardware Requirements

Running Llama via llama.cpp is free in terms of software, but you pay for hardware. A 7B model at 4-bit needs about 4–6 GB RAM—most modern laptops have that. A 13B model needs 8–10 GB. A 70B model needs 35–40 GB, which means a workstation or a cloud instance.

Electricity is a real cost for long sessions. A CPU running at full load draws 50–150 W; a GPU can draw 200–450 W. Running a model for an hour might cost a few cents to a few dozen cents depending on your electricity rate. Cloud APIs charge per token, which can be cheaper for occasional use but more expensive for heavy daily use.

There is no licensing fee for Llama weights, but Meta’s community license imposes restrictions on commercial use for models over 700M parameters if you have more than 700 million monthly active users. For most individuals and small businesses, this is irrelevant.

Safety and Privacy Considerations

The biggest advantage of llama.cpp over cloud APIs is privacy: your prompts never leave your machine. This matters for confidential documents, medical data, or proprietary code. However, local models are not automatically safe—they can generate biased, toxic, or incorrect content just like cloud models, and they have no built-in content filter unless you add one.

Also be aware that downloading model files from unofficial sources carries a supply-chain risk. Always download GGUF files from the official Hugging Face repository of the model creator or a trusted uploader like TheBloke. Verify checksums if available.

Finally, running a local model does not make your computer invulnerable. Maliciously crafted model files could exploit parser vulnerabilities, though no widespread attacks have been reported. Stick to reputable sources.

Frequently Asked Questions

Is llama.cpp faster than the original Llama code? Yes, for inference on consumer hardware. The original Llama codebase is a PyTorch reference implementation designed for research, not speed. Llama.cpp is heavily optimized C/C++ that often runs 2–5× faster on the same CPU.

Can I run Llama 3 70B on a 32 GB Mac? Yes, with 4-bit quantization it needs about 35 GB, so a 32 GB machine will struggle unless you use a smaller context or accept swapping. A 64 GB Mac runs it comfortably. Use the calculator above to check your specific setup.

What is the difference between GGUF and GGML? GGML was the older format used by llama.cpp; GGUF is the newer replacement that supports more metadata and is backward-compatible. Since August 2023, all current llama.cpp builds use GGUF. Older GGML files are deprecated and will not load in recent versions.

Do I need a GPU to use llama.cpp? No. It runs on CPU-only machines, which is its main selling point. A GPU makes it faster, but even a 7B model works on a Raspberry Pi 5 at a few tokens per second.

Which is better: llama.cpp or Ollama? Ollama is built on top of llama.cpp and adds a simple CLI, model management, and an OpenAI-compatible API. If you want a one-command setup, use Ollama. If you want fine-grained control over compilation flags, GPU layers, and sampling parameters, use llama.cpp directly.

Can I fine-tune a model with llama.cpp? No, llama.cpp is inference-only. For fine-tuning, use the original PyTorch code, Hugging Face’s PEFT library, or tools like Axolotl. After fine-tuning, you can convert the result to GGUF and run it with llama.cpp.

Related

Formula used
in m | Model size (B params) | - | 7 | 1 | 70
in q | Quantization bits | - | 4 | 1 | 8
in r | RAM for overhead (GB) | - | 2 | 0 | 16
out gb | Minimum RAM needed | GB | m * q / 8 + r | 1

Each line is one input (in, sel, list) or one result (out: label, unit, formula, decimals). How the calculators work

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.

More on this shelf

The whole Numbers and Units shelf →