Llama.cpp Batch Size: What It Does and How to Set It

The batch size in llama.cpp controls how many tokens the model processes in parallel during a single forward pass, directly affecting generation speed and memory usage. You set it with the -b or --batch-size flag, and the optimal value depends on your hardware, model size, and whether you are doing prompt processing or text generation.

Formula

The calculator above gives two rough estimates. The first, estimated GPU memory, approximates the memory needed to hold the model weights and the KV cache for the batch. A 7B model in 16-bit precision uses about 14 GB for weights (7B × 2 bytes), plus a small amount per token in the batch. The formula is:

Memory (GB) ≈ n × 2 + b × 0.001

where n is the model size in billions of parameters and b is the batch size in tokens. The 0.001 GB per token is a very rough placeholder; the actual KV cache size depends on the number of layers, attention heads, and context length.

The second output, tokens per second, is a heuristic that shows how batch size affects speed: larger batches increase throughput (tokens per second) but also increase latency per token. The formula 30 / (1 + b / 1024) assumes a base speed of 30 tokens/s at batch size 1 and halves the speed when the batch reaches 1024. This is illustrative, not a benchmark.

Worked Example

Using the default values: batch size = 512, model size = 7B, GPU memory = 8 GB.

Memory estimate: 7 × 2 + 512 × 0.001 = 14 + 0.512 = 14.512 GB. That is far above the 8 GB GPU, so the model would not fit entirely in VRAM; you would need CPU offloading or a smaller batch.

Speed estimate: 30 / (1 + 512/1024) = 30 / 1.5 = 20 tokens/s. This suggests that with a batch of 512, the model might produce around 20 tokens per second in ideal conditions, but real performance varies wildly.

What Batch Size Actually Does in llama.cpp

In llama.cpp, the batch size is the number of tokens that the model processes in a single matrix multiplication. During prompt processing, the entire prompt is processed in chunks of this size. During generation, the batch size determines how many tokens are generated in parallel when using speculative decoding or when batching multiple sequences together.

A larger batch size means more tokens are computed at once, which improves GPU utilization and throughput. However, it also increases the size of the intermediate matrices and the KV cache, requiring more memory. If the batch is too large, you run out of VRAM and the program crashes or falls back to slow CPU processing.

For single-sequence generation (the typical chat use case), the batch size during decoding is effectively 1, because you generate one token at a time. The batch size matters most during the prompt processing phase, where a larger batch can significantly speed up the initial response time.

How to Choose the Right Batch Size

There is no universal “best” batch size; it depends on your hardware and workload. Here are practical guidelines:

• Start with the default. llama.cpp’s default batch size is 512, which works well for most CPU and GPU setups.
• For GPU inference, try increasing the batch size to 1024 or 2048 if you have enough VRAM. Monitor memory usage; if it crashes, lower it.
• For CPU inference, a smaller batch (128–256) often gives better latency, while a larger batch improves throughput if you are processing many prompts at once.
• For prompt processing, a larger batch (1024+) speeds up the initial response, but the benefit diminishes beyond a point.
• For batch generation (multiple sequences at once), set the batch size to at least the number of sequences you want to generate in parallel.

Common Problems and Misconceptions

“Increasing batch size always makes generation faster.” Not true. For single-token generation, the batch size is effectively 1, so changing it has no effect on per-token speed. It only affects prompt processing and multi-sequence batching.

“Batch size is the same as context length.” No. Context length is the total number of tokens the model can attend to; batch size is how many tokens are processed in one compute step. You can have a context of 4096 but a batch size of 512, meaning the prompt is processed in 8 chunks.

“A batch size of 1 is always slow.” For prompt processing, yes, but for generation it is the norm. The speed of generation is mostly determined by the model size and hardware, not the batch size.

“Batch size affects output quality.” No. The batch size is a performance parameter; it does not change the model’s predictions. You get identical outputs regardless of batch size (assuming no numerical differences from floating-point rounding).

Batch Size vs. Other Performance Flags

llama.cpp has several flags that interact with batch size:

• -c (context size): the maximum number of tokens the model can process in total. Batch size must be ≤ context size.
• -ub (batch size for prompt processing): overrides the batch size specifically for the prompt phase.
• -n (number of tokens to generate): the length of the output. Batch size does not limit this.
• -t (threads): on CPU, more threads can help with larger batches, but too many threads can cause overhead.

If you set -b larger than the context size, llama.cpp will error. If you set it too high for your memory, you may get an “out of memory” error or the program will be killed by the OS.

Memory Usage and Batch Size

The main cost of a larger batch size is memory, specifically the KV cache. For a model with n_layers layers and n_heads attention heads, the KV cache per token is roughly 2 × n_layers × n_heads × head_dim bytes. For a 7B model, this can be around 1 MB per token. So a batch of 512 uses about 512 MB of extra memory, while a batch of 2048 uses 2 GB.

If you have limited VRAM, you can reduce the batch size or use the --no-mmap flag to offload weights to CPU. Alternatively, use quantized models (e.g., Q4_K_M) which reduce memory usage significantly, allowing larger batches.

Frequently Asked Questions

What is the default batch size in llama.cpp? The default is 512. You can see it in the help output (./main --help) and change it with -b.

Does a larger batch size use more RAM? Yes, both VRAM and system RAM increase with batch size because the KV cache grows. For CPU inference, the memory usage also increases, but less dramatically.

Can I set batch size higher than context size? No. The batch size must be less than or equal to the context size. If you try, llama.cpp will throw an error.

Why does my generation speed not improve when I increase batch size? Because during generation, the model processes one token at a time regardless of the batch size. The batch size only matters for prompt processing and when generating multiple sequences in parallel.

What is the difference between batch size and threads? Batch size is the number of tokens processed in parallel; threads are the number of CPU cores used for matrix multiplication. They are independent, but both affect performance.

Is there a recommended batch size for a 7B model on 8GB VRAM? With a 7B model in 4-bit quantization (about 4 GB), you can use a batch size of 512 or 1024. For 16-bit, the model alone uses 14 GB, so you would need CPU offloading and a smaller batch (128–256).

Practical Tips for Tuning

When tuning batch size, monitor the following:

• Memory usage: Use nvidia-smi on Linux or Task Manager on Windows to see VRAM usage. If it exceeds your GPU’s capacity, reduce batch size.
• Time to first token: Measure how long it takes to get the first output after sending a prompt. A larger batch size often reduces this.
• Tokens per second: Use llama.cpp’s built-in timing output (--verbose-prompt or the final stats) to see the speed. If it plateaus, increasing batch size further won’t help.
• CPU vs GPU: On CPU, a batch size of 512 may be slower than 128 due to cache thrashing. Experiment with powers of two: 64, 128, 256, 512, 1024.

Remember that the optimal value changes with the model (7B vs 70B), the quantization, the hardware, and the workload. Always test with your actual use case.

Related

• 431mm to Inches Conversion & How to Do It
• 2 x 2 Inches in cm: Quick Conversion & Uses
• 50mm to Inches: Quick Answer & Conversion Guide

in b | Batch size | tokens | 512 | 1 | 8192
in n | Model size (params) | billions | 7 | 1 | 70
in g | GPU memory | GB | 8 | 1 | 128
out m | Estimated memory (GPU) | GB | n * 2 + b * 0.001 | 1
out t | Tokens per second (rough) | tok/s | 30 / (1 + b / 1024) | 1