KV Cache Quantization in llama.cpp for Consumer VRAM

A model that loads into GPU memory can still run out of room when you raise its context limit. Weights and the key/value (KV) cache occupy different allocations: the weight file sets much of the starting cost, while the cache reserves space for attention state across the configured context. You can estimate the cache before launch. You still have to measure the complete process on your own card; a cache calculation is not a guarantee that 32k or 64k context will fit or run well.

This guide uses a hypothetical full-attention model with 32 layers, 8 KV heads, and 128 values per head. Check the actual model's metadata and llama.cpp startup log before using those dimensions. For a separate look at weight-file tradeoffs, see the local quantization test. Weight quantization and KV-cache quantization are independent choices.

What the KV Cache Stores

At each layer, attention stores a key and a value representation for retained tokens. In this simple full-attention example, the number of scalar values in each of K and V is:

layers × KV heads × head dimension × allocated context tokens

Multiply that count by the storage size for K and again for V, then add the two results. This is an estimate of cache tensor storage, not total GPU allocation. Some models use sliding-window attention, differing K and V dimensions, or other cache arrangements; do not put their dimensions into this simple table without checking the runtime layout.

Context sets capacity rather than a count of useful words. Parallel sequences and extra application buffers may reserve more space, and a model’s claimed maximum says nothing about accuracy on every task at that length. This calculation estimates the bytes required by the chosen K and V tensor allocation. Test useful context length separately.

Count the Quantization Blocks, Not Just the Bits

For the hypothetical dimensions above, 32,768 context tokens give 1,073,741,824 values in K and the same number in V. At f16, each value uses two bytes, so K plus V occupy 4,294,967,296 bytes: 4.00 GiB, or about 4.29 GB. At 65,536 tokens, that doubles.

The q8_0 and q4_0 formats are block quantizers, so their effective sizes are not exactly one byte or half a byte per value. In ggml's quantization structures^[1], q8_0 stores 32 signed bytes plus a two-byte scale: 34 bytes per 32 values. q4_0 stores 16 packed bytes plus a two-byte scale: 18 bytes per 32 values. Ignoring the scales understates cache size.

The table uses binary GiB and estimates only the cache tensors for these model dimensions. Leave room for weights, GPU compute buffers, graphics/display use, allocator padding, and other runtime allocations. Calculate K and V separately if their types differ. For mixed attention or another architecture, take the startup allocation from the runtime rather than projecting capacity from this table.

Here is a small calculator you can change for a model whose KV dimensions you have verified. Its integer arithmetic rounds each quantized tensor up to whole 32-value blocks:

layers = 32
kv_heads = 8
head_dim = 128
context = 32768
values = layers * kv_heads * head_dim * context

bytes_per_block = {"f16": (1, 2), "q8_0": (32, 34), "q4_0": (32, 18)}

def tensor_bytes(kind):
    block_values, block_bytes = bytes_per_block[kind]
    return ((values + block_values - 1) // block_values) * block_bytes

for kind in bytes_per_block:
    gib = (tensor_bytes(kind) + tensor_bytes(kind)) / 1024**3
    print(f"{kind}: {gib:.3f} GiB")

For the chosen dimensions, it prints 4.000, 2.125, and 1.125 GiB. Change context to 65536 and the values double. A calculator result is only a starting estimate. Confirm the model's KV allocation and overall GPU memory at startup and under a representative prompt.

Apply the Types in llama.cpp

Current llama-server accepts separate --cache-type-k and --cache-type-v options. Its server reference^[2] lists f16 as their default and --flash-attn as on, off, or auto (default auto). Use a pinned build and the actual path to a compatible GGUF file. On a backend that supports the selected types, a test invocation is:

llama-server \
  --model /path/to/model.gguf \
  --ctx-size 32768 \
  --n-gpu-layers 99 \
  --flash-attn on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0

This command describes settings; it has not been validated on an NVIDIA card here. It may reject the model, allocate additional buffers, offload differently, or exceed available memory. Confirm the model's actual supported context, the selected backend and cache types, the startup allocation reported by llama.cpp, and measured device memory. Then send a representative long prompt, because successful startup alone does not prove useful latency or accuracy.

Quantized V cache requires a Flash Attention path in current builds; forcing --flash-attn off is not a portable way to retain quantized V. Conversely, Flash Attention support varies by backend and compiled kernel set. The CUDA build options^[3] list which K/V type pairs are compiled by default and allow additional pairs with GGML_CUDA_FA_QUANTS. K and V can use different types where the build and backend support the pair. Inspect the log: a mixed pair may work, fall back or fail. Start a comparison with q8_0/q8_0 if it is supported, then judge another pair against the same workload.

If 32k fails to fit, do not jump straight to a promised 64k configuration. Reduce context, use a smaller or more heavily quantized weight file, reduce parallel work, or test a lower-bit cache type one change at a time. A card advertised as 8 GB or 12 GB cannot devote all of that capacity to the KV tensors. A weight file's disk size is also not an exact prediction of its GPU allocation.

Check Quality and Workload, Too

Lower-bit cache types trade precision for memory. The resulting quality change depends on the model, prompt, and task; a single perplexity number from another model is not a transferable promise. Compare answers on your own long-context set, including retrieval of details placed early and late, code references, and refusal when the answer is absent. Record the model file hash, llama.cpp commit or release, cache types, context, backend, and prompt. If a setting changes, rerun the same cases.

Watch both prompt processing and generation. Some backends may accept a type pair but take a slower path than expected. A successful short prompt is insufficient: it does not fill the reserved context, and it cannot show whether long-range retrieval remains reliable. Keep an f16 or q8_0 baseline where it fits, then compare the lower-bit candidate rather than presuming its memory saving is free.

Ollama has its own service settings and release cadence. If you use Ollama rather than bare llama.cpp, consult the current Ollama FAQ^[4] for its Flash Attention and KV-cache controls and test those separately. A llama-server flag is not an Ollama command-line flag, and the two programs need not expose a feature at the same time.

Read the model dimensions, calculate K and V with block scales, and budget for the rest of the runtime. That narrows the configurations worth trying on the target GPU. A representative run of the pinned build then answers the remaining questions: whether it fits, whether latency is adequate, and whether answer quality holds up.