AI & Local Models
KV Cache Quantization in llama.cpp for Consumer VRAM
Calculate llama.cpp KV-cache memory with block overhead, configure K/V types, and test fit and quality on your GPU with an illustrative 32k/64k example.
Tagged
quantization
RSS Tag2 articles
AI & Local Models
How to Test Llama 3.1 8B Quantizations on an RTX 3060 12 GB
Compare Q4_K_M, Q5_K_M, and Q6_K responsibly with pinned GGUF files, repeated llama-bench runs, measured VRAM, and retained quality tests.