Determining VRAM Requirements for Running LLMs Locally
When deploying a local LLM, the primary concern is straightforward: will it fit within your GPU's capacity? The answer hinges on the model's size, the quantization method used, and the length of the context. This guide offers a practical baseline for selecting the appropriate amount of VRAM.
The Impact of Quantization on VRAM
Quantization lowers the precision used for storing model weights. Reducing the bit depth makes the model smaller and reduces VRAM consumption, albeit with some trade-off in output quality.
| Quant | Bits per weight | Typical use |
|---|---|---|
| Q8_0 | 8 | Very high quality |
| Q6_K | ~6.6 | Very good quality |
| Q5_K_M | ~5.5 | Good balance of size and quality |
| Q4_K_M | ~4.5 | Good balance of size and quality |
| Q3_K_M | ~3.5 | Lower VRAM usage, greater quality loss |
Q4_K_M is a popular choice when VRAM resources are constrained. If you have more VRAM to spare, Q5 or Q6 allows you to run the same model with less aggressive quantization.
Estimated VRAM by Model Size
The figures below are rough estimates for the model weights alone. Actual VRAM requirements will be higher, as the runtime, KV cache, and context window also consume memory.
| Model size | Q8_0 | Q6_K | Q4_K_M | Q3_K_M |
|---|---|---|---|---|
| 4B | ~5 GB | ~4 GB | ~3 GB | ~2.5 GB |
| 8B | ~9 GB | ~7 GB | ~5.5 GB | ~4.5 GB |
| 12B | ~13 GB | ~10 GB | ~8 GB | ~6.5 GB |
| 14B | ~16 GB | ~12 GB | ~9 GB | ~7.5 GB |
| 27B | ~30 GB | ~22 GB | ~17 GB | ~13 GB |
| 32B | ~36 GB | ~27 GB | ~20 GB | ~16 GB |
| 70B | ~80 GB | ~60 GB | ~42 GB | ~34 GB |
These are estimates rather than strict limits. Variations in model architecture and quantization format can influence the final size.
Performance by VRAM Capacity
| VRAM | Practical range | Current examples |
|---|---|---|
| 8 GB | Small models, roughly 4B to 9B | Gemma 4 E4B, Qwen3.5 9B |
| 12 GB | Small to mid-sized models, roughly 9B to 14B | Gemma 4 12B, Qwen3.5 9B |
| 16 GB | 12B to 27B with lower quantization | Gemma 4 26B-A4B, Qwen3.6 27B at Q4 |
| 24 GB | 27B to 35B at Q4 to Q6 | Qwen3.8 27B, Gemma 4 31B |
| 32 GB | 27B to 35B at higher quantization | Qwen3.8 27B, Gemma 4 31B |
| 48 GB | Large dense models at lower quantization | 70B-class models at Q3 to Q4 |
| 80 GB | Large dense models at higher quantization | 70B-class models at Q4 to Q6 |
These ranges apply to models whose weights can reside on the GPU. Large MoE models operate differently: only a subset of parameters is active per token, but the full set of weights must still be stored. Therefore, a model with 100B or more total parameters will not fit in a 100B-sized VRAM budget simply because fewer parameters are active.
MoE Models
Mixture-of-Experts models consist of multiple groups of parameters known as experts. Since only certain experts are activated for each token, inference can be more efficient than a dense model with the same total parameter count.
However, inactive experts remain part of the model structure. Consequently, large MoE models may demand significantly more memory than their active parameter count implies. Very large models might require multiple GPUs or offloading to system RAM.
Context Length and VRAM Usage
Model weights represent only a portion of the total memory requirement. The KV cache expands as the context length increases; thus, running the same model at a 64K context may consume substantially more VRAM than running it at 4K.
- Longer contexts require more VRAM.
- KV cache precision influences memory consumption.
- Batch size and concurrent user counts also increase memory demands.
- Reserve some VRAM for the runtime instead of saturating the GPU entirely with model weights.
Practical Recommendations
- Verify the actual size of the specific quantized model you intend to run.
- Do not assume the model file size equals the exact VRAM requirement. Allow for KV cache and runtime overhead.
- If a model exceeds VRAM capacity, parts can be offloaded to system RAM, though inference speed will typically decrease.
- For long-context or agentic workloads, allocate more VRAM than the model weights alone would suggest.
- Multiple GPUs can be used to split a model if a single GPU lacks sufficient VRAM.
Run on DaDesktop
You do not need to purchase a GPU to run a local LLM. DaDesktop provides a cloud desktop equipped with the necessary VRAM, enabling you to run models directly without owning the hardware.
Select the VRAM tier that suits your model, load it, and begin using it. No setup, no hardware purchase, and no driver issues. Visit available GPUs to view your options.