LLM VRAM Calculator & Guide

Quick answer and tools

Short answer: Use the simple formula M = P × (Q/8) × 1.2 as a baseline, then test with a calculator. Try the VRAM calculator, the Hugging Face VRAM tool, or the GPU Memory Calculator on GitHub for quick estimates.

Why VRAM matters

If you run an LLM on a GPU, VRAM holds the model weights, temporary activations, and the key-value (KV) cache used by attention. Too little VRAM means the model won’t load or will run very slowly. Estimating VRAM helps you pick the right GPU or cloud instance and avoid surprise costs.

Core formula and rule of thumb

Two easy rules:

• Rule of thumb: about 2 GB of VRAM per 1B parameters for half precision (FP16). This is quick and good for rough planning (source: Modal).
• Precise baseline: use M = P × (Q/8) × 1.2. Here P is parameters in billions, Q is bits per weight (16, 8, or 4), and 1.2 adds ~20% overhead for KV cache and small extras (see this guide and Hamel's notes).

How to read the formula

• If you load weights in FP16, set Q=16. The formula becomes M = P × 2 × 1.2.
• For 8-bit quantization, use Q=8. For 4-bit, use Q=4.

Examples (use the formula to verify)

All numbers give a ballpark estimate. Real runs vary by framework, drivers, and extra features.

Model	Precision	Estimate (GB)
Llama3 8B	FP16	8 × (16/8) × 1.2 = 19.2 GB
Llama3 8B	4-bit	8 × (4/8) × 1.2 = 4.8 GB
Mistral 7B	FP16	7 × (16/8) × 1.2 = 16.8 GB
Llama3 70B	FP16	70 × (16/8) × 1.2 = 168 GB
Llama3 70B	4-bit	70 × (4/8) × 1.2 = 42 GB

Note: Some sources show lower FP16 numbers (for example a simple 2 GB/B rule gives 140 GB for 70B). Different calculators use slightly different overhead assumptions. See APXML and AIMultiple for contrasting views and tools.

What else uses VRAM?

• KV cache: Larger sequence lengths (longer inputs) need more KV cache memory.
• Batch size: More inputs at once raises activation memory and can multiply VRAM needs.
• Activations: Some frameworks store activations to speed up decoding.
• Library overhead: Runtime libraries (CUDA, kernels, driver reserves) also use space.

How to pick numbers for the calculator

1. Model size: enter parameters in billions (e.g., 7 for 7B).
2. Precision/Quantization: choose FP16, 8-bit, or 4-bit (Q4). The calculator uses the Q value.
3. Sequence length: set expected max tokens (512, 1024, 2048). Longer sequences increase KV cache.
4. Batch size: set to 1 for single chat users; increase for throughput needs.

Practical VRAM saving techniques

If your GPU is close but short on VRAM, try these options.

• Quantize weights to 8-bit or 4-bit (gguf Q4_K_M is common). This cuts model weight size dramatically.
• KV cache precision: Some tools allow lower-precision KV cache to save memory.
• Offload parts of the model to CPU RAM or NVMe with frameworks that support tensor offloading.
• Paged attention / FlashAttention: These methods reduce activation memory. See community tools and research posts for options.
• vLLM and server frameworks: Use optimized servers like vLLM or specialized runtime to increase throughput and sometimes reduce memory use.
• Model sharding / multi-GPU: Split the model across GPUs if a single GPU does not have enough VRAM.

Tools and links to try

• APXML VRAM Calculator — quick estimates with sequence length and batch size options.
• Hugging Face Space — community-run calculator for many HF models.
• GitHub GPU Memory Calculator — simple open-source page you can run locally.
• Modal blog — quick rules and cloud examples.
• Medium guide and Hamel's notes — deeper reads on the math and caveats.
• Video walk-throughs: GPU VRAM Calculation (YouTube) and How Much VRAM My LLM Model Needs.

Quick checklist before you buy or rent GPU

• Run the formula for your model and desired precision.
• Estimate extra VRAM for sequence length and batch size.
• Try the calculators linked above for a second opinion.
• If close to limits, plan quantization or offloading.
• Consider server frameworks like Ollama or vLLM for production use.

FAQ

Can I run a 7B model on a 6GB GPU?

Maybe, with aggressive 4-bit quantization and small sequence lengths. Use a calculator and test. Many hobbyist setups run 7B models on 6–8 GB with Q4 and optimized runtimes.

Why do different calculators give different answers?

Calculators use different overhead assumptions for drivers, kernels, and KV cache. Use a few tools and add a safety margin of ~1–2 GB when planning.

Where to go next

If you want a fast test, pick one model, run the Hugging Face calculator, then try a local run with Ollama or a lightweight server. Testing on real hardware is the final check.

Want a one-page cheat sheet for popular models and precisions? Download the reference table from the GitHub tool or re-run the calculators linked above to build one for your setup.