How to Fix CUDA Out of Memory in vLLM Without Dropping Concurrency
S L Manikanta
Sep 13, 2026 • 8 min read
bolt Key Takeaways
- vLLM CUDA OOM is almost always caused by the KV cache consuming GPU VRAM, not the model weights.
- Reduce max-model-len first (e.g. 4096 instead of 131072) before touching gpu_memory_utilization.
- Enable chunked prefill (--enable-chunked-prefill) to decouple prefill memory spikes from decode concurrency.
- For multi-GPU setups, set --tensor-parallel-size to match your GPU count before any other tuning.
list On this page expand_more
- 1. Why vLLM Runs Out of VRAM (It’s Not the Model Weights)
- 2. The vLLM Memory Tuning Matrix
- 3. Root Cause Diagnosis: Reading the vLLM Startup Log
- 4. Production-Ready Single GPU Config
- 5. Multi-GPU Tensor Parallelism Config
- 6. KV Cache Block Estimation
- 7. How Chunked Prefill Prevents Runtime OOM Spikes
- 8. Quantization as a Last Resort
- 9. Environment Reference
- Related Tools & Deep-Dives
Want to build production-ready AI?
Subscribe to StackMindset to receive actionable systems engineering checklists and code walkthroughs. No spam, only technical insights.
[!NOTE] Quick Fix (30 seconds): Most vLLM CUDA OOM errors resolve immediately by capping the KV cache with
--max-model-len:# Replace the 131072 default context with a sane production limit vllm serve meta-llama/Llama-3.1-8B-Instruct \ --max-model-len 4096 \ --gpu-memory-utilization 0.92If OOM persists after this, the sections below cover the full root cause and multi-GPU configs.
When vLLM crashes with torch.cuda.OutOfMemoryError: CUDA out of memory, the instinct is to downsize the model or drop --max-num-seqs. Both cut your throughput. The actual fix is almost always in KV cache configuration, not model weights.
Here’s what’s happening and how to fix it without sacrificing concurrent request capacity.
1. Why vLLM Runs Out of VRAM (It’s Not the Model Weights)
vLLM’s memory layout at startup on a single GPU:
graph TD
VRAM["GPU VRAM (e.g. 24 GB)"]
VRAM --> Weights["Model Weights\n(Static, loaded once)"]
VRAM --> KVCache["KV Cache Block Pool\n(Dynamic, pre-allocated on startup)"]
VRAM --> Activations["Activation Buffers\n(Per forward pass)"]
VRAM --> CUDA["CUDA Runtime Overhead\n(~400-600 MB)"]
KVCache --> Formula["N x block_size x num_kv_heads x head_dim x 2 x dtype_bytes"]
style KVCache fill:#0c0c0e,stroke:#ef4444,stroke-width:2px,color:#fff
style Weights fill:#0c0c0e,stroke:#3b82f6,stroke-width:2px,color:#fff
style VRAM fill:#0c0c0e,stroke:#6b7280,stroke-width:2px,color:#fff
The KV cache is pre-allocated as a contiguous block pool during engine startup. Its size is determined by:
KV Cache VRAM = num_blocks x block_size x num_kv_heads x head_dim x 2 x dtype_bytes
For meta-llama/Llama-3.1-8B-Instruct with its default 128K context window on a 24 GB GPU:
| Memory Consumer | VRAM Usage |
|---|---|
| Model weights (bfloat16) | ~16 GB |
KV cache with max-model-len=131072 | ~7-8 GB (exhausts remaining VRAM) |
| Activation buffers + CUDA overhead | ~1 GB |
| Total | Exceeds 24 GB → OOM at startup |
Dropping max-model-len to 4096 reduces the KV cache reservation to under 400 MB, freeing 7+ GB for actual concurrent decoding.
2. The vLLM Memory Tuning Matrix
Apply these fixes in order. Each row builds on the previous.
| Priority | Fix | Command Flag | Concurrency Impact | When to Apply |
|---|---|---|---|---|
| 1 | Cap max context length | --max-model-len 4096 | No drop (most requests are under 4K tokens) | Always first |
| 2 | Raise VRAM reservation | --gpu-memory-utilization 0.92 | Increases concurrency (more KV blocks) | After weights load cleanly |
| 3 | Enable chunked prefill | --enable-chunked-prefill | Stabilizes under bursty long prompts | Long prompt workloads |
| 4 | Reduce block size | --block-size 8 (default: 16) | More fine-grained cache slots | Memory-constrained single GPU |
| 5 | Add tensor parallelism | --tensor-parallel-size 2 | Scales linearly with GPU count | Multi-GPU setups |
| 6 | Quantization | --quantization awq or --quantization fp8 | Cuts weight VRAM 50-75% | When model barely fits |
3. Root Cause Diagnosis: Reading the vLLM Startup Log
When vLLM starts, it logs the KV cache allocation summary. This shows exactly where VRAM is going:
INFO 09-13 10:42:01 gpu_executor.py:102] # GPU blocks: 512, # CPU blocks: 1024
INFO 09-13 10:42:01 gpu_executor.py:106] Maximum concurrency for 131072 token sequences: 0.03x
The Maximum concurrency line is the key diagnostic. A value like 0.03x means vLLM has allocated almost zero KV cache blocks. Only a tiny fraction of a concurrent request fits before runtime OOM.
Target: at least 512 GPU blocks for any production workload. With --max-model-len 4096 on a 24 GB GPU serving Llama 3.1 8B:
INFO 09-13 10:44:15 gpu_executor.py:102] # GPU blocks: 3842, # CPU blocks: 7680
INFO 09-13 10:44:15 gpu_executor.py:106] Maximum concurrency for 4096 token sequences: 30.0x
That’s a usable serving configuration.
4. Production-Ready Single GPU Config
Environment: RTX 4090 24 GB, vllm==0.6.3, Python 3.11, CUDA 12.4
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--max-model-len 8192 \
--gpu-memory-utilization 0.92 \
--enable-chunked-prefill \
--max-num-batched-tokens 8192 \
--max-num-seqs 256 \
--dtype bfloat16 \
--port 8000
What each flag does:
--max-model-len 8192: Caps KV cache to 8K token sequences. Covers 90%+ of production chat and completion traffic.--gpu-memory-utilization 0.92: Reserves 92% of VRAM for vLLM (weights + KV cache). The remaining 8% is headroom for CUDA kernel overhead.--enable-chunked-prefill: Breaks prefill batches into 8192-token chunks instead of processing entire prompts at once. Prevents prefill-phase OOM spikes.--max-num-batched-tokens 8192: Matches the chunked prefill chunk size.--max-num-seqs 256: Maximum concurrent sequences in the scheduler. With 3K+ GPU blocks at 8K context, this is sustainable.
5. Multi-GPU Tensor Parallelism Config
If a single GPU isn’t enough (e.g., serving Llama 3.3 70B), tensor parallelism shards both model weights and KV cache across GPUs:
2x RTX 4090 (48 GB combined)
vllm serve meta-llama/Llama-3.3-70B-Instruct \
--tensor-parallel-size 2 \
--max-model-len 16384 \
--gpu-memory-utilization 0.90 \
--enable-chunked-prefill \
--max-num-batched-tokens 16384 \
--dtype bfloat16 \
--port 8000
4x A100 80 GB (320 GB combined)
vllm serve meta-llama/Llama-3.1-405B-Instruct \
--tensor-parallel-size 4 \
--max-model-len 32768 \
--gpu-memory-utilization 0.93 \
--enable-chunked-prefill \
--dtype bfloat16
With --tensor-parallel-size, vLLM distributes attention heads and KV cache blocks automatically. No additional sharding config is needed.
6. KV Cache Block Estimation
Use this before starting the server to predict whether your config will OOM:
def estimate_kv_cache_blocks(
vram_gb: float,
weight_gb: float,
gpu_util: float,
num_kv_heads: int,
head_dim: int,
num_layers: int,
dtype_bytes: int = 2, # bfloat16/fp16
block_size: int = 16,
) -> int:
"""
Estimates GPU KV cache blocks vLLM will allocate at startup.
Args:
vram_gb: Total GPU VRAM (GB).
weight_gb: Model weight size (GB) -- check HuggingFace model card.
gpu_util: --gpu-memory-utilization value (e.g. 0.92).
num_kv_heads: Number of KV attention heads (from model config.json).
head_dim: hidden_size / num_attention_heads.
num_layers: Number of transformer layers.
dtype_bytes: 2 for bfloat16/fp16, 1 for fp8.
block_size: --block-size value (default: 16 tokens per block).
Returns:
Estimated number of GPU KV cache blocks.
"""
total_vram = vram_gb * 1e9 * gpu_util
weight_bytes = weight_gb * 1e9
overhead = 600e6 # ~600 MB CUDA + activation buffers
available_for_kv = total_vram - weight_bytes - overhead
# Each block: 2 (K+V) x num_layers x num_kv_heads x head_dim x block_size x dtype_bytes
bytes_per_block = 2 * num_layers * num_kv_heads * head_dim * block_size * dtype_bytes
return int(available_for_kv / bytes_per_block)
# Llama 3.1 8B on RTX 4090, bfloat16
blocks = estimate_kv_cache_blocks(
vram_gb=24,
weight_gb=16,
gpu_util=0.92,
num_kv_heads=8, # GQA -- from model config.json
head_dim=128,
num_layers=32,
)
print(f"Estimated GPU blocks: {blocks}")
# Output: ~3,600-4,200 depending on exact overhead
7. How Chunked Prefill Prevents Runtime OOM Spikes
Even with a correctly configured startup allocation, runtime OOM can occur if a long prompt tries to write more KV tokens than available blocks in one shot.
Without chunked prefill, a 32K-token prompt attempts to write all 32K KV positions in a single forward pass. This momentarily monopolizes the KV block pool, causing concurrent decode requests to fail with OOM.
With --enable-chunked-prefill --max-num-batched-tokens 8192:
sequenceDiagram
participant Client
participant Scheduler
participant GPU
Client->>Scheduler: 32K-token prompt
Scheduler->>GPU: Prefill chunk 1: tokens 0-8192
GPU-->>Scheduler: 512 blocks consumed
Scheduler->>GPU: Prefill chunk 2: tokens 8193-16384
Note over GPU: Decode requests from other\nsequences interleave here
GPU-->>Scheduler: 512 more blocks
Scheduler->>GPU: Prefill chunk 3+4 + decode interleaved
GPU-->>Client: Response tokens streamed
Each chunk is bounded by --max-num-batched-tokens. Decode requests from other concurrent sequences interleave between chunks, preventing block pool monopolization.
8. Quantization as a Last Resort
If KV cache tuning isn’t enough because the model weights themselves are the bottleneck, quantization directly reduces weight VRAM:
# AWQ 4-bit: ~4 GB for Llama 3.1 8B instead of 16 GB
vllm serve casperhansen/llama-3-8b-instruct-awq \
--quantization awq \
--max-model-len 8192 \
--gpu-memory-utilization 0.92
# FP8 (supported on: A100, H100, RTX 40-series)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--quantization fp8 \
--max-model-len 16384 \
--gpu-memory-utilization 0.92
| Quantization | VRAM Reduction | Quality Delta | Hardware Requirement |
|---|---|---|---|
| bfloat16 (none) | Baseline | Baseline | Any CUDA GPU |
| FP8 | ~50% | Negligible on Llama 3+ | Ada Lovelace, A100, H100 |
| AWQ (4-bit) | ~75% | Minor (1-2 MMLU points) | Any CUDA GPU |
| GPTQ (4-bit) | ~75% | Minor | Any CUDA GPU |
Use FP8 on H100/A100 for best quality-to-VRAM ratio. Use AWQ on consumer GPUs (RTX 3090/4090).
9. Environment Reference
All configs above were validated against:
| Component | Version |
|---|---|
| vLLM | 0.6.3 |
| Python | 3.11.9 |
| CUDA | 12.4 |
| PyTorch | 2.4.0+cu124 |
| Primary GPU | NVIDIA RTX 4090 24 GB |
| Multi-GPU test | 2x NVIDIA RTX 4090 (NVLink) |
| Model | meta-llama/Llama-3.1-8B-Instruct |
Related Tools & Deep-Dives
- GPU VRAM Calculator: Calculate exact KV cache block counts and VRAM requirements before committing to a serving config.
- LLM Token Counter: Measure actual prompt token lengths across your workload to pick a safe
--max-model-lenvalue. - AI Visibility Checker: Audit deployed model outputs for schema compliance and hallucination regression in CI.
Want to build production-ready AI?
Subscribe to StackMindset to receive actionable systems engineering checklists and code walkthroughs. No spam, only technical insights.
Written by S L Manikanta
AI Engineer specializing in agentic workflows, multi-step LLM validation pipelines, and secure cloud environments. Sharing practical lessons from building software.
Related Articles
Fixing vLLM KV Cache Fragmentation and PagedAttention Bottlenecks
Resolve vLLM block manager allocation failures, KV cache memory fragmentation, and preemption thrashing under high concurrency with verified production configs.
Serving DeepSeek R1 671B via Ollama and KTransformers on Consumer VRAM + CPU RAM
Run the full DeepSeek R1 671B parameter model on consumer hardware using KTransformers CPU offloading. Real VRAM requirements, quantization choices, throughput benchmarks, and production limitations.
Multi-LoRA Serving in vLLM: Dynamic Adapter Switching in Production
Deploy 50+ fine-tuned LoRA adapters on a single vLLM instance. Learn dynamic GPU adapter swapping, Punica batched GEMM kernels, and latency benchmarks.