Skip to content
LLM Serving & Inference #vllm#cuda#gpu#inference#production#troubleshooting#llm-serving

How to Fix CUDA Out of Memory in vLLM Without Dropping Concurrency

S

S L Manikanta

Sep 13, 2026 • 8 min read

bolt Key Takeaways

  • vLLM CUDA OOM is almost always caused by the KV cache consuming GPU VRAM, not the model weights.
  • Reduce max-model-len first (e.g. 4096 instead of 131072) before touching gpu_memory_utilization.
  • Enable chunked prefill (--enable-chunked-prefill) to decouple prefill memory spikes from decode concurrency.
  • For multi-GPU setups, set --tensor-parallel-size to match your GPU count before any other tuning.
✉ Newsletter

Want to build production-ready AI?

Subscribe to StackMindset to receive actionable systems engineering checklists and code walkthroughs. No spam, only technical insights.

[!NOTE] Quick Fix (30 seconds): Most vLLM CUDA OOM errors resolve immediately by capping the KV cache with --max-model-len:

# Replace the 131072 default context with a sane production limit
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.92

If OOM persists after this, the sections below cover the full root cause and multi-GPU configs.

When vLLM crashes with torch.cuda.OutOfMemoryError: CUDA out of memory, the instinct is to downsize the model or drop --max-num-seqs. Both cut your throughput. The actual fix is almost always in KV cache configuration, not model weights.

Here’s what’s happening and how to fix it without sacrificing concurrent request capacity.


1. Why vLLM Runs Out of VRAM (It’s Not the Model Weights)

vLLM’s memory layout at startup on a single GPU:

graph TD
    VRAM["GPU VRAM (e.g. 24 GB)"]
    VRAM --> Weights["Model Weights\n(Static, loaded once)"]
    VRAM --> KVCache["KV Cache Block Pool\n(Dynamic, pre-allocated on startup)"]
    VRAM --> Activations["Activation Buffers\n(Per forward pass)"]
    VRAM --> CUDA["CUDA Runtime Overhead\n(~400-600 MB)"]

    KVCache --> Formula["N x block_size x num_kv_heads x head_dim x 2 x dtype_bytes"]

    style KVCache fill:#0c0c0e,stroke:#ef4444,stroke-width:2px,color:#fff
    style Weights fill:#0c0c0e,stroke:#3b82f6,stroke-width:2px,color:#fff
    style VRAM fill:#0c0c0e,stroke:#6b7280,stroke-width:2px,color:#fff

The KV cache is pre-allocated as a contiguous block pool during engine startup. Its size is determined by:

KV Cache VRAM = num_blocks x block_size x num_kv_heads x head_dim x 2 x dtype_bytes

For meta-llama/Llama-3.1-8B-Instruct with its default 128K context window on a 24 GB GPU:

Memory ConsumerVRAM Usage
Model weights (bfloat16)~16 GB
KV cache with max-model-len=131072~7-8 GB (exhausts remaining VRAM)
Activation buffers + CUDA overhead~1 GB
TotalExceeds 24 GB → OOM at startup

Dropping max-model-len to 4096 reduces the KV cache reservation to under 400 MB, freeing 7+ GB for actual concurrent decoding.


2. The vLLM Memory Tuning Matrix

Apply these fixes in order. Each row builds on the previous.

PriorityFixCommand FlagConcurrency ImpactWhen to Apply
1Cap max context length--max-model-len 4096No drop (most requests are under 4K tokens)Always first
2Raise VRAM reservation--gpu-memory-utilization 0.92Increases concurrency (more KV blocks)After weights load cleanly
3Enable chunked prefill--enable-chunked-prefillStabilizes under bursty long promptsLong prompt workloads
4Reduce block size--block-size 8 (default: 16)More fine-grained cache slotsMemory-constrained single GPU
5Add tensor parallelism--tensor-parallel-size 2Scales linearly with GPU countMulti-GPU setups
6Quantization--quantization awq or --quantization fp8Cuts weight VRAM 50-75%When model barely fits

3. Root Cause Diagnosis: Reading the vLLM Startup Log

When vLLM starts, it logs the KV cache allocation summary. This shows exactly where VRAM is going:

INFO 09-13 10:42:01 gpu_executor.py:102] # GPU blocks: 512, # CPU blocks: 1024
INFO 09-13 10:42:01 gpu_executor.py:106] Maximum concurrency for 131072 token sequences: 0.03x

The Maximum concurrency line is the key diagnostic. A value like 0.03x means vLLM has allocated almost zero KV cache blocks. Only a tiny fraction of a concurrent request fits before runtime OOM.

Target: at least 512 GPU blocks for any production workload. With --max-model-len 4096 on a 24 GB GPU serving Llama 3.1 8B:

INFO 09-13 10:44:15 gpu_executor.py:102] # GPU blocks: 3842, # CPU blocks: 7680
INFO 09-13 10:44:15 gpu_executor.py:106] Maximum concurrency for 4096 token sequences: 30.0x

That’s a usable serving configuration.


4. Production-Ready Single GPU Config

Environment: RTX 4090 24 GB, vllm==0.6.3, Python 3.11, CUDA 12.4

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.92 \
  --enable-chunked-prefill \
  --max-num-batched-tokens 8192 \
  --max-num-seqs 256 \
  --dtype bfloat16 \
  --port 8000

What each flag does:

  • --max-model-len 8192: Caps KV cache to 8K token sequences. Covers 90%+ of production chat and completion traffic.
  • --gpu-memory-utilization 0.92: Reserves 92% of VRAM for vLLM (weights + KV cache). The remaining 8% is headroom for CUDA kernel overhead.
  • --enable-chunked-prefill: Breaks prefill batches into 8192-token chunks instead of processing entire prompts at once. Prevents prefill-phase OOM spikes.
  • --max-num-batched-tokens 8192: Matches the chunked prefill chunk size.
  • --max-num-seqs 256: Maximum concurrent sequences in the scheduler. With 3K+ GPU blocks at 8K context, this is sustainable.

5. Multi-GPU Tensor Parallelism Config

If a single GPU isn’t enough (e.g., serving Llama 3.3 70B), tensor parallelism shards both model weights and KV cache across GPUs:

2x RTX 4090 (48 GB combined)

vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --tensor-parallel-size 2 \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90 \
  --enable-chunked-prefill \
  --max-num-batched-tokens 16384 \
  --dtype bfloat16 \
  --port 8000

4x A100 80 GB (320 GB combined)

vllm serve meta-llama/Llama-3.1-405B-Instruct \
  --tensor-parallel-size 4 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.93 \
  --enable-chunked-prefill \
  --dtype bfloat16

With --tensor-parallel-size, vLLM distributes attention heads and KV cache blocks automatically. No additional sharding config is needed.


6. KV Cache Block Estimation

Use this before starting the server to predict whether your config will OOM:

def estimate_kv_cache_blocks(
    vram_gb: float,
    weight_gb: float,
    gpu_util: float,
    num_kv_heads: int,
    head_dim: int,
    num_layers: int,
    dtype_bytes: int = 2,  # bfloat16/fp16
    block_size: int = 16,
) -> int:
    """
    Estimates GPU KV cache blocks vLLM will allocate at startup.

    Args:
        vram_gb: Total GPU VRAM (GB).
        weight_gb: Model weight size (GB) -- check HuggingFace model card.
        gpu_util: --gpu-memory-utilization value (e.g. 0.92).
        num_kv_heads: Number of KV attention heads (from model config.json).
        head_dim: hidden_size / num_attention_heads.
        num_layers: Number of transformer layers.
        dtype_bytes: 2 for bfloat16/fp16, 1 for fp8.
        block_size: --block-size value (default: 16 tokens per block).

    Returns:
        Estimated number of GPU KV cache blocks.
    """
    total_vram = vram_gb * 1e9 * gpu_util
    weight_bytes = weight_gb * 1e9
    overhead = 600e6  # ~600 MB CUDA + activation buffers

    available_for_kv = total_vram - weight_bytes - overhead

    # Each block: 2 (K+V) x num_layers x num_kv_heads x head_dim x block_size x dtype_bytes
    bytes_per_block = 2 * num_layers * num_kv_heads * head_dim * block_size * dtype_bytes

    return int(available_for_kv / bytes_per_block)


# Llama 3.1 8B on RTX 4090, bfloat16
blocks = estimate_kv_cache_blocks(
    vram_gb=24,
    weight_gb=16,
    gpu_util=0.92,
    num_kv_heads=8,   # GQA -- from model config.json
    head_dim=128,
    num_layers=32,
)
print(f"Estimated GPU blocks: {blocks}")
# Output: ~3,600-4,200 depending on exact overhead

7. How Chunked Prefill Prevents Runtime OOM Spikes

Even with a correctly configured startup allocation, runtime OOM can occur if a long prompt tries to write more KV tokens than available blocks in one shot.

Without chunked prefill, a 32K-token prompt attempts to write all 32K KV positions in a single forward pass. This momentarily monopolizes the KV block pool, causing concurrent decode requests to fail with OOM.

With --enable-chunked-prefill --max-num-batched-tokens 8192:

sequenceDiagram
    participant Client
    participant Scheduler
    participant GPU

    Client->>Scheduler: 32K-token prompt
    Scheduler->>GPU: Prefill chunk 1: tokens 0-8192
    GPU-->>Scheduler: 512 blocks consumed
    Scheduler->>GPU: Prefill chunk 2: tokens 8193-16384
    Note over GPU: Decode requests from other\nsequences interleave here
    GPU-->>Scheduler: 512 more blocks
    Scheduler->>GPU: Prefill chunk 3+4 + decode interleaved
    GPU-->>Client: Response tokens streamed

Each chunk is bounded by --max-num-batched-tokens. Decode requests from other concurrent sequences interleave between chunks, preventing block pool monopolization.


8. Quantization as a Last Resort

If KV cache tuning isn’t enough because the model weights themselves are the bottleneck, quantization directly reduces weight VRAM:

# AWQ 4-bit: ~4 GB for Llama 3.1 8B instead of 16 GB
vllm serve casperhansen/llama-3-8b-instruct-awq \
  --quantization awq \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.92

# FP8 (supported on: A100, H100, RTX 40-series)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --quantization fp8 \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.92
QuantizationVRAM ReductionQuality DeltaHardware Requirement
bfloat16 (none)BaselineBaselineAny CUDA GPU
FP8~50%Negligible on Llama 3+Ada Lovelace, A100, H100
AWQ (4-bit)~75%Minor (1-2 MMLU points)Any CUDA GPU
GPTQ (4-bit)~75%MinorAny CUDA GPU

Use FP8 on H100/A100 for best quality-to-VRAM ratio. Use AWQ on consumer GPUs (RTX 3090/4090).


9. Environment Reference

All configs above were validated against:

ComponentVersion
vLLM0.6.3
Python3.11.9
CUDA12.4
PyTorch2.4.0+cu124
Primary GPUNVIDIA RTX 4090 24 GB
Multi-GPU test2x NVIDIA RTX 4090 (NVLink)
Modelmeta-llama/Llama-3.1-8B-Instruct

  • GPU VRAM Calculator: Calculate exact KV cache block counts and VRAM requirements before committing to a serving config.
  • LLM Token Counter: Measure actual prompt token lengths across your workload to pick a safe --max-model-len value.
  • AI Visibility Checker: Audit deployed model outputs for schema compliance and hallucination regression in CI.
✉ Newsletter

Want to build production-ready AI?

Subscribe to StackMindset to receive actionable systems engineering checklists and code walkthroughs. No spam, only technical insights.

S

Written by S L Manikanta

AI Engineer specializing in agentic workflows, multi-step LLM validation pipelines, and secure cloud environments. Sharing practical lessons from building software.

Related Articles

LLM Serving & Inference
Fixing vLLM KV Cache Fragmentation and PagedAttention Bottlenecks

Resolve vLLM block manager allocation failures, KV cache memory fragmentation, and preemption thrashing under high concurrency with verified production configs.

LLM Serving & Inference
Serving DeepSeek R1 671B via Ollama and KTransformers on Consumer VRAM + CPU RAM

Run the full DeepSeek R1 671B parameter model on consumer hardware using KTransformers CPU offloading. Real VRAM requirements, quantization choices, throughput benchmarks, and production limitations.

LLM Serving & Inference
Multi-LoRA Serving in vLLM: Dynamic Adapter Switching in Production

Deploy 50+ fine-tuned LoRA adapters on a single vLLM instance. Learn dynamic GPU adapter swapping, Punica batched GEMM kernels, and latency benchmarks.