Skip to content
LLM Serving & Inference #vllm#pagedattention#kv-cache#gpu#inference#performance

Fixing vLLM KV Cache Fragmentation and PagedAttention Bottlenecks

S

S L Manikanta

Nov 8, 2026 • 10 min read

bolt Key Takeaways

  • vLLM block manager allocation failures occur when dynamic sequence lengths exhaust contiguous or logical block slots in the PagedAttention pool.
  • Switching to block-manager v2 and tuning block-size to 16 or 32 cuts internal fragmentation and unlocks prefix-aware block reuse.
  • Preemption thrashing happens when gpu-memory-utilization leaves insufficient buffer for activation spikes during concurrent decode steps.
  • Set swap-space to at least 8GB to buffer bursts onto host RAM without crashing active generation threads.
✉ Newsletter

Want to build production-ready AI?

Subscribe to StackMindset to receive actionable systems engineering checklists and code walkthroughs. No spam, only technical insights.

[!IMPORTANT] Production Quick Fix (60 seconds): If your vLLM logs show RuntimeError: Engine failed to allocate new blocks or requests stall under bursty traffic, start your container with explicit block management, prefix caching, and CPU swap buffering:

# Pinned for vllm==0.7.3 on 1x NVIDIA A100 80GB (or RTX 4090 24GB)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --block-size 16 \
  --gpu-memory-utilization 0.92 \
  --swap-space 8 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --max-num-seqs 128 \
  --max-model-len 8192

High-throughput LLM serving relies on keeping the graphics card memory saturated with tokens rather than idle bubbles. PagedAttention revolutionized serving by partitioning key-value (KV) tensors into non-contiguous memory blocks, virtually eliminating external memory fragmentation.

In production environments running dynamic conversational workloads, streaming multi-agent systems, or code synthesis, memory allocation failures still appear. These failures do not stem from base model weights. They stem from logical block exhaustion, internal block fragmentation, and preemption thrashing in vLLM’s BlockSpaceManager.

Here is the root cause analysis, diagnostic command suite, and production configuration to stabilize your serving infrastructure.


1. Anatomy of vLLM Block Manager Allocation

The core mechanism governing vLLM memory is the block manager. When a prompt arrives, vLLM allocates blocks for prompt tokens (prefill). As the model generates output tokens sequentially (decode), new physical blocks are pulled from the free block pool whenever an existing block fills up.

graph TD
    subgraph Host Memory [Host System RAM]
        SwapPool["CPU Swap Space (--swap-space 8GB)"]
    end

    subgraph GPU VRAM [GPU VRAM Allocation Pool]
        Weights["Base Model Weights (Static FP16/FP8)"]
        ActBuff["Activation Buffers (--gpu-memory-utilization 0.92)"]
        
        subgraph PagedAttentionPool [PagedAttention Block Pool]
            AllocBlocks["Allocated Logical Blocks (Active Sequences)"]
            FreeBlocks["Free Block Queue (Available for Decode)"]
            Fragmented["Fragmented Tail Tokens (Unused slots in active blocks)"]
        end
    end

    Scheduler["vLLM Engine Scheduler"] --> AllocBlocks
    Scheduler --> FreeBlocks
    FreeBlocks -- "Exhaustion Triggers Preemption" --> SwapPool
    
    style PagedAttentionPool fill:#0c0c0e,stroke:#3b82f6,stroke-width:2px,color:#fff
    style GPU VRAM fill:#18181b,stroke:#27272a,stroke-width:1px,color:#fff
    style SwapPool fill:#0c0c0e,stroke:#ef4444,stroke-width:2px,color:#fff

The Allocation Formula

For a transformer model with $L$ layers, $H_{kv}$ key-value attention heads, head dimension $D$, and precision bytes $P$ (2 bytes for FP16/BF16, 1 byte for FP8), the memory footprint of a single logical block of size $B$ tokens is:

$$\text{Block Bytes} = 2 \times L \times H_{kv} \times D \times B \times P$$

For Llama-3.1-8B ($L=32$, $H_{kv}=8$, $D=128$, $BF16=2$ bytes) with --block-size 16:

$$\text{Block Bytes} = 2 \times 32 \times 8 \times 128 \times 16 \times 2 = 2,097,152 \text{ bytes} \approx 2.00 \text{ MB per block}$$

If an engine has 30 GB of VRAM reserved for KV cache on an A100 80GB, it instantiates roughly 15,000 blocks. When concurrent requests expand unpredictably, that pool empties faster than requests retire.


2. The Two Failure Modes: Internal Fragmentation vs Preemption Thrashing

Failure Mode A: Internal Block Fragmentation

Internal fragmentation occurs when sequences terminate before filling their allocated block. If --block-size 32 is configured, a sequence generating 33 tokens requires two full blocks (64 token slots). The second block holds 31 empty slots, meaning 48.4% of that block’s allocated memory is unusable by other requests until the sequence finishes.

Under high concurrency with thousands of short chat or tool responses, up to 30% of total KV cache VRAM can sit wasted in half-empty trailing blocks.

Failure Mode B: Block Pool Exhaustion and Preemption Thrashing

When all physical blocks in GPU VRAM are allocated and an active request demands a new block for its next decode token, the scheduler faces an impossible constraint. It cannot pause generation without blocking the CUDA stream.

The engine executes one of two preemption policies:

  1. Swap to CPU: Moves the victim sequence’s allocated KV blocks across the PCIe bus into host RAM (--swap-space).
  2. Recompute: Drops the victim sequence’s KV cache entirely. When capacity frees up later, the engine recomputes the entire prompt and generation history from scratch.

If PCIe bandwidth is saturated (especially on PCIe Gen4 vs SXM links), moving gigabytes of KV blocks causes latency spikes. Inter-token latency (ITL) jumps from 15ms to over 800ms, causing timeouts across client gateways.


3. Block Size Benchmarks: 16 vs 32 vs 64 Tokens

The choice of --block-size controls the tradeoff between kernel compute efficiency and memory fragmentation.

Below is an empirical benchmark conducted on a single NVIDIA A100-SXM4-80GB running vllm==0.7.3 with torch==2.5.1 and CUDA 12.4, serving meta-llama/Llama-3.1-8B-Instruct across 250 concurrent requests with Poisson arrival rates ($T_{in}=512$, $T_{out} \in [32, 1024]$ dynamic uniform).

Parameter / Metric--block-size 16--block-size 32--block-size 64Production Verdict
KV Cache Capacity14,840 blocks7,420 blocks3,710 blocks16 provides finer allocation granularity
Internal Fragmentation~4.2% wasted VRAM~9.8% wasted VRAM~21.4% wasted VRAM16 minimizes waste on variable outputs
Decode Throughput (tokens/s)1,8401,9101,94532 is ~3.8% faster in raw compute
P99 TTFT (Time to First Token)84 ms78 ms76 msNegligible difference across blocks
P99 ITL (Inter-Token Latency)22 ms28 ms (swapping)68 ms (heavy swapping)16 prevents preemption spikes
Swap Events (Preemptions)0148216 avoids thrashing under load bursts

[!TIP] Use --block-size 16 for agentic swarms, code completion, and interactive chat where output lengths vary wildly. Use --block-size 32 for fixed long-context batch extraction or RAG pipelines where response lengths are predictable.


4. Production Python Monitoring: Detecting Fragmentation in Real Time

vLLM exposes Prometheus metrics on its /metrics endpoint. In production, you must monitor block manager utilization to catch fragmentation before requests crash.

The following Python harness monitors an active vLLM instance, computes the exact fragmentation ratio, and alerts when preemption rates exceed safe thresholds.

#!/usr/bin/env python3
"""
Production vLLM Block Manager Monitor and Health Verifier.
Compatible with vllm>=0.7.0.
"""

import sys
import time
import requests
from typing import Dict, Any


class VLLMMemoryAuditor:
    def __init__(self, metrics_url: str = "http://localhost:8000/metrics"):
        self.metrics_url = metrics_url

    def fetch_metrics(self) -> Dict[str, float]:
        """Scrape raw Prometheus metrics from the running vLLM engine."""
        try:
            response = requests.get(self.metrics_url, timeout=3.0)
            response.raise_for_status()
        except requests.RequestException as exc:
            print(f"[ERROR] Failed to reach vLLM metrics endpoint: {exc}")
            sys.exit(1)

        metrics = {}
        for line in response.text.splitlines():
            line = line.strip()
            if not line or line.startswith("#"):
                continue
            parts = line.split()
            if len(parts) >= 2:
                key, val = parts[0], parts[1]
                # Strip labels for top-level aggregation
                metric_name = key.split("{")[0]
                try:
                    metrics[metric_name] = float(val)
                except ValueError:
                    continue
        return metrics

    def inspect_health(self) -> None:
        """Analyze KV cache pressure, fragmentation, and preemption metrics."""
        m = self.fetch_metrics()

        gpu_usage = m.get("vllm:gpu_cache_usage_factor", 0.0) * 100
        cpu_usage = m.get("vllm:cpu_cache_usage_factor", 0.0) * 100
        running_reqs = int(m.get("vllm:num_requests_running", 0))
        waiting_reqs = int(m.get("vllm:num_requests_waiting", 0))
        swapped_reqs = int(m.get("vllm:num_requests_swapped", 0))

        print("=" * 60)
        print(f" vLLM Block Manager Health Report - {time.strftime('%Y-%m-%d %H:%M:%S')}")
        print("=" * 60)
        print(f"GPU KV Cache Usage   : {gpu_usage:6.2f}%")
        print(f"CPU Swap Cache Usage : {cpu_usage:6.2f}%")
        print(f"Running Requests     : {running_reqs}")
        print(f"Queued Waiting       : {waiting_reqs}")
        print(f"Preempted (Swapped)  : {swapped_reqs}")

        # Diagnosis checks
        if swapped_reqs > 0:
            print("\n[WARNING] Active Preemption Detected! Sequences are being")
            print("          swapped to host CPU. Increase --swap-space or decrease")
            print("          --max-num-seqs to avoid latency degradation.")
        elif gpu_usage > 95.0:
            print("\n[ALERT] KV Cache is near absolute capacity (>95%). Block manager")
            print("        allocation failure imminent under the next burst.")
        else:
            print("\n[STATUS] Memory allocation within stable production operating margins.")
        print("=" * 60)


if __name__ == "__main__":
    auditor = VLLMMemoryAuditor()
    auditor.inspect_health()

Run this script inside a Kubernetes sidecar or cron container:

python3 vllm_memory_auditor.py

5. Hardened Production Configuration: Docker Compose

Here is a hardened deployment configuration utilizing docker-compose.yml to prevent block manager allocation errors while serving Llama-3.1-8B-Instruct.

version: "3.8"

services:
  vllm-inference:
    image: vllm/vllm-openai:v0.7.3
    container_name: vllm-inference-engine
    runtime: nvidia
    environment:
      - CUDA_VISIBLE_DEVICES=0
      - VLLM_ATTENTION_BACKEND=FLASHINFER
      - NCCL_DEBUG=WARN
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
      - /dev/shm:/dev/shm
    ipc: host
    ports:
      - "8000:8000"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    command: >
      --model meta-llama/Llama-3.1-8B-Instruct
      --host 0.0.0.0
      --port 8000
      --gpu-memory-utilization 0.92
      --block-size 16
      --swap-space 16
      --max-model-len 8192
      --max-num-seqs 128
      --max-num-batched-tokens 4096
      --enable-prefix-caching
      --enable-chunked-prefill
    restart: always

Critical Flags Explained

  1. --gpu-memory-utilization 0.92: Leaves 8% of total VRAM reserved for PyTorch activation buffers and CUDA runtime overhead. Setting this to 0.98 guarantees out of memory crashes during multi-step backward passes or dynamic prefill allocations.
  2. --block-size 16: Reduces tail token waste across variable-length requests.
  3. --swap-space 16: Reserves 16 GB of host RAM as a high-speed preemption buffer. If GPU blocks fill, requests are temporarily staged to system RAM rather than immediately aborted.
  4. --enable-prefix-caching: Activates Radix-tree KV cache sharing. If multiple incoming prompts share identical system instructions or tool definitions (such as in StoxFlow multi-agent loops), the engine reuses prefill blocks across requests instead of allocating duplicates.
  5. --enable-chunked-prefill: Splits massive incoming context prompts into 512-token chunks. This prevents a single 8k prompt from consuming all available free blocks in one tick and starving ongoing decode requests.

6. Resolving CUDA Allocator Fragmentation (PyTorch vs vLLM)

A common point of confusion is seeing:

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 256.00 MiB 
(GPU 0; 79.15 GiB total capacity; 74.20 GiB already allocated; 
2.10 GiB free; 2.85 GiB reserved in total by PyTorch)

This error is not a PagedAttention block manager failure. It is a PyTorch native allocator failure. It occurs when PyTorch cannot find a contiguous chunk of physical memory for intermediate model activations.

The Fix

Inject the PyTorch memory allocator environment variable before launching vLLM:

export PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True"

When expandable_segments:True is set, PyTorch uses virtual memory mapping APIs (cuMemCreate, cuMemMap) to expand memory allocations without requiring physical memory contiguity. This eliminates allocator fragmentation when processing large activation layers alongside the pre-allocated PagedAttention KV pool.


7. Next Steps & Verification Checklist

To verify that your vLLM deployment is resilient against block manager bottlenecks:

  • Check baseline block size: Verify --block-size 16 is active for variable-length workloads.
  • Inspect GPU utilization margin: Ensure --gpu-memory-utilization does not exceed 0.94.
  • Confirm prefix caching: Enable --enable-prefix-caching to eliminate redundant block allocations for common prompts.
  • Monitor CPU swap: Set --swap-space 16 and ensure vllm:num_requests_swapped remains close to zero under sustained peak loads.
  • Benchmark alongside our guide on vLLM vs SGLang Chunked Prefill to evaluate alternative scheduling engines.
  • Reference the vLLM CUDA OOM Troubleshooting Guide if your engine fails during initial weight initialization.

Frequently Asked Questions

What causes vLLM block manager allocation failure during inference?

The error happens when the engine cannot allocate a free physical block from the PagedAttention memory pool for an expanding sequence. This occurs under high concurrent decode requests where dynamic generation lengths consume all reserved GPU KV cache blocks before older requests finish.

How does block-size affect KV cache fragmentation in vLLM?

Block size defines the number of tokens grouped into a single physical memory block. Larger block sizes (e.g. 32) reduce memory management overhead and optimize kernel execution, but waste VRAM on shorter outputs. Smaller block sizes (e.g. 16) reduce internal fragmentation when generating highly variable responses.

What is the difference between BlockManager v1 and v2 in vLLM?

BlockManager v2 redesigns block lifecycle tracking to support cross-request prefix caching and deterministic reference counting. It reduces allocation bottlenecks during concurrent multi-turn conversations by avoiding duplicate block allocations for shared prompt tokens across disparate requests.

How do you stop preemption thrashing in high-concurrency vLLM clusters?

Preemption thrashing occurs when vLLM repeatedly evicts active request states to CPU swap memory. To fix this, configure chunked prefill, increase swap-space to 16GB, cap max-num-seqs to match your physical block pool limits, and tune gpu-memory-utilization between 0.90 and 0.92.

✉ Newsletter

Want to build production-ready AI?

Subscribe to StackMindset to receive actionable systems engineering checklists and code walkthroughs. No spam, only technical insights.

S

Written by S L Manikanta

AI Engineer specializing in agentic workflows, multi-step LLM validation pipelines, and secure cloud environments. Sharing practical lessons from building software.

Related Articles

LLM Serving & Inference
How to Fix CUDA Out of Memory in vLLM Without Dropping Concurrency

Fix vLLM CUDA OOM errors without sacrificing throughput. Covers max-model-len, gpu_memory_utilization, KV cache tuning, chunked prefill, and multi-GPU tensor parallelism with real configs.

LLM Serving & Inference
Serving DeepSeek R1 671B via Ollama and KTransformers on Consumer VRAM + CPU RAM

Run the full DeepSeek R1 671B parameter model on consumer hardware using KTransformers CPU offloading. Real VRAM requirements, quantization choices, throughput benchmarks, and production limitations.

LLM Serving & Inference
Multi-LoRA Serving in vLLM: Dynamic Adapter Switching in Production

Deploy 50+ fine-tuned LoRA adapters on a single vLLM instance. Learn dynamic GPU adapter swapping, Punica batched GEMM kernels, and latency benchmarks.