Executive summary
Production inference for large language models (LLMs) is fundamentally a memory‑scheduling challenge. The model must keep a key‑value (KV) cache for every active request, and each new token depends on the cached keys and values of all prior tokens. Traditional serving systems reserve a contiguous cache slab sized for the maximum possible context length, which leads to wasted memory when requests finish early. The vLLM project introduced PagedAttention, a mechanism that breaks the KV cache into fixed‑size blocks and allocates those blocks only when a token is generated. This on‑demand allocation eliminates most of the unused memory and enables continuous batching, where new requests can join an ongoing batch as soon as free blocks appear. Quantisation methods such as AWQ further reduce the memory needed for model weights, freeing additional capacity for KV cache. Together, paged allocation, quantisation, and continuous batching allow a single GPU to serve many more concurrent sequences without sacrificing latency.
Introduction
Enterprises that expose LLMs as APIs must balance three competing goals: low latency for end‑users, high throughput to maximise hardware utilisation, and predictable cost. Training workloads stress compute and inter‑node bandwidth, while inference workloads stress GPU memory because each active conversation holds a KV cache that grows with the context length. The shift from static cache reservation to dynamic block allocation changes the resource model for platform teams. Instead of provisioning a fixed amount of memory per request, operators now need a scheduler that tracks block usage, a KV‑cache manager that maps logical blocks to physical memory, and a batching engine that can add or remove requests at any moment.
Technical fundamentals
KV cache and attention
The attention operation multiplies a query vector by a matrix of keys and values derived from previously generated tokens. In a transformer LLM, the keys and values for each token are stored in the KV cache. The cache size grows linearly with the number of tokens and with the number of attention heads. If a request has a maximum context of C tokens and H heads, the naive memory reservation is C × H × block_size bytes, even if the request only generates c < C tokens.
PagedAttention
PagedAttention splits the KV cache into blocks of a fixed token count (the block size). A block table maps a sequence’s logical blocks to the physical blocks allocated by the GPU memory manager. When the attention kernel needs a key or value, it consults the block table to locate the correct physical memory. This design mirrors virtual memory: logical pages are allocated on demand, and only the final block of a sequence may be partially filled. The original vLLM design note reported under four percent waste compared with contiguous allocation, and a 13‑billion‑parameter model’s KV cache could occupy roughly 1.7 GB when fully allocated.
Continuous batching
Because blocks are allocated per token, a new request can be admitted to the running batch as soon as enough free blocks exist. The batching engine does not need to wait for a previous request to finish its full context. This continuous batching model reduces queue latency and improves GPU utilisation, especially when request lengths vary widely.
Quantisation (AWQ)
Activation‑aware weight quantisation (AWQ) reduces the precision of most weights to three‑ or four‑bit integers while preserving a small subset of salient weights in higher precision. The Hugging Face Transformers documentation shows a configuration that uses 4‑bit weights with a group size of 128 and 16‑bit activations. The quantised model occupies roughly a quarter of the memory of the full‑precision model, allowing more room for the KV cache on the same GPU.
Architecture and components
The serving stack can be divided into the following layers:
- Request ingress – HTTP/gRPC front‑end that receives prompts and returns generated tokens.
- Scheduler – decides when a request can join the batch, tracks block availability, and updates the block table.
- KV‑cache manager – implements the block allocator, maintains the block table, and interfaces with the attention kernels.
- Inference engine – runs the transformer model, using either full‑precision or quantised weights. The engine may employ tensor or pipeline parallelism for very large models.
- Post‑processing – detokenisation, safety filters, and response formatting.
The diagram below illustrates the logical‑to‑physical block mapping and how a new request joins an existing batch.
flowchart TB
subgraph GPU Memory
direction LR
B1[Physical Block 1]
B2[Physical Block 2]
B3[Physical Block 3]
B4[Physical Block 4]
end
subgraph Request A
A1[Logical Block A1]
A2[Logical Block A2]
end
subgraph Request B
B1L[Logical Block B1]
end
A1 -->|block table| B2
A2 -->|block table| B4
B1L -->|block table| B3
style GPU Memory fill:#eef,stroke:#333,stroke-width:1px
style Request A fill:#cfe,stroke:#333,stroke-width:1px
style Request B fill:#cfe,stroke:#333,stroke-width:1px
Detailed implementation approach
- Select the model and quantisation path – Use the vLLM README to enable either GPTQ or AWQ quantisation. For AWQ, the Transformers config snippet is:
model = AutoModelForCausalLM.from_pretrained(
"model-name",
load_in_4bit=True,
quantization_config=BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_compute_dtype=torch.float16,
bnb_4bit_use_double_quant=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_group_size=128,
),
)
- Deploy vLLM with PagedAttention – Launch the server with the
--enable-paged-attention flag (the exact flag is documented in the vLLM project). The server will initialise the KV‑cache manager and expose a continuous‑batching endpoint.
- Configure block size – Choose a block size that matches the typical token granularity (e.g., 16 tokens per block). Smaller blocks increase fragmentation overhead; larger blocks increase waste.
- Tune scheduler thresholds – Set maximum concurrent blocks per GPU and a latency budget for queue admission. The scheduler should prefer adding a request when at least one full block is free.
- Monitor metrics – Track block allocation rate, cache utilisation, queue time, and GPU memory fragmentation. The vLLM metrics endpoint provides these counters.
- Scale out – If a single GPU cannot hold the desired number of concurrent sequences, add more GPUs and enable tensor or pipeline parallelism. The KV‑cache manager remains per‑GPU; cross‑GPU coordination is handled by the scheduler.
Architecture and data‑flow
The data‑flow can be summarised in three stages:
- Prefill – The initial prompt is tokenised and the first set of KV blocks is allocated. The attention kernel reads the prompt tokens and writes the corresponding keys and values into the allocated blocks.
- Decode – For each generated token, the scheduler checks for a free block, allocates it, and updates the block table. The attention kernel then reads all previously allocated blocks to compute the next token.
- Post‑decode – The token is detokenised and returned. If the request is finished, the scheduler releases all blocks belonging to that sequence, making them available for other requests.
Security and governance
Serving LLMs in production raises several governance concerns:
- Model provenance – Verify the source of the weights and the quantisation method. Use signed model artifacts when possible.
- Data leakage – KV cache resides in GPU memory and may contain user‑provided prompt text. Ensure that memory is cleared when a request completes.
- Access control – Gate the inference endpoint behind authentication and rate limiting to prevent abuse.
- Audit logging – Record request identifiers, timestamps, and any sanitisation steps applied to outputs.
Enterprise use cases
- Customer support chatbots – Continuous batching allows many simultaneous conversations to share a GPU, reducing per‑interaction cost.
- Document summarisation – Large context windows benefit from paged allocation because only the portion of the document actually processed consumes memory.
- Code generation assistants – Quantised models can run on cheaper GPUs while still fitting the KV cache for multi‑turn interactions.
Benefits and limitations
| Component | What changes | Claims not made in sources |
|---|
| Full‑precision weights | Occupies the largest memory portion; no reduction in KV cache size. | No tokens‑per‑second figure. |
| AWQ 4‑bit weights | Reduces weight memory to roughly one‑quarter; leaves more room for KV cache. | No accuracy percentage provided. |
| KV cache with PagedAttention | Allocates blocks on demand; waste under four percent according to the 2023 essay. | No guarantee that the current kernel matches the historical design line‑by‑line. |
Benefits include higher GPU utilisation, lower per‑request cost, and the ability to serve longer contexts. Limitations involve added complexity in the scheduler, potential fragmentation if block size is poorly chosen, and the need to validate quantised model quality for each workload.
Operational considerations
- Capacity planning – Use historical block‑allocation traces to estimate the maximum concurrent blocks per GPU.
- Monitoring – Set alerts on block‑allocation failures, cache fragmentation rates, and queue latency.
- Failover – Deploy multiple inference nodes behind a load balancer; the scheduler should be stateless or replicate its state.
- Software upgrades – The vLLM design page notes that the original helper module is now only a layout reference; upgrades may change the kernel implementation while preserving the paged‑attention semantics.
Best practices
- Start with the default block size recommended by vLLM and adjust only after measuring fragmentation.
- Quantise only after baseline validation – Run a small sample of prompts through the full‑precision model and the quantised model to ensure acceptable output quality.
- Enable continuous batching – Do not rely on static batch windows; let the scheduler admit requests dynamically.
- Separate KV cache from user data – Zero out cache memory on request termination to avoid inadvertent data retention.
- Document configuration – Keep the scheduler thresholds, block size, and quantisation settings version‑controlled.
Analysis
The sources describe the mechanisms that enable efficient LLM inference, but they do not provide a complete performance model. The 2023 essay reports less than four percent memory waste and a 1.7 GB KV cache for a 13‑billion‑parameter model, yet the current vLLM documentation cautions that the historical design note is not a line‑by‑line representation of the present code. Consequently, operators must treat the waste figure as an illustrative bound rather than a guaranteed metric. Likewise, the AWQ documentation states qualitative compression to four bits with minimal performance degradation, but it does not quantify the impact on downstream tasks. Real‑world deployments therefore need empirical testing to validate latency, throughput, and output quality.
What the sources do not settle
- Exact throughput – No source gives a definitive tokens‑per‑second rate for a given hardware configuration, making capacity planning reliant on internal benchmarking.
- Quantisation accuracy impact – The AWQ docs mention “minimal performance degradation” without providing numerical accuracy loss, leaving a gap for model‑specific validation.
- Kernel implementation details – The design page warns that the older helper module no longer reflects the current kernel, so the exact algorithmic steps of PagedAttention in the latest release remain undocumented.
- Cross‑GPU KV‑cache sharing – While the README mentions parallelism across tensors and pipelines, it does not detail how KV‑cache blocks are coordinated when a sequence spans multiple GPUs.
Conclusion
Production inference for large language models hinges on efficient memory management. PagedAttention’s block‑based KV cache eliminates the waste inherent in static allocation, enabling continuous batching that keeps GPUs busy even with highly variable request lengths. Quantisation techniques such as AWQ further shrink the model’s weight footprint, freeing additional memory for cache and allowing larger contexts or more concurrent users on the same hardware. Implementing these mechanisms requires a scheduler that tracks block usage, a KV‑cache manager that maintains block tables, and careful monitoring of fragmentation and latency. While the available sources describe the concepts and provide qualitative claims, operators must perform their own measurements to confirm throughput, accuracy, and stability in a production environment.
References
These published CloudExchangers articles are further desk reading. They are not sources for the technical claims above.