Overview
PATTON represents a Processing-in-Memory (PIM) runtime designed to enable commodity PIM for production-level Large Language Model (LLM) serving. This system addresses the complexities of dynamic KV cache management and memory allocation within LLM serving engines when integrating with PIM architectures. Its design focuses on harmonizing the requirements of efficient Generalized Matrix-Vector (GEMV) operations, single-token write efficiency, and memory capacity utilization, particularly for the Value cache.
Research Context
Processing-in-Memory (PIM) holds promise for accelerating memory-bound decode attention operations in LLMs. However, the acceleration of attention alone is insufficient for practical production LLM serving environments. These environments necessitate engines that can dynamically allocate, populate, share, cache, and reclaim logical KV cache blocks. Implementing this lifecycle on commodity PIM hardware introduces specific challenges, including the need for efficient physical memory allocation, effective block-to-address mapping mechanisms, and accurate command generation.
A fundamental conflict arises concerning the Value cache, specifically between GEMV efficiency, the efficiency of single-token writes, and overall memory capacity. Layouts optimized for GEMV operations tend to scatter newly generated Value vectors across multiple memory rows, which consequently increases the cost associated with writing individual tokens. Conversely, employing finer-grained memory sharing to improve capacity utilization can lead to fragmentation of GEMV reductions, thereby hindering their efficiency.
Approach
PATTON introduces several architectural features to integrate production LLM serving engines with commodity PIM:
Hierarchical Granule Allocation
- Block-sized Key and Value granules are mapped one-to-one to logical token blocks. This mapping fixes their physical placements and associated commands.
- Coarser granules are employed to group these blocks. This grouping aims to facilitate efficient GEMV execution and optimize memory utilization.
Commit Zone
- A dedicated Commit Zone is utilized to stage partial Value blocks. This staging mechanism is designed to enable efficient single-token writes.
- After staging, these partial Value blocks are committed to locations optimized for GEMV operations.
Placement Tracking and Command Generation
- PATTON tracks the placements of KV cache blocks.
- This tracking enables the generation of necessary KV cache write commands.
- It also facilitates the generation of QK-transpose and SV commands, which are critical for attention mechanisms.
Findings
The evaluation of PATTON demonstrated specific performance improvements:
- PATTON achieved an average 1.95x speedup when measured across attention execution and runtime-induced prefill recomputation. This speedup is relative to evaluated baselines.
- The system exhibited 4.83x higher energy efficiency compared to the evaluated baselines for the same workloads.
- PATTON operates without requiring modifications to the PIM processing units themselves.
- The KV cache hit rate maintained by PATTON was comparable to that of the native GPU KV cache implemented in vLLM.
Why This Matters
The ability of PATTON to integrate production LLM serving engines with commodity PIM, while improving speed and energy efficiency without modifying PIM hardware, addresses key practical challenges in deploying LLMs. By maintaining a comparable KV cache hit rate to existing GPU solutions, it suggests a viable path for PIM adoption in real-world LLM inference systems.
Potential Applications
The research suggests PATTON's utility in accelerating memory-bound decode attention within production LLM serving, offering performance and efficiency benefits for LLM inference workflows.