Overview
Long-context inference faces increasing limitations due to the KV cache, where resident memory scales with context length. Decoding performance becomes constrained by repeated High Bandwidth Memory (HBM) streaming rather than arithmetic operations. Existing strategies, including eviction, windowing, quantization, and offloading, aim to reduce the memory footprint. However, these often only partially address the critical-path bottleneck, particularly when compressed states require reconstruction into dense vectors during the decoding process.
Spherical KV is proposed as a method for long-context inference that approaches KV allocation as a rate-distortion problem. This method is informed by attention geometry and is designed to facilitate efficient decoding. The approach integrates two core concepts: representing directional information economically within the decode hot loop and allocating retention and precision based on estimated future utility.
Approach
The Spherical KV method is built upon two distinct components: Angle-Domain Attention (ADA) and Rate-Distortion Retention (RDR).
Angle-Domain Attention (ADA)
ADA constitutes the first component. It involves storing keys using a spherical parameterization. This parameterization comprises a scalar radius and compact angle codes. A key aspect of ADA is its ability to compute attention logits directly from these angle codes, bypassing the need to reconstruct dense keys. This design is intended to preserve a paged, block-local, and fusion-friendly decode path. By computing logits directly from compact representations, ADA specifically targets HBM traffic in realistic serving environments.
Rate-Distortion Retention (RDR)
RDR is the second component of Spherical KV. This mechanism jointly determines keep/drop decisions and assigns precision tiers for each token and head, operating within a predefined budget. RDR aims to produce tier-homogeneous pages, which are characterized by lightweight metadata and enable coalesced reads. The combined operation of ADA and RDR offers a deployment-oriented mechanism focused on reducing KV residency while simultaneously preserving decode efficiency.
Research Context
The development of Spherical KV directly addresses challenges observed in long-context inference, where the KV cache can become a limiting factor. The current methods often fall short in fully mitigating the bottleneck associated with HBM streaming. The need for reconstruction of compressed states into dense vectors during decoding further exacerbates this issue. Spherical KV seeks to provide an alternative that specifically targets these efficiency and memory residency concerns.