Overview
CrossPool is a serving engine specifically designed for cold Mixture of Experts (MoE) models in multi-LLM environments. It addresses the GPU memory challenges that arise when hosting numerous sparse MoE models, many of which receive infrequent requests. The core mechanism involves disaggregating FFN weights and KV-cache into two distinct GPU memory pools: a weights pool and a KV-cache pool.
Research Context
Emerging LLM services frequently host multiple sparse MoE models. A significant proportion of these models receive sparse requests, classifying them as "cold." This scenario creates a GPU memory management problem because model weights are stable and determined by the model's structure, whereas KV-cache is transient and demand-dependent. The predominant issue is that cold models rarely experience peak KV-cache demand concurrently. Consequently, reserving worst-case KV capacity for each model individually results in wasted memory resources. While a shared KV-cache pool can provision for aggregate active demand, this solution is insufficient when weights and KV-cache reside within a monolithic GPU memory pool. In such setups, static weights compete with dynamic KV-cache. Furthermore, under cold, low-concurrency traffic, attention mechanisms become KV-head-limited, exposing only a fraction of replicated KV capacity. This leads to low GPU memory utilization and weakened support for long-context requests.
Approach
CrossPool's approach centers on the disaggregation of GPU memory into two dedicated pools: a weights pool and a KV-cache pool. The weights pool consolidates FFN weights across cold models. The KV-cache pool dynamically serves active requests, maintaining attention local to the KV-cache. The system integrates several components to achieve its functionality:
- A KV-cache planner and virtualizer.
- A layer-wise pipeline scheduler, designed to hide hidden-state transfers.
- Persistent kernels with control lowering, aimed at reducing CPU-GPU control overhead.
Findings
CrossPool's implementation of efficient GPU memory pooling underpins bursty long-context requests. The system exhibited performance improvements against kvcached-based multi-LLM serving systems, which are considered state-of-the-art. Specifically, CrossPool achieved reductions in P99 TBT (Time Between Tokens) by a factor of up to 10.4x.
Why This Matters
The described approach, by disaggregating KV-cache and weights, offers a method to manage GPU memory more efficiently in environments hosting numerous, often 'cold,' sparse MoE models. This can lead to improved performance, particularly for long-context requests, by addressing memory utilization issues arising from the transient nature of KV-cache versus the static nature of weights.