Overview
LLM-based agents, which execute multi-turn workflows, often involve an interplay of LLM calls, tool invocations, and environmental feedback. A primary challenge in these systems is managing the continuously growing context length to maintain model quality. Existing agent frameworks employ context engineering strategies, such as offloading, reduction, and isolation, to control this length. However, these strategies introduce significant overhead due to context transformation, which necessitates invalidating existing KV caches and triggering re-prefill, thereby increasing the time-to-first-token (TTFT).
Research Context
Modern LLM-based agent frameworks rely on context engineering to manage the continuous growth of context in multi-turn workflows. Strategies such as offloading, reduction, and isolation are implemented to control context length and preserve model quality. A known issue with these context transformation techniques is the computational burden they impose. Each transformation causes the invalidation of existing Key-Value (KV) caches and necessitates a re-prefill operation, directly contributing to increased TTFT.
Approach
The research identifies that context transformations possess a segment-decomposable property, meaning that the transformation of a prefix can be executed independently of future tokens. This insight forms the basis for a lookahead programming model. This model allows agent frameworks to express context transformations as asynchronous operations without requiring modifications to their existing execution logic.
The proposed runtime proactively executes these asynchronous transformations. This proactive execution prepares transformed KV caches in advance. This pre-computation enables direct context replacement without blocking the primary execution flow. Furthermore, the researchers designed a lookahead-aware scheduler specifically for LLM serving systems. This scheduler is engineered to support the newly introduced asynchronous requests alongside existing latency-critical workloads, while aiming to control interference between these different types of requests.
The implementation of this approach was designed to support representative context engineering strategies. It was integrated into both existing agent frameworks and LLM serving systems.
Findings
- Context transformations are identified as segment-decomposable, implying that the transformation of a prefix is independent of future tokens.
- The lookahead programming model enables agent frameworks to express context transformations as asynchronous operations.
- The runtime proactively executes these asynchronous transformations and prepares transformed KV caches in advance.
- Direct context replacement is enabled without blocking due to the pre-prepared KV caches.
- A lookahead-aware scheduler supports asynchronous requests in LLM serving systems alongside latency-critical workloads with controlled interference.
- The approach effectively eliminates transformation overhead.
- The proposed approach reduces the time-to-first-token (TTFT) by up to 11.9x.