Ghost in the Context: Understanding Policy-Carriage Failures in LLMs

Recent research, detailed in a new arXiv pre-print titled "Ghost in the Context: Measuring Policy-Carriage Failures in Decision-Time Assembly," sheds light on a critical vulnerability in how large language model (LLM) agents process and act upon directives. The study identifies and analyzes a specific failure mode where policy-bearing instructions, crucial for guiding LLM behavior, are compromised during the construction of the agent's decision state. This phenomenon, which the researchers term 'Ghost in the Context' failures, occurs not due to inherent model flaws or persistent memory issues, but during the dynamic assembly of the contextual information an LLM uses to make decisions.

This investigation focuses on how a bounded decision state is created for LLM agents, a process involving various mechanisms such as truncation, summarization, reordering, and rewriting of interaction history. The core problem arises when directive-bearing state—information containing explicit instructions or policy boundaries—is dropped, weakened, or rebound during these assembly steps. Such compromises can lead to an agent acting outside its intended policy boundaries, even without a direct prompt override, changes to its underlying model, or any compromise of its long-term memory.

The systematic nature of this unmitigated risk, as highlighted by the researchers, underscores a fundamental challenge in ensuring reliable and controllable LLM behavior. Despite the development of sophisticated LLM architectures, the process of assembling the immediate context for decision-making remains a measurable part of the control path that is susceptible to these specific failures.

Research Goal: Quantifying Policy-Carriage Failures

The primary objective of this research was to study the identified failure mode, specifically how directive-bearing state can be compromised during the decision-time assembly of an LLM's context. The researchers aimed to quantify the extent of these failures and explore potential mitigation strategies. The study precisely defines the problem as a context assembly issue, distinct from other known LLM challenges.

The investigation operationalized this by evaluating two key metrics: judged exact constraint respect and direct audits of assembled-state visibility. These metrics allowed for a concrete assessment of whether LLMs were adhering to their specified policies and whether the crucial directive information was indeed visible within their dynamic decision-making context. The focus was on understanding the mechanisms by which policy boundaries could be crossed without explicit prompt overrides, model changes, or persistent memory compromises.

Examining LLM Performance Across Models

The researchers conducted their study across several local LLMs, including Llama 3.1 8B, Qwen 2.5 7B, and Mistral 7B. This selection of models allowed for an examination of the generalizability of the observed failure mode across different architectures and sizes within the specified parameter ranges. The use of specific, publicly available models ensures that the findings, within the scope of the study, are grounded in concrete implementations rather than abstract theoretical constructs.

Further, the study extended its analysis to larger models, specifically Qwen 14B and Llama 70B, to ascertain if the same failure object persists at increased scales. This comparative approach is critical for understanding whether "Ghost in the Context" failures are scale-dependent or a more pervasive issue across different LLM capacities. The observed sign and magnitude of these failures, according to the research, remained policy-conditional even with larger models, indicating that the problem is not simply ameliorated by increasing model size.

Key Findings: Systematic Risk and Partial Hardening

The research uncovered several significant findings regarding policy-carriage failures in LLMs. A central conclusion is that unmitigated risk associated with these failures is systematic. This means that the problem is not an infrequent occurrence but rather a consistent vulnerability in how LLM agents process directives within their decision-time context assembly.

Low Absolute Exact Policy Respect

Despite the systematic nature of the risk, the study found that absolute exact respect for constraints remains low. This implies that even without explicit failures due to corrupted context assembly, LLMs do not perfectly adhere to directives in all instances. The "Ghost in the Context" failures exacerbate an already imperfect adherence, making the overall control path less reliable. This suggests a baseline level of policy non-adherence that exists independently of the contextual assembly issues, though these issues represent a distinct and measurable component.

The challenge of achieving perfect policy adherence is amplified by the dynamic nature of context assembly. Even when the policy is clearly defined, its faithful carriage through the truncation, summarization, reordering, and rewriting processes inherent in context construction is not guaranteed. This contributes to the observed low absolute exact respect for constraints, further motivating the need for robust control layers.

Impact of SafeContext: Gains Against Truncation

To address these identified vulnerabilities, the researchers evaluated a control layer named SafeContext. SafeContext is designed with specific mechanisms: it pins control state, reuses retained control prefixes, and optionally injects reminders under pressure. Crucially, it operates while keeping the model weights fixed, meaning it's an external control mechanism rather than an alteration to the LLM's core neural architecture.

The study found that SafeContext yields small gains against truncation. Truncation is a common method for limiting the size of the interaction history, and it can inadvertently discard directive-bearing information. SafeContext's ability to provide small gains in this scenario indicates some success in preserving crucial policies even when the context is shortened. This suggests that the pinning of control state and the reuse of control prefixes can offer a modest buffer against the loss of directives due to context clipping.

Diminished Gains Against Strong Compaction Policies

However, the effectiveness of SafeContext was shown to be highly dependent on the compaction strategy employed. Against a strong structured-compaction policy, most aggregate lift or benefit from SafeContext disappears. This finding is significant because compaction strategies are often used to manage the limited context window of LLMs, and a strong compaction policy might aggressively summarize or consolidate information, thereby weakening or removing the directive-bearing state that SafeContext attempts to protect.

The implication is that while SafeContext can offer some protection against simpler context management techniques like basic truncation, its utility is significantly reduced when more sophisticated and aggressive compaction methods are in place. This highlights a limitation in current hardening techniques and points to the complex interplay between control layers and context assembly algorithms.

Residual Benefit in Overflow Eviction and Aliasing Slices

Despite the diminished aggregate lift against strong compaction, SafeContext still demonstrated residual benefit, primarily in overflow eviction and selected aliasing slices. Overflow eviction refers to the process of removing older or less relevant information when the context window is full. SafeContext's ability to protect directives during this process suggests it can help maintain policy adherence even when context size constraints necessitate the removal of other information.

Selected aliasing slices refer to specific instances or segments where directives might be obscured or misinterpreted due to contextual ambiguities. The residual benefit in these areas indicates that SafeContext can, in certain specific circumstances, help clarify or reinforce policy-bearing information, preventing it from being misunderstood or bypassed. These targeted benefits, though not comprehensive, demonstrate that the control layer offers some utility even under challenging conditions of context management.

Replay-Only Not the Explanation

The research also explicitly states that "Replay-only does not explain the effect." This refers to the possibility that the observed policy-carriage failures could merely be a consequence of the LLM replaying or repeating previous behaviors without direct reference to the current directives. By negating this as the primary explanation, the study reinforces its central hypothesis: the failures are indeed due to compromises in the dynamic decision-time assembly of the context, rather than a simple behavioral pattern stemming from memory or training issues.

This distinction is crucial for understanding the root cause of the problem and for developing effective mitigation strategies. If it were merely a replay effect, different solutions related to behavioral conditioning or memory might be pursued. However, by pinpointing context assembly as the culprit, the focus remains on ensuring the integrity of the information presented to the LLM at the moment of decision.

Failure Object Under Larger Models

The study extended its investigation to larger models, specifically Qwen 14B and Llama 70B, and confirmed that the same failure object exists under these larger models. This indicates that the "Ghost in the Context" phenomenon is not merely an artifact of smaller LLMs but a persistent challenge that scales with increased model size. While the presence of the failure object was consistent, the sign and magnitude of the failures remained policy-conditional, suggesting that the specific parameters of the policy and the context assembly process continue to play a significant role in how these failures manifest.

This finding is particularly important for the scaling laws of LLMs and their deployment in real-world applications where reliability and adherence to safety or operational policies are paramount. It suggests that simply increasing the model's capacity does not inherently solve the problem of policy-carriage failures during context assembly, necessitating continued research into robust control mechanisms.

Methodology: Judged Constraint Respect and Direct Audits

The research employed a methodology centered on two principal evaluation approaches: judged exact constraint respect and direct audits of assembled-state visibility. These methods were applied to local Llama 3.1 8B, Qwen 2.5 7B, and Mistral 7B models, and extended to Qwen 14B and Llama 70B for a broader analysis.

Judged exact constraint respect involved a qualitative or quantitative assessment of whether an LLM's output precisely adhered to specified directives or constraints embedded within its operational policy. This metric directly measured the success or failure of the LLM in carrying out its prescribed functions according to policy. The direct audits of assembled-state visibility involved inspecting the exact context that the LLM was operating on at the decision point. This allowed researchers to verify whether the directive-bearing state was present, weakened, or unbound, directly linking the LLM's behavior to the integrity of its immediate decision context.

The evaluation of SafeContext involved comparing the performance of LLMs with and without the control layer under various context management scenarios, including against truncation policies and strong structured-compaction policies. This comparative analysis allowed for a systematic assessment of SafeContext's effectiveness in mitigating policy-carriage failures.

Implications: Hardening the Control Path

A key implication of this research is that decision-time context assembly is a measurable part of the control path that can be partially hardened. This finding suggests that while policy-carriage failures are systematic and impact LLMs across various scales, they are not insurmountable. The development and evaluation of control layers like SafeContext demonstrate that targeted interventions can offer some mitigation, even if complete policy adherence remains a challenge.

The ability to partially harden this control path opens avenues for future research and development in making LLMs more reliable and controllable. It emphasizes that ensuring directive-bearing state integrity during context creation is a critical area for improving LLM safety and robustness, distinct from issues related to model training or persistent memory. The persistence of the failure object even in larger models underscores the need for continued focus on this specific mechanism rather than assuming that model scaling alone will resolve all control challenges.

What's Next: Further Exploration of Control Layers

While the study confirms that decision-time context assembly is a measurable part of the control path that can be partially hardened, it also highlights the limitations of current solutions, particularly against strong compaction policies. This suggests a clear direction for future research: further exploration and refinement of control layers that can more effectively preserve directive-bearing state under aggressive context management strategies.

Understanding the interplay between different compaction methods and control mechanisms will be crucial. Developing more sophisticated control layers that can dynamically adapt to varying context assembly techniques, potentially by integrating more deeply with the underlying LLM architecture without altering its weights, could be a next step. The conditional nature of the sign and magnitude of failures across different policies indicates that fine-grained, policy-aware control mechanisms might be necessary to achieve higher levels of exact constraint respect.