ICANEWS

Engineering & Technology

Robustness of Retrieval-Augmented Generation to Document Poisoning: An Empirical Study on Llama 3.1 8B

This research investigates the robustness of Retrieval-Augmented Generation (RAG) systems against document poisoning, focusing on the impact of corrupted retrie

Overview

This research investigates the robustness of Retrieval-Augmented Generation (RAG) systems against document poisoning, focusing on the impact of corrupted retrieved documents on a language model's performance in a fact-checking task. The study specifically quantifies the degradation in accuracy of a small quantized model, Llama 3.1 8B, when varying fractions of its retrieved context are intentionally poisoned. The inquiry explores how different corruption strategies affect the model's ability to maintain factual integrity and its response patterns, such as abstention versus fabrication.

Research Context

Retrieval-augmented generation (RAG) is characterized by its reliance on retrieved documents to ground a language model's responses. This grounding mechanism is known to reduce instances of hallucination in language model outputs. However, this dependency concurrently introduces a new potential vulnerability: if the retrieved text is maliciously tampered with, the integrated language model may inadvertently reproduce or propagate the introduced falsehoods. This study addresses this emergent security concern by empirically measuring the susceptibility of RAG systems to such document poisoning.

Approach

The research employed a factorial experimental design to assess RAG robustness. The subject language model was a small quantized model, Llama 3.1 8B. The fact-checking task utilized in the study was constructed from the FEVER dataset.

Corruption Strategies

Three distinct corruption strategies were applied to the retrieved passages:
  • Entity swap: Replacement of named entities.
  • Number swap: Alteration of numerical values.
  • Negation: Introduction of contradictory statements or reversal of facts.

Poisoning Extent

These corruption strategies were systematically applied to different proportions of the retrieved context. For each query, a total of three passages were retrieved. The corruption was applied to zero, one, two, or all three of these retrieved passages, creating a gradient of poisoning intensity.

Experimental Execution

This methodology resulted in a factorial sweep comprising 588 distinct experimental runs. Accuracy measurements were recorded for each condition to quantify performance degradation. Additionally, query-level bootstrap intervals were employed to re-check specific patterns observed, particularly concerning number-based corruption. A lexical overlap proxy was used to assess unsupported generation, aiming to distinguish between explicit falsehoods and abstentions.

Findings

The study yielded several key observations regarding the robustness of Llama 3.1 8B under document poisoning:

  • Overall Accuracy Degradation: The model's accuracy substantially decreased when confronted with poisoned contexts. Accuracy fell from 77.9% when presented with clean context to 43.5% when all three retrieved passages were corrupted.
  • Impact of Corruption Strategies: Among the tested strategies, entity swap was identified as the most effective in flipping answers that were initially correct on clean context. This suggests that altering named entities has a particularly strong disruptive effect on the model's fact-checking capabilities.
  • Number-Based Corruption Pattern: For number-based corruption, the model's accuracy remained relatively stable as long as poisoned passages constituted a minority of the retrieved context. However, a distinct drop in performance was observed once the poisoned passages formed a majority. This pattern was re-checked and affirmed using query-level bootstrap intervals.
  • Model's Response to Attack: A notable finding was the model's primary reaction to corrupted context: it rarely generated new falsehoods. Instead, its dominant reaction was to abstain from providing a definitive answer. A lexical overlap proxy, used to measure unsupported generation, indicated that this metric fell under attack conditions rather than increasing, further supporting the observation that the model tended to abstain rather than invent.

Why This Matters

The findings from this research are significant for understanding the vulnerabilities and behavioral characteristics of RAG systems in adversarial environments. The demonstrated substantial drop in accuracy under document poisoning highlights a critical attack surface for RAG models, necessitating development of defense mechanisms. The observation that the model largely abstains rather than inventing new falsehoods offers insight into its intrinsic safety mechanisms, suggesting a tendency towards caution rather than confident propagation of misinformation when its grounding context is compromised.

Key Limitations Mentioned by Researchers

The researchers explicitly acknowledge several limitations of this study:

  • The study constitutes a small-scale measurement.
  • It relies on coarse automated labels for evaluation.
  • The strategy contrasts observed are considered suggestive, pending further investigation where decoding is controlled.
  • Stronger adjudication mechanisms are required for more definitive conclusions.

Key Findings

  • Accuracy of Llama 3.1 8B fell from 77.9% on clean context to 43.5% when all three retrieved passages were corrupted.
  • Entity swap flipped the largest share of answers that were correct on clean context.
  • Number-based corruption showed stable performance when poisoned passages were a minority, then jumped in degradation once they became a majority.
  • The model's dominant reaction to poisoning was to abstain, rarely inventing new falsehoods; lexical overlap proxy for unsupported generation fell under attack.