Mitigating Package Hallucinations in LLMs via Localized Model Editing Framework

Overview

Large Language Models (LLMs) display capabilities in software engineering tasks including code generation, library recommendation, and dependency configuration. However, LLMs can generate non-existent or invalid package names, a phenomenon termed package hallucination. This issue carries security implications, as malicious actors could exploit such hallucinations by registering harmful packages under these fabricated names, posing a risk to the software supply chain.

To address this, the BOUND framework was introduced as a lightweight, localized model editing system specifically designed to mitigate package hallucinations in LLMs. BOUND operates by conceptualizing package hallucination mitigation as a boundary editing problem concerning package validity. This boundary represents an LLM's capacity to discriminate between valid and hallucinated package names within a given task context.

Research Context

The application of LLMs in software development has highlighted their utility but also revealed vulnerabilities. Package hallucination is one such vulnerability, wherein LLMs produce package names that do not correspond to actual, valid packages. Prior research has identified that these hallucinations could be leveraged for software supply chain attacks. Attackers might register malicious packages with names that an LLM has hallucinated, thereby compromising the security and reliability of LLM-assisted software development processes. Therefore, developing strategies to mitigate package hallucination is presented as crucial for enhance the security of outputs from LLMs in software engineering.

Approach

BOUND tackles package hallucination by formulating its mitigation as a 'package-validity boundary editing problem.' The framework's methodology encompasses two main stages:

1. Risk-Aware Localization: BOUND first identifies and isolates modules within the LLM that are pertinent to package hallucination. This localization strategy is described as 'risk-aware.'
2. Boundary-Aware Editing: Following localization, the identified modules are edited using lightweight LoRA (Low-Rank Adaptation) adapters. The editing process employs a 'boundary-aware objective' designed to reinforce the recognition of valid packages while simultaneously suppressing the generation of hallucinated package names. A key aspect of this editing is the preservation of existing 'locality behavior' within the model.

Findings

Experimental results indicated that BOUND is effective in reducing package hallucinations while maintaining the quality of valid package recommendations. Specific findings include:

• In package recommendation tasks, BOUND achieved a 79.9% reduction in the Package-level Hallucination Rate (Package-HR) on prompts that were used for editing.
• For unseen prompts in the package recommendation task, BOUND reduced the Package-HR by 65.4%.
• The refined package-validity boundary learned through BOUND demonstrated generalization across other package-related tasks.
• In code generation tasks, BOUND reduced Package-HR by 12.8%.
• In pip install recommendation tasks, the framework achieved a 34.0% reduction in Package-HR.

These outcomes suggest that BOUND successfully refines the LLM's ability to distinguish between valid and hallucinated packages, enhancing the reliability of package-related outputs.

Why This Matters

The mitigation of package hallucinations directly addresses a identified security vulnerability in LLM-assisted software development. By reducing the generation of non-existent package names, BOUND helps prevent potential software supply chain attacks where malicious packages could be registered under hallucinated names. This improves the reliability and security of software engineering tasks that leverage LLMs, making their outputs more trustworthy for developers.