Overview
This work introduces and formalizes the phenomenon of incomplete prompt jailbreaks (IPJ) in large language models (LLMs). The study provides an empirical characterization of how and when incomplete prompts can elicit harmful continuations from LLMs. It investigates the behavior of LLMs in responding to various types of incomplete harmful prompts and explores the effectiveness of current safeguard training methods.
Research Context
LLMs are frequently released as open-weight models, typically incorporating safeguards designed to prevent responses to harmful requests. Despite these safeguards, the research indicates that LLMs remain vulnerable when presented with incomplete harmful prompts, leading to the generation of harmful continuations. This vulnerability centers on the LLM's response mechanism during sentence completion, particularly how refusal is managed or delayed in the presence of potentially harmful input that is not fully formed.
Approach
The research involved a systematic empirical characterization of IPJ scenarios. This included an analysis of diverse 'attractor types' associated with incomplete sentence continuation. The study aimed to understand the conditions under which incomplete prompts result in harmful outputs. Subsequently, the researchers evaluated the efficacy of training models to refuse incomplete harmful prompts through parameter tuning. This evaluation specifically assessed whether such training generalizes across different content domains and attractor types. Finally, the approach extended to identifying specific functional neurons—termination and continuation neurons—to understand their roles in sentence completion and to explore their potential for more precise intervention.
Findings
- LLMs systematically delay refusal to harmful input until incomplete sentences are terminated.
- Training models to refuse incomplete harmful prompts via parameter tuning is insufficient. This method fails to generalize across both content domains and attractor types.
- Two functional neurons, identified as termination and continuation neurons, play a role in sentence completion.
Why This Matters
This research identifies a specific vulnerability in safeguards designed for open-weight LLMs, where harmful continuations can be elicited by incomplete prompts. The findings suggest that current training methods for refusal are not generalizable enough, highlighting a need for more nuanced control mechanisms within LLMs to manage harmful content effectively.
Potential Applications
The clarification of the roles of termination and continuation neurons suggests the potential for neuron-level interventions. Such interventions could provide a more precise and robust approach to developing defenses against incomplete prompt jailbreaks in LLMs.