Overview
Accurately assessing adversarial robustness presents a persistent challenge in evaluating large language models (LLMs). When attack designs are flawed, they can inaccurately inflate robustness estimates, which subsequently compromises the reliability of deployment risk assessments and the comparative evaluation of different defense mechanisms. In the context of image classifiers, standardized attack methodologies, such as AutoAttack, have addressed this issue by providing a consistent evaluation baseline, enabling systematic comparisons across defenses. However, an equivalent standardized framework for LLM jailbreak evaluation does not yet exist.
Developing such a framework for LLMs is considered more complex. A reliable attack for this domain must inherently be compatible with black-box systems, applicable to diverse defense pipelines, and efficient in its operation. Existing methods do not concurrently satisfy these criteria. This research introduces Indirect Harm Optimization (IHO), a masked diffusion language model attacker designed to address these requirements.
Approach
The Indirect Harm Optimization (IHO) method operates as a masked diffusion language model attacker. Its training process involves iterative preference optimization, conducted against a harmfulness judge. A defining characteristic of IHO is its requirement for only black-box access to the target system. This design allows the method to function without needing internal information about the target LLM or its defense mechanisms.
IHO is adaptable for various attack scenarios. It can be employed without modification as a strong adaptive attack targeting individual harmful behaviors. Alternatively, it can serve as an efficient amortized policy. In this latter application, it demonstrates transferability to hold-out behaviors and to previously unseen target models, all without requiring further fine-tuning. This inherent flexibility and transferability are central to its proposed utility in LLM robustness evaluation.
Findings
The IHO method demonstrates considerable improvement in attack success when deployed against layered defenses. This includes scenarios involving a Circuit Breaker-trained model combined with an auxiliary detector. Importantly, this enhanced performance is achieved without requiring any defense-specific adaptation during the attack process.
The observed efficacy of IHO, particularly its ability to operate effectively against complex defense configurations and its black-box compatibility, positions it as a practical step toward establishing standardized jailbreak evaluation. Such standardized evaluation methodologies have historically contributed to improved reliability in the assessment and comparison of defense mechanisms, as seen in other domains like image classification.
Potential Applications
The development of Indirect Harm Optimization (IHO) is intended to advance the field of LLM jailbreak evaluation. By providing a method that is black-box compatible, applicable to arbitrary defense pipelines, and efficient, IHO serves as a step towards a standardized evaluation approach. Standardized attacks, exemplified by AutoAttack for image classifiers, have historically improved the reliability of robustness estimates and facilitated systematic comparisons between different defense strategies. IHO aims to fulfill a similar role for LLM jailbreak assessment, contributing to more reliable risk assessment and defense comparisons for LLM deployment.