Overview
Research explored the implementation of character training to cultivate risk aversion in AI agents, aiming to mitigate potential harm from misaligned agents. The study utilized persona traits as a mechanism for instilling specific risk preferences. A model constitution describing constant absolute risk aversion (CARA) over an agent's resources was developed and integrated into agents through on-policy distillation.
The findings indicated that character-trained models could achieve competitive performance against baselines, even without prior exposure to the benchmark's decision format during training. Furthermore, these models demonstrated improved out-of-distribution generalization in two out of four evaluated models. The research identified token budget and model choice as key influential factors in the effectiveness of character training for instilling risk aversion.
Research Context
The primary concern addressed by this research is the potential for misaligned AI agents to cause catastrophic harm. The premise is that risk aversion, when present in such agents, could act as a preventative measure. Specifically, misaligned but risk-averse agents would tend to prioritize safer strategies, such as engaging in deals with humans, over higher-risk alternatives like rebellion. This suggests that instilling risk aversion could be a critical component in ensuring AI safety.
Approach
The core methodology involved character training agents to be risk-averse. This was achieved by constructing a model constitution designed to describe constant absolute risk aversion (CARA) over an agent's resources. The instillation of this constitution occurred through a process termed on-policy distillation. This approach aimed to embed risk preferences directly into the agent's character via persona traits.
The study also involved modulating different aspects of the constructed constitution to assess their impact on the instillation of risk aversion. This allowed for an evaluation of which components were most influential in shaping the agent's risk preferences.
Findings
- Character training, utilizing persona traits, was found to be a robust mechanism for instilling risk preferences in AI agents.
- Models trained via on-policy distillation of a constant absolute risk aversion (CARA) constitution demonstrated competitive performance compared to baselines. This competitiveness was observed despite the character-trained models never having seen the benchmark's decision format during their training phase.
- These character-trained models exhibited better out-of-distribution generalization than the baselines in two of the four models evaluated.
- Modulation of different aspects of the constitution revealed that token budget and model choice were the most influential factors in effectively instilling risk aversion through character training.
Why This Matters
The results suggest that character training represents a promising and scalable method for instilling broad dispositions in AI agents. The ability to instill specific traits, such as risk aversion, into misaligned AI agents holds potential for mitigating risks associated with their actions. This approach could be leveraged to encourage AI agents to favor safer strategies over riskier ones, thereby reducing the likelihood of catastrophic outcomes.