Overview
MoReBench is introduced as a novel benchmark designed to evaluate the procedural and pluralistic moral reasoning capabilities of language models. It aims to understand not only the decisions made by AI systems but also the methods by which those decisions are reached. The benchmark utilizes moral dilemmas as a testbed for process-focused evaluation, acknowledging that such problems can have multiple defensible conclusions.
Research Context
The increasing reliance on AI systems for decision-making necessitates an understanding of how these systems arrive at their conclusions, particularly concerning alignment with human values. Reasoning language models, which offer both final responses and intermediate thinking traces, provide an opportunity to study AI procedural reasoning. Unlike tasks with objectively correct answers, moral dilemmas present a suitable context for assessing the process of reasoning.
Approach
The development of MoReBench involved two primary components:
- MoReBench Core: This component consists of 1,000 moral scenarios. Each scenario is paired with a set of rubric criteria, identified by experts as essential elements to include or avoid during the reasoning process. The benchmark incorporates over 23,000 such criteria, encompassing aspects like identifying moral considerations, weighing trade-offs, and providing actionable recommendations. These criteria are intended to cover scenarios where AI advises humans on moral decisions, as well as situations where AI makes autonomous moral decisions.
- MoReBench-Theory: This specialized set comprises 150 examples specifically curated to test an AI's ability to reason under five major frameworks within normative ethics.
Findings
- Scaling laws and existing benchmarks, which assess performance in tasks such as math, code, and scientific reasoning, do not predict models' capacities for moral reasoning.
- Language models evaluated using MoReBench demonstrate partiality towards specific moral frameworks. Examples cited include Benthamite Act Utilitarianism and Kantian Deontology.
- This observed partiality towards certain moral frameworks may be a side effect of commonly used training paradigms.
Why This Matters
These benchmarks contribute to advancing process-focused reasoning evaluation. Such evaluation is positioned as a step towards achieving safer and more transparent AI systems.
Key Limitations Mentioned by Researchers
The report suggests that the observed partiality of models towards specific moral frameworks might be an artifact of popular training paradigms.