Overview
This research introduces a family of advantage-aligned wagering mechanisms for LLM aggregation (WALLA). The primary goal of WALLA is to enable the aggregation of predictions from multiple Large Language Models (LLMs) in decentralized environments. Each model reports a prediction and a learned wager, with these wagers subsequently used as weights for aggregating predictions. The proposed mechanisms are designed to address challenges associated with determining aggregation weights in decentralized settings, particularly when models possess private information and exhibit strategic reporting behaviors.
Research Context
The aggregation of predictions from multiple LLMs has become a common practice to enhance collective prediction performance. This aggregation is particularly relevant in scenarios where individual LLMs might possess domain-specific expertise or access to proprietary tools and data. When such aggregation occurs in decentralized environments, the process of determining appropriate aggregation weights becomes complex. Traditional methods may struggle with models withholding private information or engaging in strategic reporting. The development of mechanisms robust to these challenges, while still enabling effective aggregation, forms the core research problem addressed by WALLA.
Approach
The WALLA framework incorporates a leave-one-out baseline into its net payout function. This design choice is central to achieving several properties. The mechanisms are instantiated into two variants, each balancing normality and no-arbitrage properties, while maintaining a bounded worst-case deficit. The experimental evaluation of WALLA involved its application on question-answering and forecasting benchmarks. These experiments were conducted across heterogeneous models and various private-information settings to assess the mechanism's performance and properties.
The design of WALLA aims to achieve three specific properties:
-
Dominant-Strategy Incentive Compatibility of Prediction
This property ensures that models are incentivized to report their true predictions under any belief structure, removing strategic incentives to misreport. This is achieved through the structure of the net payout function.
-
Advantage-Wager Alignment
The mechanism ensures that the optimal wager submitted by a model is proportional to its expected score advantage. This means wagers accurately reflect a model's perceived contribution to the collective prediction.
-
Prediction-Agnostic Wager Optimization
This property allows for the decentralized learning of wager policies without requiring models to simultaneously produce optimal predictions. This de-coupling simplifies the learning process for individual models in a decentralized setting.
Findings
Experiments demonstrated that WALLA achieved predictive performance comparable to centralized aggregation methods. Beyond mere performance, the mechanisms facilitated several key outcomes:
-
Decentralized Learning: WALLA enabled the learning of optimal wager policies by individual models in a distributed fashion, without reliance on a central authority for weight determination.
-
Advantage-Aligned Aggregation Weights: The wagers learned by the models and used for aggregation were found to be aligned with the expected advantage of each model's prediction.
-
Uncertainty Awareness: The wagering mechanism implicitly incorporated an element of uncertainty awareness, as wagers reflect a model's confidence in its predictions.
-
Incentive-Compatible Prediction: The design of WALLA ensured that models were incentivized to report their genuine predictions, mitigating strategic misreporting.