ICANEWS

Engineering & Technology

Fair Aggregation of Crowdsourced Noisy Labels with Demographic Parity Constraints

This research investigates the fair aggregation of crowdsourced, noisy labels, particularly focusing on methods to mitigate bias amplification concerning sensit

Overview

This research investigates the fair aggregation of crowdsourced, noisy labels, particularly focusing on methods to mitigate bias amplification concerning sensitive features. The study addresses the acknowledged gap in existing research regarding fairness in crowdsourced aggregation, specifically the absence of convergence guarantees and the limited availability of post-processing approaches for enforcing $\varepsilon$-fairness under demographic parity. The work analyzes fairness within the $\varepsilon$-fairness framework for two aggregation methods: Majority Vote and Optimal Bayesian aggregation.

Research Context

Acquiring reliable ground-truth labels is often costly or infeasible, making crowdsourcing and the subsequent aggregation of noisy human annotations a prevalent alternative. However, aggregating subjective labels carries the risk of amplifying individual biases, particularly when sensitive features are involved. This amplification can lead to fairness concerns. Prior work on fairness in crowdsourced aggregation has been characterized by a lack of convergence guarantees and restricted post-processing options for achieving $\varepsilon$-fairness under demographic parity constraints.

Approach

The research first analyzes the fairness properties of crowdsourced aggregation methods, specifically Majority Vote and Optimal Bayesian aggregation, within the $\varepsilon$-fairness framework. For the small-crowd regime, an upper bound on the fairness gap of Majority Vote is derived, expressed in terms of the fairness gaps of individual annotators. The study then examines the convergence behavior of the aggregated consensus, demonstrating its exponential convergence rate to the fairness gap of the ground-truth under interpretable conditions. Recognizing that the ground-truth itself may still exhibit unfairness, the methodology includes generalizing a state-of-the-art multiclass fairness post-processing algorithm. This generalization extends the algorithm's applicability from continuous to discrete settings, enabling it to enforce strict demographic parity constraints on any aggregation rule. Experimental validation was conducted using both synthetic and real datasets.

Findings

  • In the small-crowd regime, an upper bound for the fairness gap of Majority Vote was derived, expressed in relation to the fairness gaps of individual annotators.
  • The fairness gap of the aggregated consensus was shown to converge exponentially fast to that of the ground-truth, provided interpretable conditions are met.
  • A state-of-the-art multiclass fairness post-processing algorithm was generalized from the continuous to the discrete setting. This generalized algorithm enforces strict demographic parity constraints on any aggregation rule.
  • Experiments on synthetic datasets demonstrated the effectiveness of the proposed approach.
  • Experiments on real datasets corroborated the theoretical insights derived in the study.

Why This Matters

The findings address concerns regarding the amplification of individual biases in crowdsourced data aggregation, particularly concerning sensitive features. By providing theoretical bounds and demonstrating convergence properties, the research contributes to a more principled understanding of fairness in these systems. The generalized post-processing algorithm offers a practical method for enforcing strict demographic parity, which can be applied to diverse aggregation rules and scenarios where ground-truth data may itself be unfair.

Key Findings