Overview
Evaluation in scientific reconstruction often relies on pointwise metrics such as Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), and per-event resolution. This approach implicitly assumes that a lower error universally equates to a better reconstruction. This research argues that this assumption is structurally flawed when applied to inverse problems characterized by multimodal posteriors.
The study proposes that point estimators, when trained to minimize metrics like Mean Squared Error (MSE) or MAE, inherently produce a marginal spectrum that is narrower than the actual spectrum. This occurs specifically when the posterior distribution possesses a non-zero width, as dictated by the law of total variance. This resulting bias is described as being independent of architectural choices, training methodologies, or the size of the dataset used. The identified bias specifically compresses spectral features, including tails, modes, and shapes, which are critical for downstream scientific measurements.
To address these identified limitations, a three-part evaluation protocol is introduced. Each component of this protocol is designed to target specific failure modes that other components might miss. The protocol includes assessing per-event distributional accuracy via the Continuous Ranked Probability Score (CRPS), evaluating population-level marginal accuracy through a spectrum-fidelity diagnostic, and determining uncertainty trustworthiness using coverage-based calibration.
Research Context
The domain of scientific reconstruction frequently utilizes pointwise metrics for assessing model performance. These metrics typically quantify the difference between a reconstructed value and a true value at individual points or events. The prevailing understanding has been that minimizing these pointwise errors leads to improved overall reconstruction quality. However, this study specifically investigates inverse problems, a class of problems where multiple input configurations can lead to the same observed output, often resulting in multimodal posterior distributions. The research highlights a fundamental discrepancy between minimizing pointwise errors and accurately capturing the complexity of these multimodal posteriors.
Approach
The research evaluates the limitations of pointwise metrics and introduces a comprehensive evaluation protocol. The theoretical foundation for the failure of pointwise metrics is based on the law of total variance, which suggests that minimizing certain error functions leads to a compression of the true underlying distribution when posteriors are multimodal. This compression affects spectral features that are crucial for scientific interpretation.
The proposed evaluation protocol is structured into three distinct components:
- Per-event distributional accuracy: Assessed using the Continuous Ranked Probability Score (CRPS).
- Population-level marginal accuracy: Evaluated through a spectrum-fidelity diagnostic.
- Uncertainty trustworthiness: Determined via coverage-based calibration.
This protocol was applied to two distinct scenarios: a synthetic benchmark with a known analytic posterior distribution and a realistic many-to-one inverse problem drawn from particle physics. The application of the protocol involved comparing model rankings derived from pointwise metrics against those obtained using the new distributional metrics and calibration techniques.
Findings
- Pointwise metrics, such as RMSE and MAE, demonstrably mislead in inverse problems with multimodal posteriors.
- Point estimators trained to minimize MSE or MAE inherently produce a marginal spectrum that is strictly narrower than the true spectrum when the posterior has non-zero width, as per the law of total variance.
- This resulting bias is independent of the architecture, training methodology, or dataset size.
- The bias specifically compresses spectral features, including tails, modes, and shapes, which are essential for downstream scientific measurements.
- On both a synthetic benchmark with an analytic posterior and a realistic many-to-one inverse problem from particle physics, model rankings reversed when comparing results from pointwise metrics to those from distributional metrics.
- Calibration further differentiated architectures that appeared indistinguishable when evaluated solely using CRPS.
- The study concludes that the choice of the evaluation protocol, rather than just the model itself, dictates the scientific conclusion reached.
Why This Matters
This research highlights a fundamental flaw in the prevalent use of pointwise metrics for evaluating scientific reconstruction, particularly in contexts involving multimodal posteriors. The demonstrated structural bias means that models optimized using these traditional metrics may inaccurately represent critical spectral features, potentially leading to incorrect scientific conclusions or interpretations.