Overview
Fev-bench is proposed as a new benchmark for time series forecasting, designed to provide a more realistic and rigorously evaluated environment for model assessment. It comprises 100 forecasting tasks spanning seven distinct domains, with 46 of these tasks incorporating covariates. The benchmark is supported by fev, a lightweight Python library facilitating reproducible evaluation and integration into existing workflows. Performance reporting in fev-bench employs principled aggregation methods, including bootstrapped confidence intervals, and presents results based on win rates and skill scores.
Research Context
The evaluation and progressive advancement of time series forecasting models, particularly those that are pretrained, are significantly influenced by the quality of benchmarks. Existing benchmarks frequently exhibit limitations in their domain coverage or fail to adequately represent real-world scenarios, such as forecasting tasks that involve covariates. A recurring concern is the aggregation procedures utilized in many benchmarks, which often lack statistical rigor. This deficiency can obscure whether observed performance disparities between models signify genuine improvements or are merely attributable to random variation. Furthermore, inconsistencies in evaluation infrastructure and rigidity in benchmark design can hinder their integration into established research pipelines.
Approach
To address the identified gaps, the researchers developed fev-bench, featuring 100 forecasting tasks. These tasks are distributed across seven distinct domains, and 46 of them include covariates, aiming for broader domain coverage and realistic task representation. The benchmark is underpinned by fev, a Python library specifically designed for forecasting evaluation. This library prioritizes reproducibility and aims for ease of integration with existing analytical workflows. For performance assessment, fev-bench employs principled aggregation techniques. It utilizes bootstrapped confidence intervals to ensure a robust reporting of performance. The evaluation metrics reported are categorized into two dimensions: win rates and skill scores.
Findings
The research reported results on fev-bench for a range of models, encompassing pretrained models, statistical models, and baseline models. This evaluation allowed for an initial assessment of various model types within the new benchmark's framework. The process of evaluating these models also facilitated the identification of specific directions for future research in time series forecasting.
Why This Matters
The quality of benchmarks directly impacts the meaningful evaluation and ongoing progress within time series forecasting. Fev-bench attempts to address existing benchmark limitations concerning domain coverage, real-world task representation, and statistical rigor in performance aggregation, aiming to provide a more reliable foundation for assessing model advancements.