Overview
Research into curriculum learning (CL) within Natural Language Processing (NLP) spans over a decade, yet a coherent understanding of appropriate difficulty functions or schedulers for specific problems remains elusive. A fine-grained taxonomy has been proposed to address this gap, designed to facilitate systematic analysis of CL strategies by separating difficulty evaluation from training scheduling. This framework aims to clarify why progress has been hindered in establishing principled accounts for CL application in NLP.
Research Context
The field of curriculum learning in NLP has accumulated over ten years of research. Despite this extensive period, a fundamental challenge persists: identifying which specific difficulty function or scheduler is optimal for a given NLP problem. The absence of a principled account has led to difficulties in comparing and building upon existing research. Prior works have been observed to conflate distinct notions of difficulty and scheduling, often pursuing varied objectives under the umbrella term of CL. This conflation has created a systematic incomparability problem, impeding the accumulation of a coherent evidence base within the domain.
Approach
The proposed methodology involves a two-pronged taxonomic separation: one for difficulty evaluation and another for training scheduling. This separation is intended to enable systematic analysis of CL strategies. For difficulty evaluation, the taxonomy distinguishes between attribution source and task dependence. This differentiation reveals difficulty as a 'perspectival concept,' which encodes varying assumptions about the factors contributing to an instance's learning difficulty. For scheduling, the approach introduces the first formalization of CL schedulers. This formalization is based on the concept of expected training contribution. To facilitate comparison across diverse implementations, the taxonomy incorporates notions of retention regimes and monotonicity properties for schedulers.
This taxonomy was applied in a dedicated analysis of existing CL works specifically within NLP. The application revealed the systematic incomparability problem identified in prior research, confirming that different notions of difficulty and scheduling are often intertwined and that varied objectives are pursued under a common CL label. The objective of this taxonomic framework extends beyond diagnosis; it is designed to support the design, analysis, and comparison of CL strategies. Furthermore, it motivates the implementation of evaluation practices that can disentangle the specific sources of observed improvements in CL applications.
Findings
- The lack of a principled account for selecting difficulty functions or schedulers in NLP CL research, despite over a decade of study, is a significant impediment.
- The proposed taxonomy distinctly separates difficulty evaluation from training scheduling, which was identified as a necessary step for systematic analysis.
- Difficulty evaluation is characterized by two dimensions: attribution source and task dependence, indicating difficulty is perspectival and reflects different assumptions about learning hardness.
- CL schedulers are formalized for the first time based on their expected training contribution.
- This formalization introduces retention regimes and monotonicity properties to allow for comparisons across different scheduler implementations.
- Application of the taxonomy to NLP CL works revealed a systematic incomparability problem, where prior research conflates difficulty and scheduling notions and pursues diverse objectives under the same CL label.
- This conflation hinders direct comparison and the accumulation of a coherent evidence base in the field.
Why This Matters
The proposed taxonomy directly addresses the systematic incomparability problem within curriculum learning research in NLP. By providing a structured framework, it aims to clarify underlying assumptions and facilitate more robust comparisons of CL strategies. This allows for a more coherent accumulation of evidence and supports the development of more effective CL methods in the future.