Overview
This work addresses the challenge of pretraining large language models (LLMs) in continual or open-ended settings, where the total training horizon is not known beforehand. Existing pretraining methods frequently depend on horizon-dependent learning rate schedules and extensive tuning within a fixed computational budget, which are not suitable for 'anytime' operation. The research investigates and proposes 'anytime' learning schedules, which are independent of a predetermined training horizon.
Research Context
Large language models are increasingly deployed in environments requiring continuous or open-ended training. A significant limitation in current LLM pretraining practices is the reliance on learning rate schedules that necessitate prior knowledge of the total training horizon. Such schedules, including commonly used cosine decay, are not 'anytime' in their design, meaning they require a predefined compute budget or training duration for effective tuning and performance.
Approach
The research combines theoretical analysis with empirical evaluation. Theoretically, the study provides an analysis demonstrating the existence of anytime learning schedules. This theoretical framework specifically applies to overparameterized linear regression. A central aspect of this theoretical investigation is the role of weight averaging, also referred to as model merging, in achieving minimax convergence rates for stochastic gradient descent.
Empirically, the study evaluated language models with 150 million and 300 million parameters. These models were trained for up to 32 times the Chinchilla scale. The evaluation compared two types of anytime schedules—constant and $1/\sqrt{t}$ schedules—both incorporating weight averaging, against a carefully tuned cosine schedule. This comparison aimed to assess the practical efficacy of the proposed anytime schedules in a real-world LLM pretraining context.
Findings
The theoretical analysis indicated that anytime learning schedules exist for overparameterized linear regression. Within this theoretical context, weight averaging was identified as playing a central role in enabling stochastic gradient descent to achieve minimax convergence rates. These anytime schedules were shown to exhibit a polynomial decay with time, where the specific decay rate is determined by the problem's source and capacity conditions.
Empirically, the evaluation demonstrated that the anytime schedules, specifically the constant and $1/\sqrt{t}$ schedules combined with weight averaging, achieved final loss values comparable to those obtained using a well-tuned cosine schedule. This comparability was observed across the entire training range, extending up to 32x the Chinchilla scale, for both 150M and 300M parameter language models.
Why This Matters
The findings suggest that simple, horizon-free step sizes, when combined with weight averaging, offer a practical and effective alternative to conventional cosine learning rate schedules for pretraining large language models. This approach supports training in continual or open-ended settings where the total training horizon is not fixed in advance, potentially improving adaptability and efficiency in such dynamic environments.
Key Limitations Mentioned by Researchers
- The theoretical analysis demonstrating the existence of anytime learning schedules is specifically applied to overparameterized linear regression.