Overview
The TRACE framework addresses computational and memory overheads associated with reinforcement learning (RL) for post-training large language models (LLMs), particularly during rollout generation. It focuses on facilitating efficient RL training through low-precision rollout. Previous FP4 RL methods often optimized quantization accuracy independently for training and rollout paths, potentially leading to discrepancies between these two quantized execution processes. TRACE introduces an alternative approach that directly aims to reduce this discrepancy.
TRACE Methodology
TRACE, standing for Train-Rollout Quantization Alignment via Compact GuidancE, is an FP4 quantization framework specifically designed for RL training of Mixture-of-Experts (MoE) language models. Its core mechanism involves rollout-guided quantization-aware training. This guidance utilizes quantization outcomes from the rollout phase to inform and shape FP4 rounding decisions made during the training phase. This direct coupling is intended to minimize the disparity between the quantized execution paths of training and rollout.
To manage the additional data and communication demands introduced by this rollout guidance, TRACE incorporates an efficient quantization-information caching scheme. This scheme selectively retains specific information—mantissa and scale data—from deeper layers. The selective retention of this information serves to reduce the storage and communication overhead inherent in the rollout guidance mechanism.
Findings
- TRACE facilitates joint FP4 weight/activation and FP4 KV-cache rollout.
- The RL performance achieved with TRACE is comparable to that obtained with BF16 rollout.
- TRACE yielded up to a 5.4x rollout speedup.
- The framework demonstrated strong final FP4 performance when compared with post-hoc FP4 quantization applied to BF16-trained policies.
Approach
The evaluation of TRACE was conducted across four distinct large-scale MoE language models. These models were assessed on a range of tasks, including reasoning, coding, and long-horizon RL tasks. This multi-task evaluation aimed to ascertain the framework's effectiveness and generalizability across different types of computational challenges relevant to LLMs.
Why This Matters
The substantial computational and memory overheads in RL for LLMs, especially during rollout generation, present a bottleneck for efficient training. TRACE's ability to achieve comparable RL performance to BF16 while utilizing FP4 quantization and delivering significant rollout speedups suggests a potential pathway for more resource-efficient RL training processes for large language models.