Overview
Causal Edit Serialization (CES) is a framework designed to represent and execute localized text revisions directly within an autoregressive language model's causal token stream. Rather than separating generation and editing into distinct processing stages, CES introduces a unified interface that pairs specialized pointer tokens with predicted source addresses. This mechanism allows models to execute edits—including insertions, deletions, replacements, copies, and movements—while maintaining a conventional causal attention structure.
Research Context
Standard autoregressive language models operate by predicting tokens sequentially from left to right. While this structure is effective for open-ended text generation, revision tasks present an architectural challenge: updating text requires referencing and modifying tokens located in the preceding input context or in previously generated output. Conventional autoregressive setups cannot natively revisit prior token positions without re-encoding text or employing external multi-stage editing pipelines.
Approach
The CES framework establishes an editing mechanism that remains compatible with standard causal attention masks. The core system combines specific architectural components and training strategies:
- Pointer-Based Localization: The architecture pairs specialized pointer tokens with predicted source addresses, explicitly localizing operations such as insertion, deletion, replacement, copying, and movement.
- Localization Head and Encoding: A lightweight localization head operates alongside a dual-index rotary position encoding. This configuration supports both the prediction of target addresses and autoregressive conditioning on previously emitted pointers.
- Attention Compatibility: The design retains a standard causal attention mask, avoiding modifications that would disrupt autoregressive processing.
- Pretraining Scheme: Training integrates standard causal language modeling with serialized repair tasks derived from three data sources: random corruptions, model-generated edits, and repeated text sequences.
- Post-Training: The framework employs both supervised fine-tuning and reinforcement-learning post-training to align edit sequences for specific revision tasks.
Findings
Evaluation across model scales and editing baselines highlights several performance characteristics of CES:
- Edit Completeness: CES models successfully produce complete applied edits across varied baseline configurations and parameter scales.
- Language Modeling Metrics: When evaluated on clean text, CES models demonstrate modestly higher bits per byte than standard autoregressive models trained exclusively on clean text.
- Grammatical Error Correction: In a grammatical-error-correction (GEC) proof of concept, reference-blind model judging indicated that post-trained CES models generated a higher proportion of beneficial changes than harmful ones.
- Context Retention and Downstream Prediction: Applying correct repair programs improves subsequent token prediction while preserving the original context and its cached key-value states.
Why This Matters
By executing edits within the standard causal token sequence, CES eliminates the need to discard cached states or reprocess unchanged context when making localized revisions. This establishes a shared interface capable of handling both new generation and context-aware edits in a single model execution pass.
Potential Applications
The researchers identify grammatical error correction as an initial proof-of-concept application. Additionally, the CES framework supports potential extensions toward iterative text refinement and broader pointer-addressed sequence transformations.