Overview
KletterMix is a newly introduced German corpus developed for language model pretraining and annealing. It addresses observed deficits in existing German-language pretraining resources, which are often characterized as smaller, less curated, weakly documented, and lacking validation through controlled training experiments when compared to their English counterparts. The design of KletterMix aims to provide a reusable dataset artifact for the natural language processing and modeling community.
Research Context
The quality of pretraining data is a fundamental component for modern language models. However, the development of high-quality data resources for the German language has lagged behind English. This disparity often results in German-language resources being comparatively smaller, exhibiting less careful curation, and possessing weaker documentation. Furthermore, validation through controlled training experiments is less common for these German resources. The research identifies a need for a German corpus that matches the scale and diversity of contemporary pretraining datasets while allowing for direct comparisons to established English sources.
Approach
KletterMix was constructed by translating a state-of-the-art English pretraining corpus into German. A key aspect of this translation process was the preservation of several structural and metadata elements, including document boundaries, associated metadata, source structure, and topical diversity. This methodology aimed to create a German corpus that replicates the scale and diversity found in modern pretraining datasets, while simultaneously enabling direct comparative analysis with its originating English corpus.
The dataset was documented through a series of corpus-level analyses. These analyses encompassed translation quality, the distributions of document lengths, the breadth of topic coverage, the composition of source materials, and geographic metadata. To assess translation quality, the researchers employed COMETKiwi. For evaluating KletterMix as training data, controlled pretraining and annealing ablations were conducted. These ablations involved comparing KletterMix against established German corpora.
Findings
The application of COMETKiwi for translation quality assessment indicated that the translated documents within KletterMix achieved strong quality across various domains. This finding suggests that a careful translation approach can effectively preserve a significant portion of the semantic and stylistic richness inherent in the original English corpus.
Furthermore, evaluation of KletterMix as a training dataset revealed measurable improvements in models. Specifically, models pretrained on KletterMix, when compared to those trained on established German corpora during controlled pretraining and annealing ablations, achieved better performance on German-language downstream evaluations. These results suggest that carefully curated translated data, exemplified by KletterMix, can substantially enhance the German pretraining data ecosystem.
Why This Matters
The findings indicate that meticulously curated translated data can significantly bolster the existing German pretraining data ecosystem. This approach results in German-language resources that achieve scale and diversity comparable to modern pretraining datasets, potentially leading to more robust and capable German language models.