Lean-GAP: A Dataset of Formalized Graduate Algebra Problems from Dummit and Foote

Overview

Lean-GAP (Lean-Graduate Algebra Problems) is a dataset comprising 430 formalized graduate-level algebra problems. These problems originate from the textbook Abstract Algebra by Dummit and Foote. The development of Lean-GAP involved a scalable pipeline incorporating PDF-to-LaTeX preprocessing, autoformalization into Lean 4, and verification of informal-formal correspondence. While preprocessing and autoformalization stages demonstrated potential for automation, the verification component was identified as subtle and labor-intensive, necessitating human oversight.

Research Context

The project addresses the formalization of textbook mathematics, specifically focusing on graduate-level algebra exercises. The initiative aims to create a structured dataset of formalized exercises and to develop a systematic methodology for the formalization process. This effort also includes an analysis of challenges encountered during formalization and a comparison of different autoformalization models.

Approach

The methodology for constructing Lean-GAP involved a multi-stage pipeline:

1. PDF-to-LaTeX Preprocessing: This initial stage converted the original PDF-based problems into a LaTeX format. This process was noted to be largely amenable to automation.
2. Autoformalization into Lean 4: Following the LaTeX conversion, problems were automatically formalized into the Lean 4 proof assistant language. This stage also exhibited significant potential for automation.
3. Verification of Informal-Formal Correspondence: The final stage involved verifying that the formalized problems accurately reflected their informal counterparts. This verification was identified as the most subtle and labor-intensive part of the pipeline, requiring substantial human involvement.

In addition to constructing the dataset, the research also systematically analyzed recurring challenges within the formalization process. It further included a comparison of the performance of various autoformalization models to understand their efficacy in translating informal mathematical statements into formal language. Key bottlenecks in this translation process were also highlighted.

Findings

• The Lean-GAP dataset consists of 430 formalized graduate-level algebra problems.
• The source material for these formalized problems is the textbook Abstract Algebra by Dummit and Foote.
• A scalable pipeline for formalization involves PDF-to-LaTeX preprocessing, autoformalization into Lean 4, and human verification.
• Preprocessing and autoformalization stages can be largely automated.
• Verification of informal-formal correspondence is the most subtle and labor-intensive component, requiring careful human oversight.
• Analysis of the formalization process revealed recurring challenges.
• A comparison of different autoformalization models indicated varying performance.
• Key bottlenecks exist in translating informal statements into formal language.