Overview
Optimization modeling is a crucial component in decision-making across various sectors, including logistics, manufacturing, energy, and finance. The process of converting natural-language requirements into accurate optimization formulations and solver-executable code is labor-intensive. While large language models (LLMs) have been investigated for this task, their evaluation has predominantly relied on benchmarks with limited scale, such as toy-sized examples or synthetic datasets. These smaller-scale evaluations do not adequately represent the complexity and scale of industrial problems, which often involve $10^3$ to $10^6$ or more variables and constraints.
A recognized bottleneck in the development and assessment of LLMs for optimization modeling is the absence of benchmarks that connect natural-language specifications with reference formulations and corresponding solver code, all grounded in real-world optimization models. To address this gap, this research introduces MIPLIB-NL. MIPLIB-NL was constructed using a structure-aware reverse construction methodology, drawing from real mixed-integer linear programs (MILPs) found in MIPLIB 2017.
Approach
The construction of MIPLIB-NL involved a multi-stage pipeline designed to generate a benchmark grounded in real industrial problems. The methodology commenced with the Mixed-Integer Linear Programs (MILPs) from MIPLIB 2017.
Pipeline Steps
- Recovery of Compact Model Structure: The initial step focused on recovering compact, reusable model structures from the 'flat' solver formulations present in MIPLIB 2017. This process aimed to abstract the underlying logical and mathematical organization from the raw solver input.
- Reverse-Generation of Natural-Language Specifications: Following the recovery of compact structures, natural-language specifications were reverse-generated. These specifications were explicitly linked to the recovered mathematical structure, maintaining a unified model-data separation format. This ensures that the natural language accurately reflects the underlying optimization problem without conflating model parameters with specific data instances.
- Iterative Semantic Validation: The generated natural-language specifications and their corresponding models underwent iterative semantic validation. This validation process involved expert review to ensure accuracy and consistency. Additionally, human-LLM interaction was employed, coupled with independent reconstruction checks, to meticulously verify the semantic content and functional correctness of the problem descriptions and their mathematical representations.
This systematic pipeline resulted in the creation of 223 one-to-one reconstructions. Each reconstruction preserves the mathematical content of its original instance while being paired with a realistic natural-language description. This allows for a more accurate and robust evaluation of natural-language-to-optimization translation systems.
Findings
Experiments conducted using MIPLIB-NL revealed a significant outcome: systems that demonstrated strong performance on existing, smaller-scale benchmarks exhibited substantial performance degradation when evaluated against MIPLIB-NL. This degradation suggests that the benchmark effectively exposed failure modes in these systems. These particular failure modes were not apparent and remained invisible when evaluations were restricted to benchmarks of a toy scale.
Why This Matters
The creation of MIPLIB-NL addresses a critical need for benchmarks that reflect the complexity and scale of industrial optimization problems. It enables a more realistic evaluation of systems designed to translate natural language into optimization models, helping to identify and address deficiencies that are not visible in smaller-scale testing. This contributes to the development of more robust optimization solutions for real-world applications in logistics, manufacturing, energy, and finance.