Overview
A recent evaluation investigated the capability of Large Language Models (LLMs) to interpret and apply the International Maritime Dangerous Goods (IMDG) Code. This regulatory framework governs the transport of dangerous goods by sea, an activity characterized by high-consequence risks, including fire, explosion, toxic release, or loss of life or vessel, if errors in classification, packaging, stowage, or segregation occur. The IMDG Code is complex, requiring interpretation of hundreds of interacting provisions that are updated biennially.
Research Context
Practitioners are increasingly utilizing LLMs as tools to aid decision-making. However, a systematic evaluation of their reliability in interpreting IMDG requirements for safety-critical applications had been lacking. The study addresses this gap by assessing whether LLMs can reliably assist with IMDG Code compliance.
Approach
The research introduced DGEval, a benchmark specifically designed to evaluate LLM knowledge of IMDG Amendment 42-24. DGEval was constructed using expert-written questions derived from the NCB Hazcheck e-learning platform and structured lookups from the Dangerous Goods List (DGL).
- The benchmark comprises 1,678 questions.
- Questions are categorized into multiple-choice, open-ended, DGL lookup, and regulatory identification tasks.
The study evaluated 13 distinct LLM models. These models originated from six different providers and were assessed across various thinking configurations. A maritime domain-specific fine-tuned model was also included in the evaluation. The researchers additionally tested the influence of web search capabilities on model performance.
Findings
The evaluation yielded several key findings regarding LLM performance on IMDG Code compliance tasks:
- The top-performing model demonstrated an ability to exceed the human practitioner baseline specifically on multiple-choice questions.
- Across all evaluated models, a consistent weakness was identified in operationally safety-critical domains. These include areas such as stowage, segregation, and the recall of regulatory information.
- LLMs showed potential to support compliance tasks, particularly in structured DGL lookups, especially when augmented with web search functionality.
Why This Matters
The results indicate that while LLMs can offer support for certain compliance activities, their unreliability in critical operational areas and in recalling regulatory text necessitates continued human oversight. Verification against authoritative sources remains essential before deploying LLMs in any context involving safety-critical maritime dangerous goods operations.
Potential Applications
The DGEval benchmark itself is not intended as a definitive characterization of current LLM capabilities but rather as a safety assurance instrument. Its design facilitates continuous application as LLMs undergo further development and evolution.