Overview
MERGED (Multimodal Entity Resolution via Generated Expert Reasoning Distillation) is a distillation framework designed to address challenges in product entity resolution, particularly concerning evolving relationship definitions driven by business needs. The framework aims to transfer both labels and structured reasoning from large vision-language models (VLMs) into a compact 7B-parameter student model. This process requires no human annotation and seeks to overcome the cost and latency limitations associated with direct deployment of large VLMs at production scale.
Research Context
Product entity resolution faces a persistent challenge: relationship definitions are not static but continually evolve in response to business requirements. Traditionally, adapting to these changes has necessitated slow, costly human annotation. This human annotation is often characterized by noise and lacks explicit reasoning. Large vision-language models offer an alternative, capable of adapting to new definitions immediately through zero-shot prompting and providing reasoning that human labels typically omit. However, the direct application of these large VLMs in production environments is constrained by their prohibitive cost and latency.
Approach
The MERGED framework implements a distillation process to transfer knowledge from multiple large VLM 'teacher' models to a smaller student model. The methodology involves:
- **Teacher Labeling and Reasoning:** Multiple teacher VLMs are used to label each product pair and concurrently articulate the reasoning supporting their decisions.
- **Data Generation from Agreement:** Product pairs where teacher models agree on the label contribute to a dataset used for supervised fine-tuning of the student model.
- **Disagreement Resolution for Preference Learning:** In instances where teacher models disagree on labels, a meta-judge is employed to resolve these disagreements. The outcomes of this resolution generate preference pairs, which are then utilized for Direct Preference Optimization (DPO).
- **Student Model Architecture:** The target student model is a compact 7B-parameter architecture.
- **Adaptation Mechanism:** The framework supports re-application from an existing checkpoint, enabling adaptation to new relationship definitions using a relatively small sample size (10,000 samples).
Findings
Evaluations were conducted against human-labeled ground truth using a multilingual e-commerce dataset, yielding several key observations:
- **Performance Improvement over Human-Labeled Baselines:** The MERGED-trained student model demonstrated a 13.79% improvement in PR-AUC (Precision-Recall Area Under the Curve) compared to the same model backbone when trained solely on human-labeled data.
- **Outperformance of Larger Models:** The compact 7B-parameter student model surpassed the performance of a larger Qwen2.5-32B-VL baseline by 6.32% in PR-AUC.
- **Cost Efficiency:** This performance gain was achieved at a 6x lower cost compared to the Qwen2.5-32B-VL baseline.
- **Label-Reasoning Alignment:** The MERGED framework resulted in tighter alignment between labels and reasoning, exceeding Qwen2.5-32B-VL by over 10% in this metric.
- **Rapid Adaptation to New Definitions:** When re-applied from an existing checkpoint, MERGED enabled adaptation to a new relationship definition using only 10,000 samples. This adaptation resulted in a 6.97% improvement in PR-AUC over zero-shot methods and outperformed training from scratch for the new definition.
Why This Matters
The MERGED framework enables rapid adaptation to evolving relationship definitions, facilitating the support of new definitions within days rather than months. This capability is realized at a cost and latency profile suitable for large-scale industrial deployment, addressing a critical need in dynamic business environments where product relationships frequently change.