Overview
This research addresses the existing limitations in Vision Language Model (VLM) development, specifically the dominance of English-centric resources. It introduces a comprehensive suite of resources designed for VLM training and evaluation, spanning five European languages: English, French, German, Italian, and Spanish. These resources aim to bridge the current gaps in multilingual and multimodal datasets for training, and evaluation benchmarks across multiple languages.
Research Context
Vision Language Models have seen rapid progress, but their development is primarily grounded in English. This focus has resulted in two primary limitations: a scarcity of multilingual and multimodal datasets suitable for training, and an absence of comprehensive evaluation benchmarks that cover multiple languages.
Approach
The researchers adopted a regeneration-translation paradigm to create high-quality cross-lingual resources. This approach combines curated synthetic generation with manual annotation. For training, a corpus named Multi-PixMo was constructed. This corpus was created by regenerating examples from pre-existing datasets with permissively licensed models, specifically PixMo-Cap, PixMo-AskModelAnything, and CoSyn-400k.
For evaluation, a set of multilingual benchmarks was developed by translating widely used English datasets. These translated datasets include MMBench, ScienceQA, MME, POPE, and AI2D. The quality of these newly developed resources was assessed through both qualitative and quantitative human analyses, which included measuring inter-annotator agreement. Additionally, ablation studies were conducted to determine the impact of multilingual data, in contrast to English-only data, on VLM training.
Findings
- The multi-modal training corpus, Multi-PixMo, was successfully created using a regeneration-translation paradigm combining synthetic generation and manual annotation.
- Multilingual evaluation benchmarks were constructed by translating existing English datasets (MMbench, ScienceQA, MME, POPE, AI2D) into French, German, Italian, and Spanish.
- Qualitative and quantitative human analyses were conducted to assess the quality of the new resources, including measurement of inter-annotator agreement.
- Ablation studies, involving three different VLM models, indicated that integrating multilingual, multimodal examples into VLM training consistently provides benefits for performance on non-English benchmarks.
- These studies also suggested a positive transfer of benefits to English benchmarks when multilingual data was used in training.
Why This Matters
The developed multilingual training and evaluation resources address critical limitations in current VLM development. By providing data and benchmarks for multiple languages beyond English, this work supports the development of VLMs that are more broadly applicable and performant across different linguistic contexts, potentially reducing biases inherent in English-centric models.