Overview
VCG-Bench is presented as a unified visual-centric benchmark for structured generation and editing tasks involving diagrams. This benchmark aims to address a gap in the capabilities of Vision-Language Models (VLMs) concerning highly controllable, structured diagrammatic tasks, which are pertinent to professional workflows. The proposed approach diverges from predominant pixel-based synthesis methods by introducing a 'Diagram-as-Code' paradigm. This paradigm leverages symbolic logic, specifically mxGraph Extensible Markup Language (XML), for the precise generation and editing of diagrams.
Research Context
Rapid advancements have occurred in Vision-Language Models (VLMs). However, a critical limitation persists in their capacity to manage structured, controllable diagrammatic tasks. Existing methodologies largely depend on pixel-based synthesis. This approach operates within probabilistic pixel spaces, which inherently restricts editability and the fidelity of the generated content. The identified need is for a system that can generate and edit diagrams with greater precision and control, moving beyond the limitations of pixel-level manipulation.
The research posits that a 'Diagram-as-Code' paradigm, utilizing symbolic logic, offers a more suitable foundation for these tasks. This paradigm applies mxGraph XML for representing diagrams, enabling accurate generation and editing processes.
Approach
The development of VCG-Bench involved several key components:
- Taxonomized Dataset: The benchmark includes a dataset comprising 1,449 diverse diagrams. These diagrams span 6 distinct domains and are further categorized into 15 sub-domains. This diversity is intended to provide a comprehensive testbed for VLM capabilities.
- Paradigm Definition: VCG-Bench defines a specific paradigm that integrates two primary functionalities: Generation (referred to as Vision-to-Code) and Editability (referred to as Code-to-Code). This integration reflects the workflow required for structured diagrammatic tasks.
- Tailored Evaluation Protocol: A multi-dimensional evaluation protocol was developed for VCG-Bench. This protocol employs several metrics to assess performance. Key metrics include:
- mxGraph Execution Success Rate: Measures the successful rendering or interpretation of the generated mxGraph XML code.
- Style Consistency Score (SCS): Evaluates the stylistic coherence of the generated or edited diagrams.
Findings
Experimental results from VCG-Bench highlighted specific challenges for current State-of-the-Art (SOTA) VLMs. These challenges were observed in two primary areas:
- Structured Fidelity: SOTA VLMs encountered difficulties in accurately reproducing the structured elements of diagrams.
- Instruction Compliance: The models demonstrated limitations in adhering to specific instructions provided for diagram generation and editing tasks.
These observed difficulties reflect underlying limitations in the vision and reasoning capabilities of the tested VLMs when applied to structured, controllable diagrammatic tasks.
Why This Matters
Professional workflows frequently depend on the ability to generate and edit diagrams precisely and controllably. The findings from VCG-Bench indicate that existing Vision-Language Models face obstacles in performing these tasks at a level required for such applications due to issues with structured fidelity and instruction compliance. Addressing these limitations could improve the utility of VLMs in professional domains requiring accurate diagrammatic representation and manipulation.