Overview
The STiTch framework (Semantic Transition and Transportation in collaboration) is designed for training-free zero-shot composed image retrieval (CIR) tasks. This approach aims to address identified challenges within existing LLM-based models for CIR, specifically concerning semantic gaps in caption generation and limitations in point-to-point alignment during retrieval.
Research Context
Training-free zero-shot composed image retrieval models have garnered increasing research interest due to their capacity for generalizability and flexibility in unseen multimodal retrieval scenarios. Recent advancements leveraging Large Language Models (LLMs) in this domain have focused on generating target captions by exploring the compositional abilities of LLMs.
However, two primary issues have been identified with these LLM-based approaches:
- Generated captions often introduce unexpected features from a reference image due to a semantic gap between the input image and the text modification. This occurs because images typically contain more detailed information than can be fully captured by text.
- The point-to-point alignment mechanisms used during the retrieval stage are observed to be insufficient for capturing diverse compositions.
Approach
STiTch introduces a novel framework to mitigate the aforementioned challenges. The approach integrates two main components:
-
Caption Refinement via Transition Vector
Given a composed caption inferred by an LLM, the framework refines it using a transition vector within the embedding space. The objective of this refinement is to move the caption closer to the target image's representation. By combining LLMs with user instructions, the refined caption is designed to focus more precisely on the core modification intent, thereby filtering out extraneous information or "unnecessary noise."
-
Set-to-Set Alignment with Bidirectional Transportation Distance
To facilitate diverse alignment during the retrieval phase, STiTch models both the caption and the image as discrete distributions. This reformulation transforms the retrieval task into a set-to-set alignment task. A bidirectional transportation distance is developed to compute the retrieval score, account for fine-grained alignments across different modalities.
Findings
Extensive experiments were conducted to evaluate the STiTch method. The results indicated that the proposed method is general, effective, and beneficial across various composed image retrieval tasks.
Why This Matters
The development of generalizable and flexible models for composed image retrieval, particularly in zero-shot and training-free settings, addresses a need for systems that can respond to novel multimodal queries without requiring extensive retraining. By refining caption generation and improving multimodal alignment, STiTch offers an approach to enhance the accuracy and relevance of retrieval in scenarios where input images and text modifications require precise interpretation.
Key Limitations Mentioned by Researchers
The source identifies two primary limitations with existing LLM-based CIR models that STiTch aims to address:
- The introduction of unexpected features in generated captions from reference images due to the semantic discrepancy between multimodal inputs.
- The inability of point-to-point alignment methods to capture diverse compositions during retrieval.