Overview
A self-supervised data enrichment methodology, termed SemEnrich, has been developed to address limitations in medical vision-language datasets, specifically regarding size constraints and bias towards negative findings. The approach focuses on augmenting existing medical reports by incorporating positive or neutral observations derived from semantic clustering of sentences within these reports. This enrichment process aims to improve the performance of vision-language learning models.
Research Context
Medical vision-language datasets frequently exhibit limitations concerning their overall size. Additionally, these datasets often display a bias towards negative findings, a characteristic attributed to clinical reporting practices where abnormalities are typically documented, while some positive or neutral findings might be omitted if deemed irrelevant to the patient's condition. This reporting tendency can impact the representativeness of medical reports used for training machine learning models.
Approach
The SemEnrich method operates by leveraging semantic clustering of report sentences. This process enables the identification of distinct clusters of semantically related content within medical reports. The core of the enrichment strategy involves augmenting the findings in the training set of medical reports. This augmentation is performed by adding positive or neutral observations that originate from different semantic clusters, all within a self-supervised framework. Furthermore, the methodology introduces a mechanism to incorporate the information gleaned from semantic clusters into the reward design for Gradient-based Policy Optimization (GRPO) training.
Findings
The application of the SemEnrich approach resulted in consistent performance gains during supervised fine-tuning across multiple evaluation metrics. The observed average gains were:
- 5.63% on COMET score
- 3.04% on Bert score
- 7.40% on Sentence Bleu
- 5.30% on CheXbert-F1
- 7.47% on RadGraph-F1 scores
Ablation studies were conducted to ascertain the source of these improvements. These studies confirmed that the observed enhancements stemmed specifically from the semantic clustering component of the method, rather than from random augmentation techniques. When semantic cluster information was incorporated into the reward design for GRPO training, further performance improvements were noted:
- 2.78% average gain on COMET score
- 3.14% average gain on Bert score
- 12.80% average gain on Sentence Bleu scores
Why This Matters
The proposed SemEnrich method addresses inherent limitations in current medical vision-language datasets, specifically their restricted size and bias towards negative findings. By enriching these datasets with positive/neutral observations, it offers a pathway to potentially enhance the robustness and comprehensiveness of models trained on such data.