Overview
This study investigates the intuitive decision-making processes employed by machine learning practitioners when selecting source datasets for transfer learning applications in medical image classification. It adopts a human-computer interaction (HCI) perspective to understand these decisions, contrasting with prior research that primarily focuses on benchmarking models or experimental setups.
Research Context
Transfer learning is recognized as a crucial methodology within medical imaging. However, the selection of appropriate source datasets often relies on researchers' intuition. This reliance can impact the generalizability of algorithms developed and, consequently, affect patient outcomes. The current understanding of how practitioners make these selections is limited, particularly regarding the underlying heuristics and considerations.
Approach
The research design involved conducting a task-based survey with machine learning practitioners. This survey aimed to elicit insights into their decision-making processes regarding source dataset selection for transfer learning in medical image classification. Unlike studies that primarily evaluate model performance, this work focused on the human element in the decision-making pipeline.
Findings
- Source dataset selection decisions are task-dependent.
- These decisions are influenced by community practices in the field.
- Properties inherent to the datasets, such as their characteristics and content, play a role in selection.
- Computational factors, specifically data embedding considerations, impact practitioners' choices.
- Perceived visual or semantic similarity between datasets also guides selection.
- A discrepancy was observed between practitioners' similarity ratings of datasets and their expected performance. This finding challenges the traditional assumption that a 'more similar is better' approach consistently aligns with actual performance outcomes.
- Ethical considerations and fairness aspects are largely absent from the current practices of source dataset selection among the participants.
- Participants frequently utilized ambiguous terminology when discussing their selection criteria, suggesting a need for more precise definitions and tools to facilitate explicit and usable communication of these concepts.
Why This Matters
By articulating the heuristics currently used by practitioners and proposing a conceptual framework for transfer learning factors, this research offers practical insights. These insights aim to facilitate a more systematic approach to source selection in transfer learning, potentially improving the reliability and generalizability of medical imaging algorithms and, by extension, patient care. Addressing the absence of ethical and fairness considerations could lead to more equitable algorithm development.