Overview
This research investigates the performance of text classifiers in zero-shot cross-dataset transfer scenarios, focusing on fake news detection (FND) and sarcasm detection. The study identifies limitations in cross-dataset generalization, particularly highlighting the issue of shortcut learning from distributional artifacts. It proposes a diagnostic methodology to identify such failure modes in binary text classification.
Research Context
Cross-dataset generalization is a fundamental prerequisite for deploying text classifiers in real-world applications. However, a systematic evaluation of classifiers across corpora from different sources is often uncommon in fake news detection research and notably scarce in sarcasm detection research. Without robust cross-dataset generalization, models trained on one dataset may exhibit poor performance when applied to data from a different source, limiting their practical utility.
Approach
The study conducts a unified empirical investigation of zero-shot cross-dataset transfer across three distinct domains: Urdu fake news detection, English fake news detection, and sarcasm detection. For each domain, the researchers fine-tuned an xlm-roberta-base model on a source corpus and evaluated its performance on a target corpus derived from a different source. This approach allowed for a direct assessment of transferability. The performance of the xlm-roberta-base model was compared against baseline models, specifically TF-IDF with Logistic Regression (LR) and Support Vector Machines (SVM).
The specific datasets utilized for each domain were:
- Urdu FND: Ax-to-Grind and Notri-Fact datasets.
- English FND: WELFake and ISOT datasets.
- Sarcasm Detection: TweetEval Irony and Sarcasm Corpus V2 datasets.
To diagnose potential issues, the methodology incorporated a combination of analyses:
- Class-conditional length analysis: Examined text length differences between genuine and fake/sarcastic content within datasets.
- Bidirectional transfer asymmetry: Assessed differences in transfer performance when training on dataset A and testing on B, versus training on B and testing on A.
- Predicted label collapse inspection: Analyzed instances where model predictions heavily favored one class, indicating potential failure modes.
Findings
The study identified significant challenges in cross-dataset generalization across all investigated domains:
-
Urdu Fake News Detection
A severe length confound was identified within the Ax-to-Grind dataset. Fake articles in this dataset averaged 3.4 times more words than real articles. This confound led to a catastrophic collapse in transfer performance from Ax-to-Grind (A) to Notri-Fact (B), resulting in a macro F1 score of 0.005. Conversely, transfer from Notri-Fact (B) to Ax-to-Grind (A) achieved an F1 score of 0.771, demonstrating pronounced bidirectional transfer asymmetry.
-
English Fake News Detection
Similar failure modes were observed in English FND. While specific performance metrics were not detailed for this domain, the study indicated that issues extend beyond Urdu, confirming a broader challenge.
-
Sarcasm Detection
Failure modes in cross-dataset transfer were also evident in sarcasm detection, supporting the conclusion that shortcut learning from distributional artifacts is a wide-ranging problem. The study confirmed that these issues were present when transferring between TweetEval Irony and Sarcasm Corpus V2 datasets.
Overall, the research indicated that shortcut learning from distributional artifacts, such as length confounds, is a cross-lingual and cross-domain challenge in binary text classification. This phenomenon can lead to models learning superficial correlations rather than robust, generalizable features, causing poor performance in zero-shot transfer settings. The observed failures were not limited to specific languages or domains, suggesting a systematic issue in current text classification approaches when applied in diverse real-world contexts.
Why This Matters
The findings emphasize that seemingly high performance on a single dataset does not guarantee effective deployment in varied real-world scenarios. The presence of shortcut learning and length confounds significantly hinders the ability of text classifiers to generalize across different data sources. This highlights the critical need for diagnostic methodologies to identify and mitigate such issues before deploying models, particularly in sensitive applications like fake news detection where robust and reliable classification is paramount.