Overview
This research investigates the presence of shortcut learning within geospatial foundation model embeddings when applied to cross-disaster damage assessment. It examines whether high accuracy on familiar disaster scenarios may conceal models’ reliance on event- or scene-specific characteristics rather than transferable evidence for new disaster contexts.
Research Context
Geospatial foundation models offer potential support for disaster damage assessment. However, their generalization capabilities, particularly to entirely new disaster types, remain a critical area of investigation. The study explores the phenomenon of shortcut learning, where models achieve high performance using superficial correlations or contextual cues rather than robust, transferable features, which can lead to reduced performance on out-of-domain data.
Approach
The study utilized embeddings derived from four geospatial foundation models: TerraMind, Prithvi-EO-2.0, DOFA, and DINOv3. These embeddings were evaluated on two multi-disaster datasets: xView2 and BRIGHT. A key methodological distinction was the evaluation strategy: models were assessed by holding out an entire disaster event for testing, as opposed to the more common practice of randomly splitting image tiles across all disaster types. This approach was designed to reveal generalization issues to unseen disaster domains.
Four distinct sets of tests were conducted to identify the information sources supporting prediction within these models:
- Assessment of event and acquisition context.
- Analysis of before/after image inputs.
- Evaluation of the predictive power of building count.
- Examination of the impact of correct image pairing.
Motivated by initial findings, the research further tested the effectiveness of feature transformations, specifically those learned exclusively from training disasters, to enhance generalization to unseen disasters.
Findings
-
The evaluation strategy significantly impacted reported accuracy. When an entire disaster was held out for testing, the mean balanced accuracy for damage assessment dropped from $0.807$ to $0.638$ on the xView2 dataset. Similarly, on the BRIGHT dataset, accuracy decreased from $0.818$ to $0.568$. This reduction suggests that models were exhibiting shortcut learning when tested on unfamiliar disaster scenarios.
-
Pre-event images maintained predictive capability in damage assessment. This indicates that features present before a disaster event contribute to model predictions, potentially indicating reliance on contextual information that might not directly reflect damage itself.
-
A classifier trained solely on the annotated number of buildings achieved a balanced accuracy of $0.629$ on xView2 and $0.653$ on BRIGHT. This finding highlights the substantial predictive power of a simple covariate like building count, potentially contributing to shortcut learning if not properly accounted for.
-
Combining features from both before- and after-event imagery generally performed best on the xView2 dataset. However, on the BRIGHT dataset, combining features did not offer an improvement over using pre-event features alone.
-
Shuffling image pairs within the same disaster and damage class resulted in only modest losses in performance. This observation suggests that models may not be heavily reliant on the specific pairing of before-and-after images for prediction in all contexts.
-
Feature transformations learned exclusively from training disasters improved generalization to unseen disasters. Among the tested transformations, supervised contrastive learning demonstrated the best performance improvement on xView2, while Principal Component Analysis (PCA) was most effective on BRIGHT.
Why This Matters
The study’s findings underscore the importance of robust evaluation methodologies for assessing the true generalization capabilities of geospatial foundation models. Specifically, it highlights that evaluating models on unseen domains is crucial for accurately gauging their robustness. The demonstrated effectiveness of simple feature-level transformations in improving generalization to new disaster types provides an actionable direction for enhancing model reliability in real-world disaster response scenarios.