Overview
HUMAID-NER represents a new dataset specifically designed for disaster-related social media analysis. It facilitates joint named entity recognition (NER) and humanitarian event classification within English disaster tweets. This initiative addresses the existing gap where many disaster tweet resources primarily offer document-level category labels but lack span-level entity annotations. The dataset encompasses 60,000 English tweets and includes approximately 175,000 labeled entity spans, categorized across ten distinct, operationally motivated entity types. Concurrently, a multitask learning framework has been developed, employing a shared RoBERTa-large encoder for both disaster-specific named entity recognition and event classification. This framework integrates homoscedastic uncertainty weighting and a two-stage training schedule to manage task conflict.
Research Context
The rapid extraction of structured information from social media platforms is critical for effective humanitarian response efforts. Current resources for disaster tweet analysis largely focus on document-level categorization, which limits the granular identification of specific entities relevant to crisis situations. The development of HUMAID-NER directly responds to this limitation by providing detailed, span-level entity annotations. The dataset builds upon the existing HumAID benchmark, enhancing it with named entity recognition capabilities. The ten entity types chosen for annotation are operationally motivated, suggesting their relevance to practical humanitarian applications.
Approach
Dataset Creation
The HUMAID-NER dataset was constructed using a three-stage hybrid annotation pipeline. This pipeline combines several methodologies to generate the entity annotations:
- A spaCy transformer model served as an initial annotation component.
- Disaster-domain EntityRuler patterns were applied to identify specific entities relevant to the context.
- Structured regular expressions were utilized for further entity identification.
- A priority-based overlap resolution mechanism was implemented to manage instances where multiple annotation sources identified the same or overlapping spans.
The resulting dataset comprises 60,000 English disaster tweets, with annotations provided in BIO format. These annotations cover ten operationally relevant entity types, accumulating approximately 175,000 labeled entity spans.
Joint Multitask Learning Framework
A joint multitask learning framework was developed for simultaneous named entity recognition and humanitarian event classification. Key components of this framework include:
- **Shared Encoder:** A RoBERTa-large model functions as a shared encoder, processing input for both NER and classification tasks.
- **Uncertainty Weighting:** To mitigate task conflict during joint training, the model incorporates homoscedastic uncertainty weighting. This method uses learnable task parameters to dynamically adjust the influence of each task on the overall loss function.
- **Training Schedule:** A two-stage training schedule was implemented to optimize performance. In the second stage of training, the lower 18 out of 24 encoder layers of the RoBERTa-large model were frozen, allowing for more focused learning in the upper layers.
Findings
The proposed system was evaluated on the HUMAID-NER validation set, demonstrating its performance in both named entity recognition and event classification tasks.
- For named entity recognition (NER), the system achieved a span micro-F1 score of 0.841.
- For humanitarian event classification, the system attained a macro-F1 score of 0.761.
These results indicate that the framework can perform both tasks simultaneously with measured effectiveness. An end-to-end deployment of the system is demonstrated through a real-time web dashboard.
Why This Matters
The rapid extraction of structured information from social media is important for humanitarian response. This dataset and framework offer a resource for crisis informatics research by providing detailed, span-level entity annotations in disaster tweets, a feature largely absent in previous resources focused on document-level categories. The release of the dataset, models, and pipeline code supports reproducibility and facilitates further research in this domain.