Memory-aware continual learning with multi-modal social media streams for unsupervised disaster classification

Social media has emerged as a major hub for information dissemination during disasters. It serves as a central repository for massive real-time updates about disasters, facilitating the organization of humanitarian assistance. Disaster types classification, which identifies and categorizes various types of disasters on social media, has become a fundamental technique for rapid disaster response. However, current disaster classification methods suffer from the following two shortcomings: (1) They rely on manual annotation to train the models, which is expensive and time-consuming in rapidly updating social media; (2) During the disaster, reports on social media are progressively enriched from initial text to multi-modal mix containing image, audio, and video. Existing methods cannot handle these incremental multi-modal data. To address these challenges, we propose a Memory-aware Continual Learning (MaCL) method, which accomplishes unsupervised disaster classification by flexibly handling the multi-modal streams on social media in a continual learning manner. First, a modal-shared contrastive encoder is constructed to mine the correlation information among the textual and visual modalities on social media. We theoretically demonstrate that mining correlation information is approximately equivalent to preserving predictive information in the original data. Then, we design a Correlation Memory Bank (CMB) to store the features of current modalities. As new modalities emerge, the correlation information between the new modality and its global neighbors in the CMB is maximized. This strategy enables the correlation information to guide the classification of the newly arrived modalities. Finally, the new correlation information generated by the CMB continuously updates and accumulates, leading to improve disaster classification performance across all modalities. Extensive experiments on seven real-world datasets validate the effectiveness of the proposed MaCL.