Cross-modal clustering with deep correlated information bottleneck method

Xiaoqiang Yan, Yiqiao Mao, Yangdong Ye, Hui Yu

Research output: Contribution to journalArticlepeer-review

31 Downloads (Pure)


Cross-modal clustering (CMC) intends to improve the clustering accuracy (ACC) by exploiting the correlations across modalities. Although recent research has made impressive advances, it remains a challenge to sufficiently capture the correlations across modalities due to the high-dimensional nonlinear characteristics of individual modalities and the conflicts in heterogeneous modalities. In addition, the meaningless modality-private information in each modality might become dominant in the process of correlation mining, which also interferes with the clustering performance. To tackle these challenges, we devise a novel deep correlated information bottleneck (DCIB) method, which aims at exploring the correlation information between multiple modalities while eliminating the modality-private information in each modality in an end-to-end manner. Specifically, DCIB treats the CMC task as a two-stage data compression procedure, in which the modality-private information in each modality is eliminated under the guidance of the shared representation of multiple modalities. Meanwhile, the correlations between multiple modalities are preserved from the aspects of feature distributions and clustering assignments simultaneously. Finally, the objective of DCIB is formulated as an objective function based on a mutual information measurement, in which a variational optimization approach is proposed to ensure its convergence. Experimental results on four cross-modal datasets validate the superiority of the DCIB. Code is released at

Original languageEnglish
Pages (from-to)1-15
Number of pages15
JournalIEEE Transactions on Neural Networks and Learning Systems
Early online date23 May 2023
Publication statusEarly online - 23 May 2023


  • correlation
  • correlation mining
  • cross-modal clustering (CMC)
  • data compression
  • deep clustering
  • information bottleneck (IB)
  • linear programming
  • mutual information
  • optimization
  • semantics
  • task analysis
  • visualization

Cite this