SSGNN: A macro and microfacial expression recognition graph neural network combining spatial and spectral domain features

Emotion recognition from macro-expression and micro-expression has been widely used in applications such as human-computer interaction, learning status evaluation and mental disorder diagnosis. However, due to the complexity of human macro-expressions, recognizing macro-expressions with high accuracy is a challenging task. Moreover, the short duration and low movement intensity of micro-expressions make its recognition more difficult. For MM-FER (macro and micro facial expression recognition), the key information can be more efficiently expressed by a graph. In this paper, a novel framework based on graph neural network named SSGNN (spatial and spectral domain features based on a graph neural network) is designed to extract spatial and spectral domain features from facial images for MM-FER, which can efficiently recognize both macro-expressions and micro-expressions under the same model. SSGNN consists of two parts, SPAGNN and SPEGNN, which are used to extract spectral and spatial domain features respectively. Experiments proved that jointly using the spectral and spatial information extracted by SSGNN can largely improve the performance of MM-FER when the training sample is limited. Firstly, the influences of different neighbours and samples to the model performance was analysed. Then the contribution of SPAGNN and SPEGNN were evaluated. It was discovered that fusing the result of SPAGNN and SPEGNN at decision level further improved the performance of MM-FER. Experiment proved that SSGNN can recognize micro-expression acquired by various sensors with higher accuracy under different image resolutions and image formats than the compared state-of-the-art methods in most cases. A cross-dataset experiment demonstrated the generalization ability of SSGNN.