How to extract multi-scale features and model semantic dependencies between remote channels remains a challenge for expression recognition networks. This paper proposes a residual network based on pyramid split attention （PSA-ResNet）， which replaces the 3 × 3 convolution in the ResNet50 residual module with PSA to effectively extract multi-scale features and enhance the correlation of cross channel information. In order to reduce the differences between similar expressions and expand the distance between different types of expressions， a joint loss function optimization parameter of Softmax loss and Center loss is introduced during the training process. The proposed model is simulated on two publicly available datasets， Fer2013 and CK+， and achieves accuracies of 74.26% and 98.35%， respectively， further confirming that this method has better recognition results compared to cutting-edge algorithms.