The training of deep convolutional neural networks becomes more and more difficult and its performance is degraded with the increase of the number of convolution layers to solve the problem. A facial expression recognition method is presented based on deep residual network. The method uses building blocks for residual learning to improve the training and optimization process of the deep convolutional neural network model and reduce the time cost of the model convergence. In addition, to improve the generalization ability of the network model, a hybrid dataset for training network model is made up of the expression image samples which are selected from the KDEF and CK+ expression datasets. The comparative experiment was conducted with 10-fold cross validation method on the hybrid dataset. In term of expression recognition accuracy, we compared the residual networks with residual learning and the conventional convolution neural networks without residual learning and demonstrated the effect of network depth on the recognition accuracy. The average recognition accuracy of 90.79% is achieved as a 74-layer deep residual network is adopted. The experimental results show that the deep convolutional neural network constructed with building blocks for residual learning can solve the contradiction between the network depth and the model convergence, and can improve the accuracy of expression recognition.