Convolutional neural networks are good at learning features, but not always optimal for classification, while extreme learning machines are good at producing decision surfaces from well-behaved feature vector, but cannot learn complicated invariances. Based on the advantages and disadvantages of convolutional neural networks and extreme learning machine, we present a hybrid system where a convolutional neural network is trained to extract features and an extreme learning machine is trained from the features learned by the convolutional neural networks to recognize faces. We also propose prefix part of the filters in the convolutional layers to reduce parameters for improving the recognition accuracy. The experimental results obtained on the ORL and XM2VTS databases show that the proposed method can effectively improve the performance of face recognition, and the method of prefixing part of the filters is better than the method of stochastic filters in small training data.