Deep learning has developed rapidly in recent years. The concept of deep learning originates from the neural networks. And the activation function is an indispensable part of the neural network model in learning to understand non-linear functions. Therefore, the common activation functions are studied and compared, aiming at the problems of slow convergence speed, local minimum or gradient disappearance of the commonly used activation functions in back propagation neural networks. In this paper, the Sigmoid and ReLU activation functions are compared, their performances are discussed respectively, and the advantages and disadvantages of several common activation functions are analyzed in detail. Finally, a new activation function, ArcReLU, is proposed by studying the possibility of applying Arctan functions in neural networks and combining with ReLU functions. Experiments show that the function can not only significantly accelerate the training speed of BP neural network, but also effectively reduce the training error and avoid the problem of gradient disappearance.