At present, bird activity failure has become one of the main hidden dangers of high-speed railway. Finding and cleaning the birds’ nest of the catenary is a countermeasure. Traditional birds’ nest object detection methods require manual extraction of features, but hand-designed features are difficult to ensure generalization in complex contact network scenarios. To solve this problem, this paper proposes to use the deep learning based object detection algorithm to identify the birds’ nest on catenary. At the same time, an improved model based on the one-stage object detection model RetinaNet is proposed. The P2 feature layer is added to expand the receptive field range of the network, so that the smaller nest can be better detected. Finally, these deep learning based object detection algorithms are trained and tested using data sets collected by on-board equipment of high-speed railways. Experimental results show that the object detection algorithm based on deep learning is excellent in the catenary birds’ nest detection task, and the improved RetinaNet model has a mAP value of 90.4%, which is better than the original model. This algorithm has certain both reference and application value for the obstacle avoidance task of high?-?speed railway.