This paper proposes a lightweight and efficient flame detection algorithm of videos. The flame detection algorithm is based on the convolutional neural network of deep learning. Considering the motion characteristics of the flame in the continuous video frame， this paper evaluates the flame detection results by the motion object detection and removes the false positive results caused by stationary objects or lights. The motion object detection algorithm based on the Gaussian mixture model is highly efficient. In addition， we collect and label a set of fire detection dataset （FDD）， including 2 487 flame images and 15 fire videos under various scenarios with different flammable materials. In conclusion， the proposed algorithm obtains 98.94% accuracy on FDD test videos.