For the intelligent video surveillance system, a motion object retrieval match approach combining with the gray histogram and self-casting histogram is proposed in this paper, which can detect the object with abnormal direction of motion rapidly. This method uses the feature combined with the gray histogram and self-casting histogram to rapidly detect and match the object among crowds; uses the motion history record list of object centroid to continuously record the centroid of object and its motion direction; and compares the motion direction to find the abnormal object among moving crowds. The experiment result shows that, compared with method only employing the information of gray histogram, the accuracy of detection is improved after introducing object self-casting histogram, meanwhile, the motion history record list is fully qualified to record the motion information of moving objects. This method has less calculating complexity and good robustness against objects covered by each other during their movement by recording the speeds of centroid motion.