Although great progress has been made in the field of computer vision target tracking, the performance of out-of-plane rotation and shape change in video tracking need to be improved. Here, HOG feature based on directional gradient histogram is proposed. Combined with the gray value of the image, the HOG feature is fused and decomposed to improve the performance of the deformation and scale transformation of the video tracking. Firstly, the 31-dimensional features of the HOG and the gray value of the image are extracted from the target region. Secondly, the gray value is regarded as one-dimensional feature, then the gray value is fused with HOG feature into 32-dimensional vector HOG32. Then the HOG32 is decomposed into two parts, namely, HOG1 and HOG2. Finally, compared the response values of HOG1,HOG2 and HOG32, the maximum position is selected as the position of the next frame predicted. The experiment is compared with the other five algorithms on OTB-2013 and OTB-2015 datasets. The results demonstrate that our method achieve better results in out-of-plane rotation, deformation and complex background.