Aiming at the existing problem of UMHexagonS algorithm in H.264, an improved fast motion estimation algorithm is proposed. The algorithm enhances the coding efficiency by optimizing the detection sequence of initial prediction motion vector(MV) according to the possibility of initial prediction MV becoming the best point, then an improved 5×5 spiral global search pattern is designed to reduce the search points of pattern, and an early termination technique for sub-macro block is proposed for further reducing the computation of motion estimation. The experimental results show that, with only negligible change of encoding performance, the improved algorithm can effectively reduce the motion estimation time and enhance the coding efficiency, and it can suit for video sequence of different motion intensities.