One key technique of video analysis is panorama mosaic. Aiming at the two-dimensional projective motion between frames which is used frequently in the video sequence, a mosaic algorithm of video sequence panoramic image was proposed in this paper. Firstly, we estimate motion parameters of adjacent frame by using improved RANSAC-based matching algorithm for feature points. Secondly, the motion parameters between every frame with panorama be adjusted accurately by direct estimation method. Finally, the video sequence panoramic image is rendering by self-adaption median filter. Experimental results show that the proposed algorithm can generate panoramic image with high quality.