Video stitching is one important branches of computer graphics and computer vision, and it is rooted on the development of static image mosaic technology. However, due to the complexity of the video information, video stitching is different from image mosaic. Aiming at the actual demanding of real-time stitching, a video stitching algorithm of fixed cameras based on control images is proposed. Firstly, the control images are captured in order to calibrate the cameras to get the internal reference and the coordinates of optical center. An improved distortion correction algorithm has been applied to deal with the distortion of the cameras so as to control the distortion of the captured control images. The scale invariant feature transform(SIFT) features of the control images are extracted and processed through a rough matching. Then random sample consensus(RANSAC) algorithm is adopted to reduce the number of error-matching and to fit the image homograhy. The method of lookup table is applied to project each image captured from the cameras onto the panorama image. After gain compensation, the overlapped portion is blended by multi-band technique. Finally, the real-time video stitching is implemented, which can reach the final speed of 25 frames per second.