Speckle noises in medical ultrasound image would decrease the quality of image and bring difficulties to the analysis and diagnosis of the subsequent image. To reduce speckle, here we propose a speckle reduction algorithm based on the adaptive Gauss filter. The algorithm distinguished the image with speckle regions and characteristic regions by a similarity deprived from local characteristic matching between the processing window and a reference speckle area. According to the similarity, this algorithm adjustes adaptively the width of the Gauss filter. Ultrasound phantom testing and in vivo imaging show that the proposed method is effective. It can reduce the numbers of iteration significantly, as well as the speckle and preserve edge.