The technique of portrait segmentation plays an important role in human face recognition, such as 3D human body reconstruction, motion capture and other practical applications, and its reliability directly affects the effect of subsequent processing. Based on the target segmentation algorithm of label propagation theory, an improved portrait segmentation algorithm is proposed. Firstly, this paper introduces the fuzzy set theory to improve the ability of complex background image segmentation. Secondly, this paper uses the super-pixel over-segmentation to preprocess, and uses the over-segmentation result to optimize the similarity definition for improving the smoothness and reliability of the segmented contour. Experimental results show that, compared with the original label propagation algorithm, the presented algorithm has higher segmentation accuracy, and the segmentation contour with higher smoothness is retained.