In order to further improve the performance of speaker recognition system based on the GMM independent of text, a new speaker recognition method is applied to the speaker recognition system with small samples and text independent. Aiming at the large quantity demanded of training data during the modeling of the GMM, the advantages of the fuzzy set theory, vector quantization and the GMM are considered. Then through replacing the output probability function in the traditional GMM with the error scale of the fuzzy VQ, the requirements of the training data amount are reduced while improving the accuracy and recognition speed of the model. Meanwhile as a result of the fuzzy set theory playing a role of ＂plastic date＂, the similarity in the data of the target speakers is enhanced. Experimental results exhibit that the speaker recognition system of the method for the small sample data, achieves a superior recognition performance than the traditional speaker recognition system based on the GMM.