The speaker clustering is an important process of speaker diarization, yet traditional method for hierarchical agglomerative clustering (HAC) with distance measurement based on Bayesian information criterion (BIC) can lead to the clustering error propagation. To solve this problem, step by step algorithm is proposed, when the minimum BIC distance between segments exceeds a predefined threshold, or the number of the categories on hierarchical clustering reaches a certain number. The current clustering result as the initial class center, and then variational Bayesian method will be exploited to tune the speaker segments among the categories iteratively. Finally, the number of speaker is determined according to the probabilistic linear discriminant analysis (PLDA) score threshold. Experiments on national institute of standards and technology (NIST) 08 summed test set show that this method improves the "class purity" and "speaker purity" compared with conventional algorithms. Moreover, performance of speaker diarization is relatively improved by 27.6%.