The estimation accuracy of the mixing matrix is influenced by the sources sparsity in the underdetermined mixtures. Based on the analytical results of th e single and non single frequencies for source signals, through clustering the co l umn vectors composed by the ratios between the observation signal frequency amp litudes, a new method for the mixing matrix estimation is proposed when the sources are little sparse to each other. Considering the non stability brought by the par t ial convergence of the classical clustering algorithm, the genetic and simulated annealing clustering algorithm possessing the global convergence characteristic is u sed to prove the robustness of the clustering result. The experiment results s how that the proposed estimation method and the clustering algorithm can provide good estimation performance under different underdetermined conditions and different noises.