Abstract:A method dealing with direction of arrival(DOA) of coherent signals in the presence of mutual coupling is proposed. Firstly, the generalized steering vector is obtained through DOA matrix method based on the second-order statistics. Secondly, subspace smoothing is applied to generalized steering vector and a linear constrained programming problem is obtained by matrix transform. So DOA and coefficients of mutual coupling can be estimated sequentially. Compared with the fourth-order cumulant method, the computation load is reduced with the second-order statistics. And there is no loss of array elements, which means less array elements are needed in this method.Computer simulations prove the effectivity of the proposed method.