To overcome the problem of decoding errors of secondary surveillance radar (SSR) due to the phenomenon of garble, an iterative projection algorithm for separating the garbled replies is proposed. Firstly, a signal model for the garbled replies is established in which the influence factors such as antenna structure, residual carrier frequency and so on are taken a full consideration, and an optimization model with double matrix variables is proposed based on the maximum likelihood estimation of the unknown variable in the model. Then, the initial value of replies is estimated by the algorithm of noncircular complex fast independent component analysis (nc-FastICA). And then, the iterative projection algorithm based on the zero/constant modulus (ZCM) property of SSR replies is proposed to separate garbled SSR replies. Numerical simulations show that the proposed method can separate multiple replies accurately in the complex environment such as shorter distance of the transponders and inaccuracy in the antenna and so on, and it outperforms the algorithm of nc-FastICA obviously in the separation performance.