An algorithm is proposed based on the affine projection (AP) adaptive filter theory for digital predistortion (DPD) system. A method of look-up table (LUT) with complex gain is applied to the system. The effects of step size parameter and the number of multiple constraints on the affine projection algorithm convergence rate are analyzed. The performance of AP algorithm and the normalized least mean square (NLMS) algorithm is compared. Simulation result shows the AP algorithm has a faster convergence rate when the numbers of step size parameter and multiple constraints are greater. The adjacent channel power ratio (ACPR) of AP algorithm is -59.3 dB while the one of NLMS algorithm is -44.2 dB at offset normalized frequency of 5 MHz with same iteration times, step size parameter and regularization coefficient. The performance of AP algorithm is better than that of NLMS.