The design of cosine-modulated subband filter banks can be convertedto a high-dimension nonlinear optimization problem. The conventional iteration design method avoids the complicated solution. The conventional iteration design method controls the aliasing error relying on the stopband attenuation of filter banks, so that the orders of the designed filter banks are generally too high. A modified iteration design method for cosine-modulated filter banks (CMFB) is proposed in this paper. The new cost function consists of system response, stopband energy and the first aliasing factor which is added to eliminate the aliasing error more effectively. This cost function is formulated as a quadratic function whose minimum point can be obtained using a closed-form solution. The minimum point is approached by iteration in this algorithm which is simpler and more realizable than the direct optimization method. Simulation results demonstrate that this algorithm can design the subband filter banks with less aliasing error and shorter length, comparing with the conventional iteration design method.