Full-duplex model can transmit data during one time slot and frequency, which doubles the transmission data rate of relay systems compared with half-duplex. To improve the transmission rate of full-duplex multi-input and multi-output(MIMO) relay system with amplify-and-forward(AF) strategy, a beamforming scheme of maximizing rate is designed using alternating iterative structure and gradient descent method (GDM). We derive the beamforming expressions at relay station, and design an minimum mean square error(MMSE) iterative algorithm as the initial condition. Simulation results reveal that the proposed alternative iterative structure converge quickly and is better than some existing algorithm like zero-forcing(ZF), minimum mean square error(MMSE), maximizing signal-to-leakage-plus-noise ratio(Max-SLNR) algorithm.