This paper proposes a fast algorithm for sparse decomposition of linear FM (LFM) signal to solve the deficiency of traditional methods that there are a large number of atoms. The atoms in the over-complete dictionary are structured based on linear FM signal, and fast sparse decomposition of LFM is completed by using combined dictionary. By analysis, the number of atoms in two over-complete dictionaries is much smaller than that in one over-complete dictionaries, and the most matched atom in another dictionary can be found with the use of searching the maximum based on fast Fourier transform. Simulation results show that the computational efficiency of the proposed method is better than that of three other algorithms using one over-completed dictionary, and the sparsity is better.