For the problem that spectrum resources is increasingly scare in electromagnetic spectrum space， the radio frequency machine learning （RFML） is purposed to design special machine learning models by introducing domain knowledge. It has the advantages of fast， few sample or even zero sample， interpretability and high performance. The state-of-the-art RFML in wireless communication is analyzed from the five layers， which are physical layer， data link layer， network layer， transmission layer and application layer. Moreover， based on the existing achievements， four RFML frameworks （serial/parallel/coupled/feedback dual-driven framework） are summarized by the interaction relationship of the data-driven model and the knowledge-driven model. Finally， the key challenges and open issues are identified and elaborated to facilitate the RFML research and practical applications.