In the continuous wave measurement while drilling (MWD) system, the accuracy of error rate prediction is low and the data transfer process is affected by interference signals.A model for error rate prediction of continuous-wave data transmission is proposed by using the improved least squares support vector machines (LS-SVM), and the genetic algorithm is used to search the optimized parameter to improve the prediction accuracy of the model. During establishing the model, Dixon criteria is used to screen the data and improve the error rate prediction accuracy. With small samples, mud continuous-wave data transmission model is established by using Matlabbased on the improved LS-SVM. The simulation results show that the model can avoid falling into local optimization problem effectively, and has strong generalization and prediction ability. Compared with back propagation(BP) and Elman neural network, the model has higher prediction accuracy, so it can be used to predict the error rate of mud continuous-wave data.