In order to overcome the weakness of least mean square error (LMS) and the recursive least squares(RLS), a new neural network speech watermarking method based on short-term energy and least relative mean square error(LRMS) was proposed in this paper. First and foremost, a synchronization sequence was embedded into the first frame of the speech. In addition, calculated the short-term energy of each frame and performed DWT(discrete wavelet transform) for the speech frame larger than the threshold. At last, the watermark was embedded and extracted via the trained LRMS based neural network. The balance of the watermarking capacity and robustness was achieved by setting a reasonable short-term energy threshold and the network converged fast by LM algorithm. The theoretical analysis and the experimental results show that, compared with [8], the improved neural network scheme converges faster and gets better robustness against attacks such as additive noise, low-pass filtering, re-sampling, re-quantifying, et al. Moreover, the performance achieves 5% increase on average.