The hybrid coding framework of the third generation audio and video coding standard （AVS3） plays an important role in eliminating redundant information in the video time domain/space domain， but needs to be further improved in eliminating perceptual redundancy and further improving coding performance. This paper proposes a just noticeable distortion （JND） prediction model of data-driven pixel domain to optimize AVS3 video encoder under the premise of ensuring the subjective quality of vision. Firstly， based on the current large subjective database of JND， the threshold of perceptive perception distortion in the pixel domain is obtained according to the human eye characteristics. Secondly， the pixel domain JND prediction model based on deep neural network is constructed. Finally， the residual filter established by the predicted pixel domain JND threshold is used to eliminate perceptual redundancy in AVS3 and reduce coding bitrate. The experimental results show that compared with the AVS3 standard test model HPM5.0， the proposed JND model can save up to 21.52% bitrate and an average of 5.11% bitrate.