Unlike RGB images， pixels in depth images represent the distance from the acquisition device to the points of the scene， and the direct use of inpainting methods for the natural image can not effectively restore the scene structure of missing areas in deep images. This paper proposes a two-stage code structure generation counter-network to solve the problem of deep image inpainting. Unlike standard generative adversarial network （GAN） models， the generator network in this paper includes depth build G1 and depth repair G2 modules. G1 obtains the predicted depth from the RGB image， replacing the missing area of the depth image to be repaired， and ensuring the local structure consistency of the repair area. G2 introduces RGB image edge structure to ensure global structure consistency. The consistency of the missing areas， which is not considered in the existing image inpainting methods， is solved by a structure consistency attention module （SCA） embeded into G2. The proposed depth image repairing model is verified on several mainstream data sets， showing that the effect of structural constraints， and the combination of the generator and discriminator is evident.