Three-dimensional （3D） reconstruction from single-view remote sensing images is an unsolvable problem， which often requires a lot of manual experience to supplement the missing information to construct a complete 3D model. To solve this problem， a 3D reconstruction method of single-view remote sensing image based on semantic segmentation and fusion residual U-Net is proposed. The method includes two stages： Semantic segmentation and height estimation of single-view remote sensing images. In the semantic segmentation stage， U-Net is used to determine the property of ground objects. On this basis， U-Net is improved to estimate the height of remote sensing image. The anchoring height regression is combined with semantic features to improve the reconstruction accuracy. Specifically， in order to improve U-Net， the feature extraction capability of encoder is enhanced by embedding residual blocks with different numbers and channels， and the decoder output layer is modified to adapt to the height regression task， so as to achieve pixel-to-pixel prediction of digital surface model （DSM） height values of remote sensing images. The results of root mean square error （RMSE） of 2.751 m and mean absolute error （MAE） of 1.446 m are obtained on the published US3D data set， and the reconstructed results are superior to those of other networks， confirming that the method can realize 3D estimation based on single-view remote sensing images and can reconstruct the distribution structure of ground objects.