Using hierarchical structure to classify image set which has the object labels identified by images is one of the important research issues in image automation classification. The previous researches have already implemented the hierarchical structure construction for the labeled images, and now there are only a few methods to consider the influence of the part of the unlabeled images. In this paper, the classical method is extended and optimized, and the hierarchical structure construction and update are realized when some object labels are unknown. The convolutional neural network (CNN) is used to encode these images, and the semi-supervised learning method is proposed. The hierarchical structure of the image set which has known the object labels is constructed according to the traditional algorithm. Through the periodic similarity comparison, the unlabeled images in the hierarchy are clustered. The construction of the semi-supervised layer-wise model (SLM) is realized. This paper adopts the real public data sets. The experimental results show that the SLM can effectively realize the construction and update of the hierarchical structure, and can achieve good prediction classification effect on the smaller scale data sets.