When the total sky imager (TSI) is used to observe the sky, the structural characteristics of the device will make the collected cloud image information incomplete, which affects the analysis of images. In order to deal with the problems, including the wrong order due to the sharp decrease to zero of the confidence level, the discontinuity of image and the large complexity of time for traversal searching the matching block in the process of repairing ground-based cloud image by the Criminisi algorithm, we propose a ground-based cloud image inpainting method based on the improved Criminisi algorithm in this paper. The calculation formula of priority is improved, and the unique red-blue ratio feature of the ground-based cloud map is introduced as a confidence term, so that the pixel block with more information has higher priority. In the process of searching for the matching block, the searching area is selected based on heuristic information in order to avoid the blocks far away from the block to be repaired and those with low correlation, which effectively shortens the searching time and reduces the time complexity of the algorithm. Experimental results show that the improved Criminisi algorithm has better image restoration effect, can reduce the time complexity and improve the image inpainting efficiency.