In the vector of locally aggregated descriptors (VLAD) method for image representation by residual accumulation, the residual values obtained by each descriptor and the corresponding nearest neighbor codeword are different, and the number of descriptors corresponding to each codeword is uncertain. Thus, there are over cumulative and under cumulative problems. In this paper, a new method for image representation by residual center aggregation of distance clustering is proposed. First, the local descriptors of the database image are extracted, and the codebook is obtained by clustering these descriptors; then, the local descriptors are quantized to the codebook by the nearest neighbor method, and the Euclidean distances between the local descriptors and the corresponding nearest neighbor codeword are obtained. Again, after clustering all the distances and obtaining the central set, the method finds the nearest neighbor of the Euclidean distance between each local descriptor and the nearest neighbor codeword on the central set, obtains the descriptor corresponding to each center in the central set, determines the center of the residual between the descriptor corresponding to each center in the center set and the nearest neighbor codeword, and accumulates and summarizes all the residual centers on each codeword. Finally, all the cumulative vectors corresponding to the codewords are cascaded in order to get the final image representation. The results of image retrieval experiments on the Holidays and UKB datasets show that the proposed image representation method is better than the VLAD method by directly accumulating residuals and performing image representation.