In this paper, a coarse-to-fine traffic sign recognition algorithm is proposed to alleviate the conflict between recognition precision and time consumption. In the coarse classification, a traffic sign region is represented with color name-histogram of gradient (CN-HOG) descriptors to describe its color and shape features. A linear support vector machine (SVM) classifier is used to classify the region into different categories:prohibitory, warning, mandatory, release of prohibitory and others. In the fine classification, the different fusion methods of color and shape features in Bag of Words model are discussed and the color-shape early fusion method is employed to combine the CN and scale-invariant feature transform (SIFT) descriptors. The final class labels of the region are obtained by Gaussian kernel SVM classifier. Experiments in public dataset show that the proposed algorithm satisfies real-time practice and meanwhile achieves a high classification precision of 99.15%.