This paper proposes a sign language recognition method based on color-depth videos and complex linear dynamic system (CLDS), which ensures that the time series modeling data can strictly correspond to the original data and accurately characterize the sign language features. Thus the classification precision is improved significantly. The depth videos are used to compensate the missing information of RGB videos, and the motion boundary histogram (MBH) features are extracted from the sign language videos to obtain the feature matrix of each behavior. The feature matrixes are modelled using CLDS method with output of the descriptor M=(A, C) which can uniquely represent the sign language video. Then the similarities between the models are calculated utilizing the subspace angles and the improved KNN algorithm is presented to achieve the final classification result. Experiments on the Chinese sign language dataset (CSL) show that the proposed sign language recognition approach has higher precision than many existing methods.