Aiming at the mismatch between training speech and test speech from different speaking manners, a kind of feature processing algorithm is proposed based on joint factor analysis and feature mapping. The speaking mode information is extracted by joint factor analysis algorithm, then the speaking mode factor and space are optimized. Before training and test，the feature is mapped by speaking mode inf ormation to reduce the speaking mode effects. The experimental results show that the proposed algorithm can effectively extract the speaker information of the training speech, and improve the recognition rate of whispered speaker recognition system.