Speaker recognition suffers severe performance degradation under noisy environments. To solve this problem, we propose a novel method based on morphological component analysis. This method employs a universal background dictionary (UBD) to model common variability of all speakers, a speaker dictionary to model special variability of each speaker and a noise dictionary to model variability of environmental noise. These three dictionaries are concatenated to be a big dictionary, over which test speech is sparsely represented and classified. To improve the discriminability of speaker dictionaries, we optimize the speaker dictionaries by removing speaker atoms which are close to the UBD atoms. To ensure the varied noises can be tracked, we design an algorithm to update the noise dictionary with the noisy speech. We also conduct experiments under various noise conditions and the results show that the proposed method can obviously improve the robustness of speaker recognition under noisy environments.