For multi-user speech transmission scenarios, deep learning based semantic communication system using federated learning based on hypernetworks (DeepSC-FedHN) is proposed. Specifically, an edge server uses a hypernetwork to measure the importance of each module in each local user's semantic encoder, and generates a personalized aggregation weight matrix to update the corresponding model parameters. Meanwhile, the federated learning (FL) algorithm is used to aggregate the model parameters of channel codec and semantic decoder. The experimental results show that the proposed DeepSC-FedHN outperforms the local training scheme, the federated averaging (FedAvg)-based scheme, the FedProx-based scheme, and deep learning based semantic communication system using layer-wised personalized federated learning (DeepSC-pFedLA) in overall performance.