Since the different parts of multimode data belong to multiple modes and there are complementary relationships between them, the traditional analysis and processing techniques cannot fuse the data from different modes and their relationships. In order to solve the problems of modeling, representation and storage of multimode data, a new data model of modal structure graph is proposed to model the multimode medical data. The model utilizes the graph structure to model and represent the modes and their relationships in the multimode data. And based on the model, the paper proposes a parallel loading algorithm, which can extract the data of different modes and relationships, and store them in a graph database. Experimental results by using batch medical data files show that the proposed algorithm can obtain good data processing efficiency.