Multi-modal Medical Entity Recognition Based on Multi-scale Attention and Graph Neural Networks
CSTR:
Author:
Affiliation:

1.School of Management, Nanjing University of Posts and Telecommunications, Nanjing 210003, China;2.Jiangsu Province Key Lab of Data Engineering and Knowledge Service, Nanjing 210023, China

Clc Number:

TP391.1;TP183

Fund Project:

  • Article
  • |
  • Figures
  • |
  • Metrics
  • |
  • Reference
  • |
  • Related
  • |
  • Cited by
  • |
  • Materials
  • |
  • Comments
    Abstract:

    With the rapid development of information technology, multi-modal data such as Chinese texts and images in the medical and health field has shown explosive growth. Multi-modal medical entity recognition (MMER) is a key step in multi-modal information extraction, and has attracted great attention recently. Aiming at the problems of image detail loss and insufficient text semantic understanding in multi-modal medical entity recognition tasks, this paper proposes a novel MMER model based on multi-scale attention and dependency parsing graph convolution(MADPG). This model introduces a multi-scale attention mechanism based on ResNet to collaborate to extract visual features fused with different spatial scales and to reduce the loss of important details of medical images. Thus the image feature representation and complementing text semantic information are enhanced. Then, the dependency syntactic structure is used to construct the graph neural network to capture the complex grammatical dependencies between words in medical texts, so as to enrich the semantic expression of texts and promote the deep integration of image text features. Experiments show that the F1 value of the proposed model reaches 95.12% on the multi-modal Chinese medical data set, and the performance of the proposed model is significantly improved compared with the mainstream single- and multi-modal entity recognition models.

    Reference
    Related
    Cited by
Get Citation

HAN Pu, LIU Senling, CHEN Wenqi. Multi-modal Medical Entity Recognition Based on Multi-scale Attention and Graph Neural Networks[J].,2025,40(4):922-933.

Copy
Related Videos

Share
Article Metrics
  • Abstract:
  • PDF:
  • HTML:
  • Cited by:
History
  • Received:January 07,2025
  • Revised:March 03,2025
  • Adopted:
  • Online: August 15,2025
  • Published:
Article QR Code