Whole Slide Images(WSIs) are generally considered the golden standard for pathological diagnosis. Accurate classification of WSIs provides detailed information on tumor type, grade, and stage, which is crucial for cancer prognosis and treatment strategy selection. Currently, in the field of computational pathology, analysis methods based on Multi-Instance Learning (MIL) are becoming the mainstream approach for the classification of Whole-Slide Pathological Images. However, these methods mostly focus on single-scale pathological images, which limits the understanding of the mechanisms of cancer development and progression at different levels. Additionally, the high-resolution nature of pathological images and the information discrepancies across different scales pose challenges for efficiently integrating and analyzing pathological image patches within a single scale as well as across multiple scales. To address these issues, this paper proposes a whole-slide pathology image classification method based on deformable attention and multi-scale multi-instance learning (DMSMIL). Specifically, this method enhances the efficiency of attention computation by designing a deformable attention branch to learn the associations among image patches within the same scale. Meanwhile, an association algorithm based on Optimal Transport (OT) is designed to integrate pathological images across different scales, achieving efficient alignment of multi-scale pathological information. Experimental results on breast cancer subtype classification and lung cancer subtype classification tasks show that the proposed method achieved classification accuracy of 85.39% and 92.00% respectively, showing improved performance compared to mainstream whole slide image classification methods.