An improved model of class attention network （CA-Net） incorporating a class attention block （CAB） is proposed to extract the primary silicon regions of the microscopic images of Al-Si alloys in this paper. The correlation information of each channel to each class is calculated from the feature map by class attention block， and the correlation information of different classes is fused to generate attention weights for correlating the weights of feature channels with their contributions to the class in the task， thus the representation of important features is enhanced and the interference of irrelevant features is suppressed. Experiments are conducted on the Al-Si alloy microscopic image dataset， and the proposed method obtains results of 94.82%， 90.16%， 94.54%， 98.80%， and 97.97% for Dice coefficient， Jaccard similarity， sensitivity， specificity， and segmentation accuracy， respectively. The proposed CA-Net can effectively improve the segmentation effect of the primary silicon region in Al-Si alloy microscopic images compared with CCNet， SPNet， TA-Net， and other methods.