To address the issues of low quality, lack of realism, and poor diversity in defect images generated by Generative Adversarial Networks (GANs) under small-sample conditions, this paper proposes a defect image generation algorithm, named Defect Image Data Augmentation Generative Adversarial Network (DID-AugGAN), aimed at enhancing defect image data under limited sample conditions. First, to overcome the difficulty of traditional convolutional networks in effectively learning non-rigid features in images from limited datasets, we design a learnable offset convolution to improve the model’s capability in capturing semantic information. Second, to prevent the loss of critical defect features and enhance the correlation among local features, we introduce a multi-scale coordinate attention module, which focuses on defect location information. Third, to enhance the discriminator’s ability to distinguish local details in input images, we redesign its architecture, transforming it from a conventional feedforward network into a UNet-like structure with symmetric encoding and decoding pathways. Finally, we conduct comparative experiments between DID-AugGAN and the baseline algorithm on the Rail-4c track fastener defect dataset, and validate the generated images using the MobileNetV3 classification network. Experimental results demonstrate that the proposed method significantly improves Inception Score (IS) while effectively reducing Fréchet Inception Distance (FID) and Learned Perceptual Image Patch Similarity (LPIPS). Moreover, the classification accuracy and F1-score of MobileNetV3 are also improved. The proposed DID-AugGAN can stably generate high-quality defect images, effectively augment defect data samples, and meet the requirements of downstream tasks.