Automatic modulation recognition (AMR) is a key component in the signal processing chain between signal detection and demodulation. To address the heavy dependence on labeled data in practical radio environments, which limits reliable recognition of previously unseen (unknown) modulation types, this paper proposes an unsupervised modulation recognition method for communication signals based on Bootstrap Your Own Latent (BYOL) self-supervised representation learning and a contrastive clustering mechanism. Conventional modulation recognition algorithms are predominantly supervised and require large amounts of labeled data, which is often costly or even infeasible to obtain in real-world scenarios. Unsupervised and self-supervised approaches can alleviate this issue, but existing methods typically suffer from insufficient representation learning capacity and suboptimal clustering performance, and thus struggle to cope with complex channel conditions. The proposed method does not rely on any manual labels. First, we employ a BYOL framework with a dual-branch architecture to encode different sub-segments of the same signal, thereby learning intrinsic and stable representations in a self-supervised manner. Then, instance-level and cluster-level contrastive learning modules are introduced: the former enhances feature consistency across different augmented views of the same signal, while the latter improves the separability of different modulation types in the feature space, thereby enabling high-accuracy blind clustering of unknown modulation types. Experiments conducted on the public RadioML2018.01A dataset show that the proposed method outperforms existing algorithms by more than 10% in various clustering evaluation metrics. Ablation studies further confirm the critical roles of the BYOL module and the contrastive clustering mechanism in improving overall performance. Confusion-matrix analysis demonstrates that, at 10 dB, the proposed method achieves near-ideal recognition accuracy for typical modulation types such as AM-DSB-WC, FM, and GMSK, and also exhibits strong robustness and anti-confusion capability for other more challenging modulation types. In summary, the proposed unsupervised modulation recognition method effectively alleviates the problem of label scarcity in real wireless communication scenarios and shows strong potential for practical deployment.