As the precondition of correcting VCO nonlinearity precisely, the detection of VCO nonlinearity is worth to study theoretically and practically. For the noise immunity and the real-time performance of the method based on element analytic and FrFT still can not satisfy the desire, a VCO nonlinearity detection method based on overlapped subsection and FrFT is proposed. The noise immunity is improved by overlapped subsection. The real-time performance is enhanced by adopting the fast converged golden section method to search the peak value in fractional Fourier transform domain. Simulation shows the better noise immunity of the proposed method. The RMSE of the proposed method is about 1/2 of the present method when the overlapped ratio is 0.25 and the real-time performance of the proposed method will exceed the present one’s while the overlapped ratio is less than 2/3.