Although automatic detection technology based on machine vision for railway infrastructure has been widely used, fence, as an important safeguard against foreign invasion to ensure safe running of train, has not been detected automaticaly yet, but manually as in traditional inspection. Based on panoramic stitching techniques, we acquire the panorama of the fence along the railway, and then extract gray-level statistical features such as the mean and variance values to construct the two-dimensional statistical histogram of panoramic image. On the bases of these data, we propose a segment method using the maximum entropy of two-dimensional gray mean-variance histogram to achieve rapid fence defect detection from the fence panorama. Experimental results verify the validity and accuracy of the proposed approach and it has the precision ratio of 87.5% and recall ration of 92.1%.