In public places, pedestrians always move by groups, which are called as motion groups. A motion group with the highest visual salienoy is the focus of the scene understanding. A new measurement of motion group′s visual saliency is defined in this paper, and the measurement includes four descriptors as follows: scale, speed, group compactness and group variation of different frame. Based on these descriptors, a new method is proposed for detecting the highest visual saliency group. Firstly, the optical flow method is used to compute optical flow vectors. Then, hierarchical clustering algorithm is used to group the crowd. Finally, the values of each group′s visual saliency are computed to find the group with the highest visual saliency value. Experimental results show that the proposed method can detect the highest visual saliency groups effectively. The research can be applied to computer visual fields such as crowd scene understanding, crowd motion analysis and crowd scene classification etc.