Aiming that the per-user unitary and rate control(PU2RC) algorithm cannot work well in the low number of users and high signal to noise ratio(SNR) in multi-user multiple-input multiple-output （MU-MIMO）system, a downlink adaptive transmission scheme with reconstructing channel information and searching optimal number of service users at the base station is proposed. The base station firstly reconstructs the downlink channel vector using the norm of channel vector and channel direction information (CDI) fed back by users. Then, the user set corresponding to each codebook matrix is expanded by sharing the users among the similar codebook vectors. Finally, the optimal set of service users is selected to maximize the system sum rate. The proposed scheme improves the probability of multi-user transmission with the low number of users and obtains the tradeoff between the inter-user interference and the number of service users. The sum rate of the proposed system is higher than that of the traditional PU2RC algorithm and the existing improved schemes.