Randomness tests of cryptographic algorithm is analyzed. In order to test and identify Zigbee protocol security of cryptographic algorithm in the internet of things(IOT), test modes are organized and divided combining with the characteristics of Zigbee networks and binary matrix rank theory test, and randomness tests method of Zigbee protocol are proposed based on matrix probability test. The method solves the one sidedness of binary matrix rank test to simply judge random sequence by linear correlation. It can determine whether the Zigbee protocol encryption mechanism and encryption strength are implemented or not. Simulation results show that th is algorithm has less errors, and higher reliability, which is much more convincing. The proposed method provides theoretical and practical guidance for information security evaluation of IOT.