Infrared and visible image fusion is designed to generate a fused image that provides a more complete description of the scene. In this paper, a novel multi-scale hybrid image decomposition algorithm is proposed, which can effectively extract the small-scale texture detail information of the visible image and the large-scale edge information of the infrared image. The large-scale edge image of the infrared image is used to be segmented and construct the fused weights, which can not only inject the multi-scale infrared spectrum into the visible image effectively, but also preserve the details of the scenes in the visible image. Experimental results show that the proposed algorithm can obtain state-of-the-art performance both in subjective and objective evaluation.