As a common dimensionality reduction method, the supervised Laplacian discriminant analysis (SLDA) for small size sample achieves a good result of dimensionality reduction via graph embedding discriminant neighborhood analysis. However, when SLDA finds the inter-class and intra-class data points in K nearest neighbors, there might exist an imbalance problem. Additionally, SLDA does not fully consider the inter-class information, which may decrease the performance of SLDA to a certain extent. To address the two problems mentioned above, we propose a double adjacent graph-based discriminant analysis (DAG-DA) algorithm for small size sample. Firstly, the algorithm tries to find K nearest neighbors in inter-class and intra-class samples, respectively, and then uses these K inter-class neighbors and K intra-class neighbors to construct the double adjacent graph. In this way, we can ensure that the adjacent graph contains both the inter-class and intra-class data points and has the same number. Secondly, the algorithm tries to add the intra-class Laplacian scatter matrix into the objective function of SLDA. Thus, the projection matrix obtained by optimization takes the information between classes into account fully. We perform experiments on Yale and ORL human face datasets. Experimental results show that the proposed algorithm can get better performance compared with other methods.