Vietnamese characters which are composed of Latin characters and diacritic symbols make recognition more challenging. On the one hand， diacritic symbols are more likely to lead to attention drift. On the other hand， Vietnamese characters include many categories， and the differences between characters are small， for example some characters only differ from diacritical symbols， which further increases difficulty of recognition. Based on the decoupled attention network （DAN） algorithm， this paper designs a visual feature and sequence feature fusion module （VSFM）， which utilizes bidirectional gated recurrent unit （Bi-GRU） to model sequences in the horizontal and vertical directions， further alleviating attention drift and enhancing correlation between diacritics and Latin characters. And an enhanced decoupled text decoder module （ETDM） is designed， which employs more feature information to identify similar characters more effectively. A series of experiments validate the effectiveness of the proposed method.