EDU: English Document Understanding, a novel transformer-based approach for recognition of handwritten academic text

Abstract

Understanding handwritten text from images plays a critical role in various domains including education, healthcare, and transportation. Contemporary text recognition systems are mostly based on traditional Optical Character Recognition (OCR) methods, which need well-structured printed text. Such systems are inadequate in the realm of education where handwritten content prevails. With this research work, a novel framework is presented intermingling multiple deep learning models for effective understanding of handwritten English script. A transformer-based hand-text recognition engine (HRE) is employed to detect and recognize handwritten text. To attain higher-level language understanding, Microsoft’s Phi-2 language model is incorporated for semantic analysis. Uncertainty in language understanding is estimated by standard Bayesian inference through MC-dropout-based sampling. Furthermore, the approach of Parameter-Efficient Fine-Tuning (PEFT) is used to minimize overhead in the encoder and decoder modules. The method requires only 1.9% of the trainable parameters, ensuring speedy training and improved recognition accuracy. Results of the systematic evaluation confirm state-of-the-art performance of the method, with remarkable scores of 0.963, 0.954, and 0.958 achieved for the standard metrics of precision, recall, and F1, respectively. Moreover, the model attains a Character Error Rate (CER) of 2.41% and a Word Error Rate (WER) of 8.13%. These low error rates demonstrate the effectiveness of the proposed framework in accurately recognizing and understanding handwritten scripts. Besides automated grading and assignment analysis, the framework is extendable to a wide range of digital archival and legal document analysis.