EDU: English Document Understanding, a novel transformer based approach for recognition of handwritten academic text

Abstract

Understanding handwritten text from images plays a critical role in various domains including education, healthcare, and transportation. The contemporary text recognition systems are mostly based on the traditional Optical Character Recognition (OCR) methods which need well-structured printed text. Such systems are inadequate in the realm of education where handwritten contents are prevalent. With this research work a novel framework is presented intermingling multiple deep learning models for effective understanding of handwritten English script. A transformer based hand-text recognition engine (HRE) is employed to detect and recognize handwritten text. To attain higher-level language understanding, the Microsoft’s Phi-2 language model is incorporated for semantic analysis. Uncertainty in language understanding is estimated by the standard Bayesian inference through MC-dropout based sampling. Furthermore, the approach of Parameter-Efficient Fine-Tuning (PEFT) is followed to have minimal overhead in the encoder and decoder modules. The method requires only 1.9% of the trainable parameters, ensuring speedy training and improved recognition accuracy. Results of the systematic evaluation affirms state-ofthe-art performance of the method as noteworthy scores of 0.963, 0.954 and 0.958 are achieved for the standard metrics of precision, recall, and F1 respectively. Moreover, the model attains a Character Error Rate (CER) of 2.41% and a Word Error Rate (WER) of 8.13%. These low error rates demonstrate effectiveness of the proposed framework in accurately recognizing and understanding handwritten scripts. Besides automated grading and assignment analysis, the framework is extendable to widerange digital archival and legal document analysis.