A Deep Learning-Based Pipeline for Teaching Control Theory: Transforming Feedback Control Systems on Whiteboard into MATLAB

The difficulty in teaching control theory is that the lecturer must not only provide all the theoretical concepts but also visualize the control system in time and frequency domains. In control system courses, the visualizations are usually provided with roughly sketches on whiteboards and thus might be difficult to understand. In this paper, a Deep Learning (DL) based pipeline is proposed that is capable to recognize Handwritten Feedback Control Architectures (HFCAs) on the whiteboard and to transform them into Matlab^® for visualization and analysis of control systems interactively. The proposed DL pipeline consists of 5 main steps that take up the frameworks of deep learning, pattern recognition and image processing. The main challenges of constructing such a pipeline are the uncertainties resulting from the lecturer's handwriting quality and lighting conditions in the classroom, which can be seen as inter- and intra-quality uncertainties. Therefore, we employed and trained deep Convolutional Neural Networks (CNNs) to recognize the HFCAs with a high performance. In the training of deep CNNs, we integrated the transfer learning approach with the deep CNN ResNet-50. To capture the inter- and intra-quality uncertainties, we constructed an image dataset of HFCAs collected from control system lecturers, who have different levels of experience, in a small-sized classroom under different lighting conditions.