Hyperparameter Optimization in Deep Learning-Based Object Detection of Branching and Endpoints on 2D Brain Vessel Images

This work presents a deep learning-based object detection technique for identifying branches and endpoints in two-dimensional brain vessel images alongside its hyperparameter optimization. Although traditional image processing methods are feasible and successful, their algorithm complexity increases exponentially with the size of the image and the filters to be applied. In contrast, our deep learning approach has shown significant improvements in accuracy and efficiency, independent of image size and number of filters. We preprocess our dataset of raw mouse brain slices from laboratory environments to remove noise from images and then extract a binary vein network using image processing methods. Finally, we labeled vessel branching and endpoints in 5x5 pixel bounding boxes. All labeled objects on images were converted to COCO format for training and testing to ensure compatibility with deep learning algorithms. Our research focused on using the Faster R-CNN method in the Detectron2 framework, which has been successful in our previous work. Evaluation using the intersection over union (IoU) metric underscores the robustness of our approach, and we achieved a success rate of over 90%. We employed Optuna for hyperparameter optimization to further enhance our model, focusing on three key hyperparameters: base learning rate, maximum iterations, and batch size per image. We systematically refined these hyperparameters by running 50 training and test runs separately, significantly improving model performance to over 98%. Our findings highlight the transformative potential of deep learning in neuroimaging analysis and promise significant advances in diagnostic and research applications in neurovascular studies.