Spatiotemporal XAI: Explaining video regression models in echocardiography videos for ejection fraction prediction

Deep learning has showcased unprecedented success in automating echocardiography analysis. However, most of the deep learning algorithms are hindered at clinical translation due to their black-box nature. This paper aims to tackle this issue by quantitatively evaluating video regression models’ focus on the left ventricle (LV) for ejection fraction (EF) prediction task spatiotemporally in apical 4 chamber (A4C) echocardiograms using a gradient-based saliency method. We performed a quantitative evaluation to assess the ratio of how many of the maximum absolute gradient values of the deep learning models fall on the left ventricle for the video regression task of ejection fraction prediction. Then, we extend the experiment and pick the most important gradients as the segmentation size and check the ratio of intersection. Finally, we picked temporally aligned sub-clips from end diastole to end systole and calculated the expected accuracies of the mentioned metrics in time. All tests are performed in 3 different models with different architectures and results are examined quantitatively. The filtered test set includes 1209 A4C echo videos of with mean EF of 55.5%. Trained models showed 0.73 to 0.83 Pointing Game scores, where it was 0.09 for the baseline random model. m_GT intersection score was 0.46 to 0.50 for the trained models, whereas the random model's score was 0.18. Models have higher pointing game scores on the end diastole and end systole compared to intermediate frames. Transformer based models’ m_GT intersection scores were negatively correlated with their error rate. All models located the left ventricle successfully and their localization performance was generally better in semantically important frames rather than the larger target area. This observation from the spatiotemporal analysis suggests possible clinical relevance to model reasoning.