Automated Evaluation of Student-Chatbot Interactions in Climate Education Using Large Language Models

This study presents an automated evaluation system for assessing middle school students’ climate-related conversations with educational chatbots using large language models (LLMs). A two-dimensional rubric evaluating content sophistication (0–3) and dialogue reasoning level (A-D) was developed through an iterative, data-driven process. Four LLMs (Claude Sonnet 4, GPT-4o, GPT-5, and Qwen-2.5-Instruct) were assessed on 461 annotated Turkish-language student messages. Claude Sonnet 4 showed the best agreement with human evaluators (Cohen’s Kappa 0.84 for content, 0.75 for dialogue); other models demonstrated moderate agreement. Fleiss’ Kappa across all four raters reached 0.85 for content and 0.74 for dialogue. Within the scope of this study, the results suggest that LLM-based automated evaluation is a promising approach for assessing climate education. The system also extracts knowledge components from student messages, which helps to identify the climate concepts students engage with during interactions.