Automated Evaluation of Student-Chatbot Interactions in Climate Education Using Large Language Models

Eceay Çeltik; Bora Şenceylan; İbrahim Delen; Gökhan İnce

doi:10.1007/978-3-032-29770-9_51

Automated Evaluation of Student-Chatbot Interactions in Climate Education Using Large Language Models

Eceay Çeltik^*
, Bora Şenceylan
, İbrahim Delen
, Gökhan İnce

^*Corresponding author for this work

Department of Computer Engineering

Istanbul Technical University
Bogazici University

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

This study presents an automated evaluation system for assessing middle school students’ climate-related conversations with educational chatbots using large language models (LLMs). A two-dimensional rubric evaluating content sophistication (0–3) and dialogue reasoning level (A-D) was developed through an iterative, data-driven process. Four LLMs (Claude Sonnet 4, GPT-4o, GPT-5, and Qwen-2.5-Instruct) were assessed on 461 annotated Turkish-language student messages. Claude Sonnet 4 showed the best agreement with human evaluators (Cohen’s Kappa 0.84 for content, 0.75 for dialogue); other models demonstrated moderate agreement. Fleiss’ Kappa across all four raters reached 0.85 for content and 0.74 for dialogue. Within the scope of this study, the results suggest that LLM-based automated evaluation is a promising approach for assessing climate education. The system also extracts knowledge components from student messages, which helps to identify the climate concepts students engage with during interactions.

Original language	English
Title of host publication	Artificial Intelligence in Education - 27th International Conference, AIED 2026, Proceedings
Editors	Emmanuel G. Blanchard, Guanliang Chen, Min Chi, Seiji Isotani
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	469-478
Number of pages	10
ISBN (Print)	9783032297693
DOIs	https://doi.org/10.1007/978-3-032-29770-9_51
Publication status	Published - 2027
Event	27th International Conference on Artificial Intelligence in Education, AIED 2026 - Seoul, Korea, Republic of Duration: 27 Jun 2026 → 3 Jul 2026

Publication series

Name	Lecture Notes in Computer Science
Volume	16585 LNAI
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	27th International Conference on Artificial Intelligence in Education, AIED 2026
Country/Territory	Korea, Republic of
City	Seoul
Period	27/06/26 → 3/07/26

Bibliographical note

Publisher Copyright:
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2027.

Keywords

automated evaluation
chatbot interactions
climate education
educational assessment
large language models

Access to Document

10.1007/978-3-032-29770-9_51

Cite this

Çeltik, E., Şenceylan, B., Delen, İ., & İnce, G. (2027). Automated Evaluation of Student-Chatbot Interactions in Climate Education Using Large Language Models. In E. G. Blanchard, G. Chen, M. Chi, & S. Isotani (Eds.), Artificial Intelligence in Education - 27th International Conference, AIED 2026, Proceedings (pp. 469-478). (Lecture Notes in Computer Science; Vol. 16585 LNAI). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-032-29770-9_51

Çeltik, Eceay ; Şenceylan, Bora ; Delen, İbrahim et al. / Automated Evaluation of Student-Chatbot Interactions in Climate Education Using Large Language Models. Artificial Intelligence in Education - 27th International Conference, AIED 2026, Proceedings. editor / Emmanuel G. Blanchard ; Guanliang Chen ; Min Chi ; Seiji Isotani. Springer Science and Business Media Deutschland GmbH, 2027. pp. 469-478 (Lecture Notes in Computer Science).

@inproceedings{8566ba9aec7947a38e198c90a56f8329,

title = "Automated Evaluation of Student-Chatbot Interactions in Climate Education Using Large Language Models",

abstract = "This study presents an automated evaluation system for assessing middle school students{\textquoteright} climate-related conversations with educational chatbots using large language models (LLMs). A two-dimensional rubric evaluating content sophistication (0–3) and dialogue reasoning level (A-D) was developed through an iterative, data-driven process. Four LLMs (Claude Sonnet 4, GPT-4o, GPT-5, and Qwen-2.5-Instruct) were assessed on 461 annotated Turkish-language student messages. Claude Sonnet 4 showed the best agreement with human evaluators (Cohen{\textquoteright}s Kappa 0.84 for content, 0.75 for dialogue); other models demonstrated moderate agreement. Fleiss{\textquoteright} Kappa across all four raters reached 0.85 for content and 0.74 for dialogue. Within the scope of this study, the results suggest that LLM-based automated evaluation is a promising approach for assessing climate education. The system also extracts knowledge components from student messages, which helps to identify the climate concepts students engage with during interactions.",

keywords = "automated evaluation, chatbot interactions, climate education, educational assessment, large language models",

author = "Eceay {\c C}eltik and Bora {\c S}enceylan and İbrahim Delen and G{\"o}khan İnce",

note = "Publisher Copyright: {\textcopyright} The Author(s), under exclusive license to Springer Nature Switzerland AG 2027.; 27th International Conference on Artificial Intelligence in Education, AIED 2026 ; Conference date: 27-06-2026 Through 03-07-2026",

year = "2027",

doi = "10.1007/978-3-032-29770-9\_51",

language = "English",

isbn = "9783032297693",

series = "Lecture Notes in Computer Science",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "469--478",

editor = "Blanchard, \{Emmanuel G.\} and Guanliang Chen and Min Chi and Seiji Isotani",

booktitle = "Artificial Intelligence in Education - 27th International Conference, AIED 2026, Proceedings",

address = "Germany",

}

Çeltik, E, Şenceylan, B, Delen, İ & İnce, G 2027, Automated Evaluation of Student-Chatbot Interactions in Climate Education Using Large Language Models. in EG Blanchard, G Chen, M Chi & S Isotani (eds), Artificial Intelligence in Education - 27th International Conference, AIED 2026, Proceedings. Lecture Notes in Computer Science, vol. 16585 LNAI, Springer Science and Business Media Deutschland GmbH, pp. 469-478, 27th International Conference on Artificial Intelligence in Education, AIED 2026, Seoul, Korea, Republic of, 27/06/26. https://doi.org/10.1007/978-3-032-29770-9_51

Automated Evaluation of Student-Chatbot Interactions in Climate Education Using Large Language Models. / Çeltik, Eceay; Şenceylan, Bora; Delen, İbrahim et al.
Artificial Intelligence in Education - 27th International Conference, AIED 2026, Proceedings. ed. / Emmanuel G. Blanchard; Guanliang Chen; Min Chi; Seiji Isotani. Springer Science and Business Media Deutschland GmbH, 2027. p. 469-478 (Lecture Notes in Computer Science; Vol. 16585 LNAI).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Automated Evaluation of Student-Chatbot Interactions in Climate Education Using Large Language Models

AU - Çeltik, Eceay

AU - Şenceylan, Bora

AU - Delen, İbrahim

AU - İnce, Gökhan

N1 - Publisher Copyright: © The Author(s), under exclusive license to Springer Nature Switzerland AG 2027.

PY - 2027

Y1 - 2027

N2 - This study presents an automated evaluation system for assessing middle school students’ climate-related conversations with educational chatbots using large language models (LLMs). A two-dimensional rubric evaluating content sophistication (0–3) and dialogue reasoning level (A-D) was developed through an iterative, data-driven process. Four LLMs (Claude Sonnet 4, GPT-4o, GPT-5, and Qwen-2.5-Instruct) were assessed on 461 annotated Turkish-language student messages. Claude Sonnet 4 showed the best agreement with human evaluators (Cohen’s Kappa 0.84 for content, 0.75 for dialogue); other models demonstrated moderate agreement. Fleiss’ Kappa across all four raters reached 0.85 for content and 0.74 for dialogue. Within the scope of this study, the results suggest that LLM-based automated evaluation is a promising approach for assessing climate education. The system also extracts knowledge components from student messages, which helps to identify the climate concepts students engage with during interactions.

AB - This study presents an automated evaluation system for assessing middle school students’ climate-related conversations with educational chatbots using large language models (LLMs). A two-dimensional rubric evaluating content sophistication (0–3) and dialogue reasoning level (A-D) was developed through an iterative, data-driven process. Four LLMs (Claude Sonnet 4, GPT-4o, GPT-5, and Qwen-2.5-Instruct) were assessed on 461 annotated Turkish-language student messages. Claude Sonnet 4 showed the best agreement with human evaluators (Cohen’s Kappa 0.84 for content, 0.75 for dialogue); other models demonstrated moderate agreement. Fleiss’ Kappa across all four raters reached 0.85 for content and 0.74 for dialogue. Within the scope of this study, the results suggest that LLM-based automated evaluation is a promising approach for assessing climate education. The system also extracts knowledge components from student messages, which helps to identify the climate concepts students engage with during interactions.

KW - automated evaluation

KW - chatbot interactions

KW - climate education

KW - educational assessment

KW - large language models

UR - https://www.scopus.com/pages/publications/105043972774

U2 - 10.1007/978-3-032-29770-9_51

DO - 10.1007/978-3-032-29770-9_51

M3 - Conference contribution

AN - SCOPUS:105043972774

SN - 9783032297693

T3 - Lecture Notes in Computer Science

SP - 469

EP - 478

BT - Artificial Intelligence in Education - 27th International Conference, AIED 2026, Proceedings

A2 - Blanchard, Emmanuel G.

A2 - Chen, Guanliang

A2 - Chi, Min

A2 - Isotani, Seiji

PB - Springer Science and Business Media Deutschland GmbH

T2 - 27th International Conference on Artificial Intelligence in Education, AIED 2026

Y2 - 27 June 2026 through 3 July 2026

ER -

Çeltik E, Şenceylan B, Delen İ, İnce G. Automated Evaluation of Student-Chatbot Interactions in Climate Education Using Large Language Models. In Blanchard EG, Chen G, Chi M, Isotani S, editors, Artificial Intelligence in Education - 27th International Conference, AIED 2026, Proceedings. Springer Science and Business Media Deutschland GmbH. 2027. p. 469-478. (Lecture Notes in Computer Science). doi: 10.1007/978-3-032-29770-9_51