Exploring Turkish Idiomaticity with Large Language Models

Idioms are challenging expressions for language models, as their meanings cannot be inferred from their component words, often leading to reduced performance on downstream tasks. While recent studies have shown that both encoder-only and decoder-only models can be effective in detecting idiomaticity in English, research on idiomaticity in Turkish remains limited. In this study, we evaluate a diverse set of encoder-only and decoder-only models on the tasks of idiomaticity detection and idiom identification for Turkish. Encoder-only models are fine-tuned in a supervised sequence labeling setup, while general purpose pre-trained decoder-only models are evaluated under zero-shot and few-shot prompting settings. Our results show that encoder-only models, particularly mDeBERTa-V3, deliver the strongest performance overall, achieving an F1 score of 90 % for idiomaticity detection task and a word-level F1 of 96 % with 85 % exact match accuracy in the idiom identification task. Among decoder-only models, OpenAI-o3 comes closest, reaching 84 % F1 and 86 % exact match accuracy under few-shot prompting, outperforming all other decoder-only models (i.e., DeepSeek-R1, Gemma3, LLaMa-3.1, Qwen3, Claude-3.7-Sonnet). These findings highlight the continued advantage of supervised fine-tuning, while also demonstrating the emerging potential of few-shot prompting for adapting decoder-only models to idiomaticity detection and idiom identification tasks in Turkish.