Social media text normalization for Turkish

Research output: Contribution to journalArticlepeer-review

26 Citations (Scopus)

Abstract

Text normalization is an indispensable stage in processing noncanonical language from natural sources, such as speech, social media or short text messages. Research in this field is very recent and mostly on English. As is known from different areas of natural language processing, morphologically rich languages (MRLs) pose many different challenges when compared to English. Turkish is a strong representative of MRLs and has particular normalization problems that may not be easily solved by a single-stage pure statistical model. This article introduces the first work on the social media text normalization of an MRL and presents the first complete social media text normalization system for Turkish. The article conducts an in-depth analysis of the error types encountered in Web 2.0 Turkish texts, categorizes them into seven groups and provides solutions for each of them by dividing the candidate generation task into separate modules working in a cascaded architecture. For the first time in the literature, two manually normalized Web 2.0 datasets are introduced for Turkish normalization studies. The exact match scores of the overall system on the provided datasets are 70.40 per cent and 67.37 per cent (77.07 per cent with a case insensitive evaluation).

Original languageEnglish
Pages (from-to)835-875
Number of pages41
JournalNatural Language Engineering
Volume23
Issue number6
DOIs
Publication statusPublished - 1 Nov 2017

Bibliographical note

Publisher Copyright:
Copyright © Cambridge University Press 2017.

Fingerprint

Dive into the research topics of 'Social media text normalization for Turkish'. Together they form a unique fingerprint.

Cite this