ViBERTgrid BiLSTM-CRF: Multimodal Key Information Extraction from Unstructured Financial Documents

Furkan Pala; Mehmet Yasin Akpınar; Onur Deniz; Gülşen Eryiğit

doi:10.1007/978-3-031-74643-7_23

ViBERTgrid BiLSTM-CRF: Multimodal Key Information Extraction from Unstructured Financial Documents

Furkan Pala^*, Mehmet Yasin Akpınar, Onur Deniz, Gülşen Eryiğit

^*Bu çalışma için yazışmadan sorumlu yazar

Yapay Zeka ve Veri Mühendisliği Bölümü

Araştırma sonucu: Kitap/Rapor/Konferans Bildirisinde Bölüm › Konferans katkısı › bilirkişi

Özet

Multimodal key information extraction (KIE) models have been studied extensively on semi-structured documents. However, their investigation on unstructured documents is an emerging research topic. The paper presents an approach to adapt a multimodal transformer (i.e., ViBERTgrid previously explored on semi-structured documents) for unstructured financial documents, by incorporating a BiLSTM-CRF layer. The proposed ViBERTgrid BiLSTM-CRF model demonstrates a significant improvement in performance (up to 2% points) on named entity recognition from unstructured documents in financial domain, while maintaining its KIE performance on semi-structured documents. As an additional contribution, we publicly released token-level annotations for the SROIE dataset in order to pave the way for its use in multimodal sequence labeling models.

Orijinal dil	İngilizce
Ana bilgisayar yayını başlığı	Machine Learning and Principles and Practice of Knowledge Discovery in Databases - International Workshops of ECML PKDD 2023, Revised Selected Papers
Editörler	Rosa Meo, Fabrizio Silvestri
Yayınlayan	Springer Science and Business Media Deutschland GmbH
Sayfalar	307-322
Sayfa sayısı	16
ISBN (Basılı)	9783031746420
DOI'lar	https://doi.org/10.1007/978-3-031-74643-7_23
Yayın durumu	Yayınlandı - 2025
Etkinlik	Joint European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2023 - Turin, Italy Süre: 18 Eyl 2023 → 22 Eyl 2023

Yayın serisi

Adı	Communications in Computer and Information Science
Hacim	2137 CCIS
ISSN (Basılı)	1865-0929
ISSN (Elektronik)	1865-0937

???event.eventtypes.event.conference???

???event.eventtypes.event.conference???	Joint European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2023
Ülke/Bölge	Italy
Şehir	Turin
Periyot	18/09/23 → 22/09/23

Bibliyografik not

Publisher Copyright:
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.

Belgeye Erişim

10.1007/978-3-031-74643-7_23

Diğer Dosyalar ve Linkler

Link to publication in Scopus

Alıntı Yap

Pala, F., Akpınar, M. Y., Deniz, O., & Eryiğit, G. (2025). ViBERTgrid BiLSTM-CRF: Multimodal Key Information Extraction from Unstructured Financial Documents. In R. Meo, & F. Silvestri (Eds.), Machine Learning and Principles and Practice of Knowledge Discovery in Databases - International Workshops of ECML PKDD 2023, Revised Selected Papers (pp. 307-322). (Communications in Computer and Information Science; Hac. 2137 CCIS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-74643-7_23

Pala, Furkan ; Akpınar, Mehmet Yasin ; Deniz, Onur et al. / ViBERTgrid BiLSTM-CRF : Multimodal Key Information Extraction from Unstructured Financial Documents. Machine Learning and Principles and Practice of Knowledge Discovery in Databases - International Workshops of ECML PKDD 2023, Revised Selected Papers. editör / Rosa Meo ; Fabrizio Silvestri. Springer Science and Business Media Deutschland GmbH, 2025. pp. 307-322 (Communications in Computer and Information Science).

@inproceedings{6763afa4d3de4347937e2eea8eeedaf6,

title = "ViBERTgrid BiLSTM-CRF: Multimodal Key Information Extraction from Unstructured Financial Documents",

abstract = "Multimodal key information extraction (KIE) models have been studied extensively on semi-structured documents. However, their investigation on unstructured documents is an emerging research topic. The paper presents an approach to adapt a multimodal transformer (i.e., ViBERTgrid previously explored on semi-structured documents) for unstructured financial documents, by incorporating a BiLSTM-CRF layer. The proposed ViBERTgrid BiLSTM-CRF model demonstrates a significant improvement in performance (up to 2% points) on named entity recognition from unstructured documents in financial domain, while maintaining its KIE performance on semi-structured documents. As an additional contribution, we publicly released token-level annotations for the SROIE dataset in order to pave the way for its use in multimodal sequence labeling models.",

keywords = "Multimodal Information Extraction, Natural Language Processing, Unstructured Financial Documents",

author = "Furkan Pala and Akpınar, {Mehmet Yasin} and Onur Deniz and G{\"u}l{\c s}en Eryiğit",

note = "Publisher Copyright: {\textcopyright} The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.; Joint European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2023 ; Conference date: 18-09-2023 Through 22-09-2023",

year = "2025",

doi = "10.1007/978-3-031-74643-7_23",

language = "English",

isbn = "9783031746420",

series = "Communications in Computer and Information Science",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "307--322",

editor = "Rosa Meo and Fabrizio Silvestri",

booktitle = "Machine Learning and Principles and Practice of Knowledge Discovery in Databases - International Workshops of ECML PKDD 2023, Revised Selected Papers",

address = "Germany",

}

Pala, F, Akpınar, MY, Deniz, O & Eryiğit, G 2025, ViBERTgrid BiLSTM-CRF: Multimodal Key Information Extraction from Unstructured Financial Documents. in R Meo & F Silvestri (eds), Machine Learning and Principles and Practice of Knowledge Discovery in Databases - International Workshops of ECML PKDD 2023, Revised Selected Papers. Communications in Computer and Information Science, hac. 2137 CCIS, Springer Science and Business Media Deutschland GmbH, pp. 307-322, Joint European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2023, Turin, Italy, 18/09/23. https://doi.org/10.1007/978-3-031-74643-7_23

ViBERTgrid BiLSTM-CRF: Multimodal Key Information Extraction from Unstructured Financial Documents. / Pala, Furkan; Akpınar, Mehmet Yasin; Deniz, Onur et al.
Machine Learning and Principles and Practice of Knowledge Discovery in Databases - International Workshops of ECML PKDD 2023, Revised Selected Papers. ed. / Rosa Meo; Fabrizio Silvestri. Springer Science and Business Media Deutschland GmbH, 2025. p. 307-322 (Communications in Computer and Information Science; Hac. 2137 CCIS).

Araştırma sonucu: Kitap/Rapor/Konferans Bildirisinde Bölüm › Konferans katkısı › bilirkişi

TY - GEN

T1 - ViBERTgrid BiLSTM-CRF

T2 - Joint European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2023

AU - Pala, Furkan

AU - Akpınar, Mehmet Yasin

AU - Deniz, Onur

AU - Eryiğit, Gülşen

PY - 2025

Y1 - 2025

N2 - Multimodal key information extraction (KIE) models have been studied extensively on semi-structured documents. However, their investigation on unstructured documents is an emerging research topic. The paper presents an approach to adapt a multimodal transformer (i.e., ViBERTgrid previously explored on semi-structured documents) for unstructured financial documents, by incorporating a BiLSTM-CRF layer. The proposed ViBERTgrid BiLSTM-CRF model demonstrates a significant improvement in performance (up to 2% points) on named entity recognition from unstructured documents in financial domain, while maintaining its KIE performance on semi-structured documents. As an additional contribution, we publicly released token-level annotations for the SROIE dataset in order to pave the way for its use in multimodal sequence labeling models.

AB - Multimodal key information extraction (KIE) models have been studied extensively on semi-structured documents. However, their investigation on unstructured documents is an emerging research topic. The paper presents an approach to adapt a multimodal transformer (i.e., ViBERTgrid previously explored on semi-structured documents) for unstructured financial documents, by incorporating a BiLSTM-CRF layer. The proposed ViBERTgrid BiLSTM-CRF model demonstrates a significant improvement in performance (up to 2% points) on named entity recognition from unstructured documents in financial domain, while maintaining its KIE performance on semi-structured documents. As an additional contribution, we publicly released token-level annotations for the SROIE dataset in order to pave the way for its use in multimodal sequence labeling models.

KW - Multimodal Information Extraction

KW - Natural Language Processing

KW - Unstructured Financial Documents

UR - http://www.scopus.com/inward/record.url?scp=85216011471&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-74643-7_23

DO - 10.1007/978-3-031-74643-7_23

M3 - Conference contribution

AN - SCOPUS:85216011471

SN - 9783031746420

T3 - Communications in Computer and Information Science

SP - 307

EP - 322

BT - Machine Learning and Principles and Practice of Knowledge Discovery in Databases - International Workshops of ECML PKDD 2023, Revised Selected Papers

A2 - Meo, Rosa

A2 - Silvestri, Fabrizio

PB - Springer Science and Business Media Deutschland GmbH

Y2 - 18 September 2023 through 22 September 2023

ER -

Pala F, Akpınar MY, Deniz O, Eryiğit G. ViBERTgrid BiLSTM-CRF: Multimodal Key Information Extraction from Unstructured Financial Documents. In Meo R, Silvestri F, editörler, Machine Learning and Principles and Practice of Knowledge Discovery in Databases - International Workshops of ECML PKDD 2023, Revised Selected Papers. Springer Science and Business Media Deutschland GmbH. 2025. p. 307-322. (Communications in Computer and Information Science). doi: 10.1007/978-3-031-74643-7_23