ViBERTgrid BiLSTM-CRF: Multimodal Key Information Extraction from Unstructured Financial Documents

Furkan Pala; Mehmet Yasin Akpınar; Onur Deniz; Gülşen Eryiğit

doi:10.1007/978-3-031-74643-7_23

ViBERTgrid BiLSTM-CRF: Multimodal Key Information Extraction from Unstructured Financial Documents

Furkan Pala^*
, Mehmet Yasin Akpınar
, Onur Deniz
, Gülşen Eryiğit

^*Corresponding author for this work

Department of Artificial Intelligence and Data Engineering

RandD and Special Projects Department
Istanbul Technical University

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

1 Citation (Scopus)

Abstract

Multimodal key information extraction (KIE) models have been studied extensively on semi-structured documents. However, their investigation on unstructured documents is an emerging research topic. The paper presents an approach to adapt a multimodal transformer (i.e., ViBERTgrid previously explored on semi-structured documents) for unstructured financial documents, by incorporating a BiLSTM-CRF layer. The proposed ViBERTgrid BiLSTM-CRF model demonstrates a significant improvement in performance (up to 2% points) on named entity recognition from unstructured documents in financial domain, while maintaining its KIE performance on semi-structured documents. As an additional contribution, we publicly released token-level annotations for the SROIE dataset in order to pave the way for its use in multimodal sequence labeling models.

Original language	English
Title of host publication	Machine Learning and Principles and Practice of Knowledge Discovery in Databases - International Workshops of ECML PKDD 2023, Revised Selected Papers
Editors	Rosa Meo, Fabrizio Silvestri
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	307-322
Number of pages	16
ISBN (Print)	9783031746420
DOIs	https://doi.org/10.1007/978-3-031-74643-7_23
Publication status	Published - 2025
Event	23rd Joint European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2023 - Turin, Italy Duration: 18 Sept 2023 → 22 Sept 2023

Publication series

Name	Communications in Computer and Information Science
Volume	2137 CCIS
ISSN (Print)	1865-0929
ISSN (Electronic)	1865-0937

Conference

Conference	23rd Joint European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2023
Country/Territory	Italy
City	Turin
Period	18/09/23 → 22/09/23

Bibliographical note

Publisher Copyright:
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.

Keywords

Multimodal Information Extraction
Natural Language Processing
Unstructured Financial Documents

Access to Document

10.1007/978-3-031-74643-7_23

Cite this

Pala, F., Akpınar, M. Y., Deniz, O., & Eryiğit, G. (2025). ViBERTgrid BiLSTM-CRF: Multimodal Key Information Extraction from Unstructured Financial Documents. In R. Meo, & F. Silvestri (Eds.), Machine Learning and Principles and Practice of Knowledge Discovery in Databases - International Workshops of ECML PKDD 2023, Revised Selected Papers (pp. 307-322). (Communications in Computer and Information Science; Vol. 2137 CCIS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-74643-7_23

Pala, Furkan ; Akpınar, Mehmet Yasin ; Deniz, Onur et al. / ViBERTgrid BiLSTM-CRF : Multimodal Key Information Extraction from Unstructured Financial Documents. Machine Learning and Principles and Practice of Knowledge Discovery in Databases - International Workshops of ECML PKDD 2023, Revised Selected Papers. editor / Rosa Meo ; Fabrizio Silvestri. Springer Science and Business Media Deutschland GmbH, 2025. pp. 307-322 (Communications in Computer and Information Science).

@inproceedings{6763afa4d3de4347937e2eea8eeedaf6,

title = "ViBERTgrid BiLSTM-CRF: Multimodal Key Information Extraction from Unstructured Financial Documents",

abstract = "Multimodal key information extraction (KIE) models have been studied extensively on semi-structured documents. However, their investigation on unstructured documents is an emerging research topic. The paper presents an approach to adapt a multimodal transformer (i.e., ViBERTgrid previously explored on semi-structured documents) for unstructured financial documents, by incorporating a BiLSTM-CRF layer. The proposed ViBERTgrid BiLSTM-CRF model demonstrates a significant improvement in performance (up to 2\% points) on named entity recognition from unstructured documents in financial domain, while maintaining its KIE performance on semi-structured documents. As an additional contribution, we publicly released token-level annotations for the SROIE dataset in order to pave the way for its use in multimodal sequence labeling models.",

keywords = "Multimodal Information Extraction, Natural Language Processing, Unstructured Financial Documents",

author = "Furkan Pala and Akpınar, \{Mehmet Yasin\} and Onur Deniz and G{\"u}l{\c s}en Eryiğit",

note = "Publisher Copyright: {\textcopyright} The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.; 23rd Joint European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2023 ; Conference date: 18-09-2023 Through 22-09-2023",

year = "2025",

doi = "10.1007/978-3-031-74643-7\_23",

language = "English",

isbn = "9783031746420",

series = "Communications in Computer and Information Science",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "307--322",

editor = "Rosa Meo and Fabrizio Silvestri",

booktitle = "Machine Learning and Principles and Practice of Knowledge Discovery in Databases - International Workshops of ECML PKDD 2023, Revised Selected Papers",

address = "Germany",

}

Pala, F, Akpınar, MY, Deniz, O & Eryiğit, G 2025, ViBERTgrid BiLSTM-CRF: Multimodal Key Information Extraction from Unstructured Financial Documents. in R Meo & F Silvestri (eds), Machine Learning and Principles and Practice of Knowledge Discovery in Databases - International Workshops of ECML PKDD 2023, Revised Selected Papers. Communications in Computer and Information Science, vol. 2137 CCIS, Springer Science and Business Media Deutschland GmbH, pp. 307-322, 23rd Joint European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2023, Turin, Italy, 18/09/23. https://doi.org/10.1007/978-3-031-74643-7_23

ViBERTgrid BiLSTM-CRF: Multimodal Key Information Extraction from Unstructured Financial Documents. / Pala, Furkan; Akpınar, Mehmet Yasin; Deniz, Onur et al.
Machine Learning and Principles and Practice of Knowledge Discovery in Databases - International Workshops of ECML PKDD 2023, Revised Selected Papers. ed. / Rosa Meo; Fabrizio Silvestri. Springer Science and Business Media Deutschland GmbH, 2025. p. 307-322 (Communications in Computer and Information Science; Vol. 2137 CCIS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - ViBERTgrid BiLSTM-CRF

T2 - 23rd Joint European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2023

AU - Pala, Furkan

AU - Akpınar, Mehmet Yasin

AU - Deniz, Onur

AU - Eryiğit, Gülşen

PY - 2025

Y1 - 2025

N2 - Multimodal key information extraction (KIE) models have been studied extensively on semi-structured documents. However, their investigation on unstructured documents is an emerging research topic. The paper presents an approach to adapt a multimodal transformer (i.e., ViBERTgrid previously explored on semi-structured documents) for unstructured financial documents, by incorporating a BiLSTM-CRF layer. The proposed ViBERTgrid BiLSTM-CRF model demonstrates a significant improvement in performance (up to 2% points) on named entity recognition from unstructured documents in financial domain, while maintaining its KIE performance on semi-structured documents. As an additional contribution, we publicly released token-level annotations for the SROIE dataset in order to pave the way for its use in multimodal sequence labeling models.

AB - Multimodal key information extraction (KIE) models have been studied extensively on semi-structured documents. However, their investigation on unstructured documents is an emerging research topic. The paper presents an approach to adapt a multimodal transformer (i.e., ViBERTgrid previously explored on semi-structured documents) for unstructured financial documents, by incorporating a BiLSTM-CRF layer. The proposed ViBERTgrid BiLSTM-CRF model demonstrates a significant improvement in performance (up to 2% points) on named entity recognition from unstructured documents in financial domain, while maintaining its KIE performance on semi-structured documents. As an additional contribution, we publicly released token-level annotations for the SROIE dataset in order to pave the way for its use in multimodal sequence labeling models.

KW - Multimodal Information Extraction

KW - Natural Language Processing

KW - Unstructured Financial Documents

UR - https://www.scopus.com/pages/publications/85216011471

U2 - 10.1007/978-3-031-74643-7_23

DO - 10.1007/978-3-031-74643-7_23

M3 - Conference contribution

AN - SCOPUS:85216011471

SN - 9783031746420

T3 - Communications in Computer and Information Science

SP - 307

EP - 322

BT - Machine Learning and Principles and Practice of Knowledge Discovery in Databases - International Workshops of ECML PKDD 2023, Revised Selected Papers

A2 - Meo, Rosa

A2 - Silvestri, Fabrizio

PB - Springer Science and Business Media Deutschland GmbH

Y2 - 18 September 2023 through 22 September 2023

ER -

Pala F, Akpınar MY, Deniz O, Eryiğit G. ViBERTgrid BiLSTM-CRF: Multimodal Key Information Extraction from Unstructured Financial Documents. In Meo R, Silvestri F, editors, Machine Learning and Principles and Practice of Knowledge Discovery in Databases - International Workshops of ECML PKDD 2023, Revised Selected Papers. Springer Science and Business Media Deutschland GmbH. 2025. p. 307-322. (Communications in Computer and Information Science). doi: 10.1007/978-3-031-74643-7_23