The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS)

Bjoern H. Menze; Andras Jakab; Stefan Bauer; Jayashree Kalpathy-Cramer; Keyvan Farahani; Justin Kirby; Yuliya Burren; Nicole Porz; Johannes Slotboom; Roland Wiest; Levente Lanczi; Elizabeth Gerstner; Marc André Weber; Tal Arbel; Brian B. Avants; Nicholas Ayache; Patricia Buendia; D. Louis Collins; Nicolas Cordier; Jason J. Corso; Antonio Criminisi; Tilak Das; Hervé Delingette; Çağatay Demiralp; Christopher R. Durst; Michel Dojat; Senan Doyle; Joana Festa; Florence Forbes; Ezequiel Geremia; Ben Glocker; Polina Golland; Xiaotao Guo; Andac Hamamci; Khan M. Iftekharuddin; Raj Jena; Nigel M. John; Ender Konukoglu; Danial Lashkari; José António Mariz; Raphael Meier; Sérgio Pereira; Doina Precup; Stephen J. Price; Tammy Riklin Raviv; Syed M.S. Reza; Michael Ryan; Duygu Sarikaya; Lawrence Schwartz; Hoo Chang Shin; Jamie Shotton; Carlos A. Silva; Nuno Sousa; Nagesh K. Subbanna; Gabor Szekely; Thomas J. Taylor; Owen M. Thomas; Nicholas J. Tustison; Gozde Unal; Flor Vasseur; Max Wintermark; Dong Hye Ye; Liang Zhao; Binsheng Zhao; Darko Zikic; Marcel Prastawa; Mauricio Reyes; Koen Van Leemput

doi:10.1109/TMI.2014.2377694

The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS)

Bjoern H. Menze^*, Andras Jakab, Stefan Bauer, Jayashree Kalpathy-Cramer, Keyvan Farahani, Justin Kirby, Yuliya Burren, Nicole Porz, Johannes Slotboom, Roland Wiest, Levente Lanczi, Elizabeth Gerstner, Marc André Weber, Tal Arbel, Brian B. Avants, Nicholas Ayache, Patricia Buendia, D. Louis Collins, Nicolas Cordier, Jason J. CorsoAntonio Criminisi, Tilak Das, Hervé Delingette, Çağatay Demiralp, Christopher R. Durst, Michel Dojat, Senan Doyle, Joana Festa, Florence Forbes, Ezequiel Geremia, Ben Glocker, Polina Golland, Xiaotao Guo, Andac Hamamci, Khan M. Iftekharuddin, Raj Jena, Nigel M. John, Ender Konukoglu, Danial Lashkari, José António Mariz, Raphael Meier, Sérgio Pereira, Doina Precup, Stephen J. Price, Tammy Riklin Raviv, Syed M.S. Reza, Michael Ryan, Duygu Sarikaya, Lawrence Schwartz, Hoo Chang Shin, Jamie Shotton, Carlos A. Silva, Nuno Sousa, Nagesh K. Subbanna, Gabor Szekely, Thomas J. Taylor, Owen M. Thomas, Nicholas J. Tustison, Gozde Unal, Flor Vasseur, Max Wintermark, Dong Hye Ye, Liang Zhao, Binsheng Zhao, Darko Zikic, Marcel Prastawa, Mauricio Reyes, Koen Van Leemput

^*Corresponding author for this work

Research output: Contribution to journal › Article › peer-review

4936 Citations (Scopus)

Abstract

In this paper we report the set-up and results of the Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) organized in conjunction with the MICCAI 2012 and 2013 conferences. Twenty state-of-the-art tumor segmentation algorithms were applied to a set of 65 multi-contrast MR scans of low- and high-grade glioma patients - manually annotated by up to four raters - and to 65 comparable scans generated using tumor image simulation software. Quantitative evaluations revealed considerable disagreement between the human raters in segmenting various tumor sub-regions (Dice scores in the range 74%-85%), illustrating the difficulty of this task. We found that different algorithms worked best for different sub-regions (reaching performance comparable to human inter-rater variability), but that no single algorithm ranked in the top for all sub-regions simultaneously. Fusing several good algorithms using a hierarchical majority vote yielded segmentations that consistently ranked above all individual algorithms, indicating remaining opportunities for further methodological improvements. The BRATS image data and manual annotations continue to be publicly available through an online evaluation system as an ongoing benchmarking resource.

Original language	English
Article number	2377694
Pages (from-to)	1993-2024
Number of pages	32
Journal	IEEE Transactions on Medical Imaging
Volume	34
Issue number	10
DOIs	https://doi.org/10.1109/TMI.2014.2377694
Publication status	Published - 1 Oct 2015
Externally published	Yes

Bibliographical note

Publisher Copyright:
© 2014 IEEE.

Funding

This research was supported by the NIH NCRR (P41-RR14075), the NIH NIBIB (R01EB013565), the Academy of Finland (133611), TEKES (ComBrain), the Lundbeck Foundation (R141-2013-13117), the Swiss Cancer League, the Swiss Institute for Computer Assisted Surgery (SICAS), the NIH NIBIB NAMIC (U54-EB005149), the NIH NCRR NAC (P41-RR13218), the NIH NIBIB NAC (P41-EB-015902), the NIH NCI (R15CA115464), the European Research Council through the ERC Advanced Grant MedYMA 2011-291080 (on Biophysical Modeling and Analysis of Dynamic Medical Images), the FCT and COMPETE (FCOM-01-0124-FEDER-022674), the MICAT Project (EU FP7 Marie Curie Grant No. PIRG-GA-2008-231052), the European Union Seventh Framework Programme under grant agreement no. 600841, the Swiss NSF project Computer Aided and Image Guided Medical Interventions (NCCR CO-ME), the Technische Universität München—Institute for Advanced Study (funded by the German Excellence Initiative and the European Union Seventh Framework Programme under Grant agreement 291763), the Marie Curie COFUND program of the European Union (Rudolf Mössbauer Tenure-Track Professorship to BHM).

Funders	Funder number
FP7 Marie Curie	PIRG-GA-2008-231052
German Excellence Initiative	291763
NIH NCI	R15CA115464
NIH NCRR	P41-RR14075
NIH NCRR NAC	P41-RR13218
NIH NIBIB NAC	P41-EB-015902
NIH NIBIB NAMIC	U54-EB005149
Swiss Institute for Computer Assisted Surgery
National Institute of Biomedical Imaging and Bioengineering	R01EB013565
National Center for Research Resources	P41RR013218
European Commission
European Research Council	MedYMA 2011-291080
Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung
Fundação para a Ciência e a Tecnologia
Academy of Finland	133611
Tekes
Lundbeckfonden	R141-2013-13117
Krebsliga Schweiz
Seventh Framework Programme	600841
Programa Operacional Temático Factores de Competitividade	FCOM-01-0124-FEDER-022674
Institute for Advanced Study, Technische Universität München

Keywords

Benchmark
Brain
Image segmentation
MRI
Oncology/tumor

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1109/TMI.2014.2377694

Cite this

Menze, B. H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., Burren, Y., Porz, N., Slotboom, J., Wiest, R., Lanczi, L., Gerstner, E., Weber, M. A., Arbel, T., Avants, B. B., Ayache, N., Buendia, P., Collins, D. L., Cordier, N., ... Van Leemput, K. (2015). The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS). IEEE Transactions on Medical Imaging, 34(10), 1993-2024. Article 2377694. https://doi.org/10.1109/TMI.2014.2377694

@article{3dc8fefaeea14e788af54dab376f55be,

title = "The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS)",

abstract = "In this paper we report the set-up and results of the Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) organized in conjunction with the MICCAI 2012 and 2013 conferences. Twenty state-of-the-art tumor segmentation algorithms were applied to a set of 65 multi-contrast MR scans of low- and high-grade glioma patients - manually annotated by up to four raters - and to 65 comparable scans generated using tumor image simulation software. Quantitative evaluations revealed considerable disagreement between the human raters in segmenting various tumor sub-regions (Dice scores in the range 74\%-85\%), illustrating the difficulty of this task. We found that different algorithms worked best for different sub-regions (reaching performance comparable to human inter-rater variability), but that no single algorithm ranked in the top for all sub-regions simultaneously. Fusing several good algorithms using a hierarchical majority vote yielded segmentations that consistently ranked above all individual algorithms, indicating remaining opportunities for further methodological improvements. The BRATS image data and manual annotations continue to be publicly available through an online evaluation system as an ongoing benchmarking resource.",

keywords = "Benchmark, Brain, Image segmentation, MRI, Oncology/tumor",

author = "Menze, \{Bjoern H.\} and Andras Jakab and Stefan Bauer and Jayashree Kalpathy-Cramer and Keyvan Farahani and Justin Kirby and Yuliya Burren and Nicole Porz and Johannes Slotboom and Roland Wiest and Levente Lanczi and Elizabeth Gerstner and Weber, \{Marc Andr{\'e}\} and Tal Arbel and Avants, \{Brian B.\} and Nicholas Ayache and Patricia Buendia and Collins, \{D. Louis\} and Nicolas Cordier and Corso, \{Jason J.\} and Antonio Criminisi and Tilak Das and Herv{\'e} Delingette and {\c C}ağatay Demiralp and Durst, \{Christopher R.\} and Michel Dojat and Senan Doyle and Joana Festa and Florence Forbes and Ezequiel Geremia and Ben Glocker and Polina Golland and Xiaotao Guo and Andac Hamamci and Iftekharuddin, \{Khan M.\} and Raj Jena and John, \{Nigel M.\} and Ender Konukoglu and Danial Lashkari and Mariz, \{Jos{\'e} Ant{\'o}nio\} and Raphael Meier and S{\'e}rgio Pereira and Doina Precup and Price, \{Stephen J.\} and Raviv, \{Tammy Riklin\} and Reza, \{Syed M.S.\} and Michael Ryan and Duygu Sarikaya and Lawrence Schwartz and Shin, \{Hoo Chang\} and Jamie Shotton and Silva, \{Carlos A.\} and Nuno Sousa and Subbanna, \{Nagesh K.\} and Gabor Szekely and Taylor, \{Thomas J.\} and Thomas, \{Owen M.\} and Tustison, \{Nicholas J.\} and Gozde Unal and Flor Vasseur and Max Wintermark and Ye, \{Dong Hye\} and Liang Zhao and Binsheng Zhao and Darko Zikic and Marcel Prastawa and Mauricio Reyes and \{Van Leemput\}, Koen",

note = "Publisher Copyright: {\textcopyright} 2014 IEEE.",

year = "2015",

month = oct,

day = "1",

doi = "10.1109/TMI.2014.2377694",

language = "English",

volume = "34",

pages = "1993--2024",

journal = "IEEE Transactions on Medical Imaging",

issn = "0278-0062",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

number = "10",

}

Menze, BH, Jakab, A, Bauer, S, Kalpathy-Cramer, J, Farahani, K, Kirby, J, Burren, Y, Porz, N, Slotboom, J, Wiest, R, Lanczi, L, Gerstner, E, Weber, MA, Arbel, T, Avants, BB, Ayache, N, Buendia, P, Collins, DL, Cordier, N, Corso, JJ, Criminisi, A, Das, T, Delingette, H, Demiralp, Ç, Durst, CR, Dojat, M, Doyle, S, Festa, J, Forbes, F, Geremia, E, Glocker, B, Golland, P, Guo, X, Hamamci, A, Iftekharuddin, KM, Jena, R, John, NM, Konukoglu, E, Lashkari, D, Mariz, JA, Meier, R, Pereira, S, Precup, D, Price, SJ, Raviv, TR, Reza, SMS, Ryan, M, Sarikaya, D, Schwartz, L, Shin, HC, Shotton, J, Silva, CA, Sousa, N, Subbanna, NK, Szekely, G, Taylor, TJ, Thomas, OM, Tustison, NJ, Unal, G, Vasseur, F, Wintermark, M, Ye, DH, Zhao, L, Zhao, B, Zikic, D, Prastawa, M, Reyes, M & Van Leemput, K 2015, 'The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS)', IEEE Transactions on Medical Imaging, vol. 34, no. 10, 2377694, pp. 1993-2024. https://doi.org/10.1109/TMI.2014.2377694

TY - JOUR

T1 - The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS)

AU - Menze, Bjoern H.

AU - Jakab, Andras

AU - Bauer, Stefan

AU - Kalpathy-Cramer, Jayashree

AU - Farahani, Keyvan

AU - Kirby, Justin

AU - Burren, Yuliya

AU - Porz, Nicole

AU - Slotboom, Johannes

AU - Wiest, Roland

AU - Lanczi, Levente

AU - Gerstner, Elizabeth

AU - Weber, Marc André

AU - Arbel, Tal

AU - Avants, Brian B.

AU - Ayache, Nicholas

AU - Buendia, Patricia

AU - Collins, D. Louis

AU - Cordier, Nicolas

AU - Corso, Jason J.

AU - Criminisi, Antonio

AU - Das, Tilak

AU - Delingette, Hervé

AU - Demiralp, Çağatay

AU - Durst, Christopher R.

AU - Dojat, Michel

AU - Doyle, Senan

AU - Festa, Joana

AU - Forbes, Florence

AU - Geremia, Ezequiel

AU - Glocker, Ben

AU - Golland, Polina

AU - Guo, Xiaotao

AU - Hamamci, Andac

AU - Iftekharuddin, Khan M.

AU - Jena, Raj

AU - John, Nigel M.

AU - Konukoglu, Ender

AU - Lashkari, Danial

AU - Mariz, José António

AU - Meier, Raphael

AU - Pereira, Sérgio

AU - Precup, Doina

AU - Price, Stephen J.

AU - Raviv, Tammy Riklin

AU - Reza, Syed M.S.

AU - Ryan, Michael

AU - Sarikaya, Duygu

AU - Schwartz, Lawrence

AU - Shin, Hoo Chang

AU - Shotton, Jamie

AU - Silva, Carlos A.

AU - Sousa, Nuno

AU - Subbanna, Nagesh K.

AU - Szekely, Gabor

AU - Taylor, Thomas J.

AU - Thomas, Owen M.

AU - Tustison, Nicholas J.

AU - Unal, Gozde

AU - Vasseur, Flor

AU - Wintermark, Max

AU - Ye, Dong Hye

AU - Zhao, Liang

AU - Zhao, Binsheng

AU - Zikic, Darko

AU - Prastawa, Marcel

AU - Reyes, Mauricio

AU - Van Leemput, Koen

PY - 2015/10/1

Y1 - 2015/10/1

N2 - In this paper we report the set-up and results of the Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) organized in conjunction with the MICCAI 2012 and 2013 conferences. Twenty state-of-the-art tumor segmentation algorithms were applied to a set of 65 multi-contrast MR scans of low- and high-grade glioma patients - manually annotated by up to four raters - and to 65 comparable scans generated using tumor image simulation software. Quantitative evaluations revealed considerable disagreement between the human raters in segmenting various tumor sub-regions (Dice scores in the range 74%-85%), illustrating the difficulty of this task. We found that different algorithms worked best for different sub-regions (reaching performance comparable to human inter-rater variability), but that no single algorithm ranked in the top for all sub-regions simultaneously. Fusing several good algorithms using a hierarchical majority vote yielded segmentations that consistently ranked above all individual algorithms, indicating remaining opportunities for further methodological improvements. The BRATS image data and manual annotations continue to be publicly available through an online evaluation system as an ongoing benchmarking resource.

AB - In this paper we report the set-up and results of the Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) organized in conjunction with the MICCAI 2012 and 2013 conferences. Twenty state-of-the-art tumor segmentation algorithms were applied to a set of 65 multi-contrast MR scans of low- and high-grade glioma patients - manually annotated by up to four raters - and to 65 comparable scans generated using tumor image simulation software. Quantitative evaluations revealed considerable disagreement between the human raters in segmenting various tumor sub-regions (Dice scores in the range 74%-85%), illustrating the difficulty of this task. We found that different algorithms worked best for different sub-regions (reaching performance comparable to human inter-rater variability), but that no single algorithm ranked in the top for all sub-regions simultaneously. Fusing several good algorithms using a hierarchical majority vote yielded segmentations that consistently ranked above all individual algorithms, indicating remaining opportunities for further methodological improvements. The BRATS image data and manual annotations continue to be publicly available through an online evaluation system as an ongoing benchmarking resource.

KW - Benchmark

KW - Brain

KW - Image segmentation

KW - MRI

KW - Oncology/tumor

UR - http://www.scopus.com/inward/record.url?scp=84949210409&partnerID=8YFLogxK

U2 - 10.1109/TMI.2014.2377694

DO - 10.1109/TMI.2014.2377694

M3 - Article

C2 - 25494501

AN - SCOPUS:84949210409

SN - 0278-0062

VL - 34

SP - 1993

EP - 2024

JO - IEEE Transactions on Medical Imaging

JF - IEEE Transactions on Medical Imaging

IS - 10

M1 - 2377694

ER -

The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS)

Abstract

Bibliographical note

Funding

Keywords

UN SDGs

Access to Document

Other files and links

Fingerprint

Cite this