ISLES 2016 and 2017-benchmarking ischemic stroke lesion outcome prediction based on multispectral MRI

Stefan Winzeck; Arsany Hakim; Richard McKinley; José A.A.D.S.R. Pinto; Victor Alves; Carlos Silva; Maxim Pisov; Egor Krivov; Mikhail Belyaev; Miguel Monteiro; Arlindo Oliveira; Youngwon Choi; Myunghee Cho Paik; Yongchan Kwon; Hanbyul Lee; Beom Joon Kim; Joong Ho Won; Mobarakol Islam; Hongliang Ren; David Robben; Paul Suetens; Enhao Gong; Yilin Niu; Junshen Xu; John M. Pauly; Christian Lucas; Mattias P. Heinrich; Luis C. Rivera; Laura S. Castillo; Laura A. Daza; Andrew L. Beers; Pablo Arbelaezs; Oskar Maier; Ken Chang; James M. Brown; Jayashree Kalpathy-Cramer; Greg Zaharchuk; Roland Wiest; Mauricio Reyes

doi:10.3389/fneur.2018.00679

ISLES 2016 and 2017-benchmarking ischemic stroke lesion outcome prediction based on multispectral MRI

Stefan Winzeck^*, Arsany Hakim, Richard McKinley, José A.A.D.S.R. Pinto, Victor Alves, Carlos Silva, Maxim Pisov, Egor Krivov, Mikhail Belyaev, Miguel Monteiro, Arlindo Oliveira, Youngwon Choi, Myunghee Cho Paik, Yongchan Kwon, Hanbyul Lee, Beom Joon Kim, Joong Ho Won, Mobarakol Islam, Hongliang Ren, David RobbenPaul Suetens, Enhao Gong, Yilin Niu, Junshen Xu, John M. Pauly, Christian Lucas, Mattias P. Heinrich, Luis C. Rivera, Laura S. Castillo, Laura A. Daza, Andrew L. Beers, Pablo Arbelaezs, Oskar Maier, Ken Chang, James M. Brown, Jayashree Kalpathy-Cramer, Greg Zaharchuk, Roland Wiest, Mauricio Reyes

^*Corresponding author for this work

Institute of Medical Informatics

153 Citations (Scopus)

Abstract

Performance of models highly depend not only on the used algorithm but also the data set it was applied to. This makes the comparison of newly developed tools to previously published approaches difficult. Either researchers need to implement others' algorithms first, to establish an adequate benchmark on their data, or a direct comparison of new and old techniques is infeasible. The Ischemic Stroke Lesion Segmentation (ISLES) challenge, which has ran now consecutively for 3 years, aims to address this problem of comparability. ISLES 2016 and 2017 focused on lesion outcome prediction after ischemic stroke: By providing a uniformly pre-processed data set, researchers from all over the world could apply their algorithm directly. A total of nine teams participated in ISLES 2015, and 15 teams participated in ISLES 2016. Their performance was evaluated in a fair and transparent way to identify the state-of-the-art among all submissions. Top ranked teams almost always employed deep learning tools, which were predominately convolutional neural networks (CNNs). Despite the great efforts, lesion outcome prediction persists challenging. The annotated data set remains publicly available and new approaches can be compared directly via the online evaluation system, serving as a continuing benchmark (www.isles-challenge.org).

Original language	English
Article number	679
Journal	Frontiers in Neurology
Volume	9
Issue number	SEP
ISSN	1664-2295
DOIs	https://doi.org/10.3389/fneur.2018.00679
Publication status	Published - 13.09.2018

Funding

1University Division of Anaesthesia, Department of Medicine, University of Cambridge, Cambridge, United Kingdom, 2Support Center of Advanced Neuroimaging (SCAN), Institute of Diagnostic and Interventional Neuroradiology, University of Bern, Inselspital, Bern University Hospital, Bern, Switzerland, 3CMEMS-UMinho Research Unit, University of Minho, Braga, Portugal, 4Moscow Institute of Physics and Technology, Dolgoprudny, Russia, 5Institute for Information Transmission Problems (RAS), Moscow, Russia, 6Instituto de Engenharia de Sostemas e Computadores Investigacã e Desenvolvimento, Lisbon, Portugal, 7Department of Statistics, Seoul National University, Seoul, South Korea, 8Department of Neurology and Cerebrovascular Center, Seoul National University Bundang Hospital, Seongnam, South Korea, 9Department of Biomedical Engineering, National University of Singapore, Singapore, Singapore, 10ESAT-PSI, KU Leuven, Leuven, Belgium, 11Electrical Engineering and Radiology, Stanford University, Stanford, CA, United States, 12Computer Science, Tsinghua University, Beijing, China, 13Institute of Medical Informatics, Universität zu Lübeck, Lübeck, Germany, 14Biomedical Engineering, University of Los Andes, Bogotá, Colombia, 15Athinoula A. Martinos Center for Biomedical Imaging, Harvard, MA, United States, 16Department of Radiology, Stanford University, Stanford, CA, United States, 17Medical Image Analysis, Institute for Surgical Technology and Biomechanics, University of Bern, Bern, Switzerland Fundacao para a Ciencia e Tecnologia (FCT), Portugal (scholarship number PD/BD/113968/2015). FCT with the UID/EEA/04436/2013, by FEDER funds through COMPETE 2020, POCI-01-0145-FEDER-006941. NIH Blueprint for Neuroscience Research (T90DA022759/R90DA023427) and the National Institute of Biomedical Imaging and Bioengineering (NIBIB) of the National Institutes of Health under award number 5T32EB1680. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. PAC-PRECISE-LISBOA-01-0145-FEDER-016394. FEDER-POR Lisboa 2020-Programa Operacional Regional de Lisboa PORTUGAL 2020 and Fundação para a Ciência e a Tecnologia. GPU computing resources provided by the MGH and BWH Center for Clinical Data Science Graduate School for Computing in Medicine and Life Sciences funded by Germany’s Excellence Initiative [DFG GSC 235/2]. National Research National Research Foundation of Korea (NRF) MSIT, NRF-2016R1C1B1012002, MSIT, No. 2014R1A4A1007895, NRF-2017R1A2B4008956 Swiss National Science Foundation-DACH 320030L_163363. The authors acknowledge the support of the Herzstiftung. This work was supported by PAC - PRECISE - LISBOA-01-0145-FEDER-016394, co-funded by FEDER through POR Lisboa 2020 -Programa Operacional Regional de Lisboa PORTUGAL 2020 and Fundação para a Ciência e a Tecnologia. This work was supported by Institute for Information & Communications Technology Promotion (IITP) grant funded by the Korea government (No. 2017-0-00451). This research was supported by the National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIT, NRF-2016R1C1B1012002). Joong-Ho Won’s research was partially supported by the National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIT, No. 2014R1A4A1007895). Myunghee Cho Paik’s research was supported by the National Research Foundation of Korea under grant NRF-2017R1A2B4008956. This work was supported by the Graduate School for Computing in Medicine and Life Sciences funded by Germany’s Excellence Initiative [DFG GSC 235/2]. We would also like to thank Nvidia Corporation for their support by providing us with a Titan Xp graphics card. Adriano Pinto was supported by a scholarship from the Fundação para a Ciência e Tecnologia (FCT), Portugal (scholarship number PD/BD/113968/2015). This work is supported by FCT with the reference project UID/EEA/04436/2013, by FEDER funds through the COMPETE 2020 - Programa Operacional Competitividade e Internacionalização (POCI) with the reference project POCI-01-0145-FEDER-006941. We acknowledge support from the Swiss National Science Foundation - DACH 320030L 163363.

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 3 Good Health and Well-being
SDG 9 Industry, Innovation, and Infrastructure

Access to Document

10.3389/fneur.2018.00679

Cite this

Winzeck, S., Hakim, A., McKinley, R., Pinto, J. A. A. D. S. R., Alves, V., Silva, C., Pisov, M., Krivov, E., Belyaev, M., Monteiro, M., Oliveira, A., Choi, Y., Paik, M. C., Kwon, Y., Lee, H., Kim, B. J., Won, J. H., Islam, M., Ren, H., ... Reyes, M. (2018). ISLES 2016 and 2017-benchmarking ischemic stroke lesion outcome prediction based on multispectral MRI. Frontiers in Neurology, 9(SEP), Article 679. https://doi.org/10.3389/fneur.2018.00679

@article{ead41690d7f2468f85e7c24ec2d36321,

title = "ISLES 2016 and 2017-benchmarking ischemic stroke lesion outcome prediction based on multispectral MRI",

abstract = "Performance of models highly depend not only on the used algorithm but also the data set it was applied to. This makes the comparison of newly developed tools to previously published approaches difficult. Either researchers need to implement others' algorithms first, to establish an adequate benchmark on their data, or a direct comparison of new and old techniques is infeasible. The Ischemic Stroke Lesion Segmentation (ISLES) challenge, which has ran now consecutively for 3 years, aims to address this problem of comparability. ISLES 2016 and 2017 focused on lesion outcome prediction after ischemic stroke: By providing a uniformly pre-processed data set, researchers from all over the world could apply their algorithm directly. A total of nine teams participated in ISLES 2015, and 15 teams participated in ISLES 2016. Their performance was evaluated in a fair and transparent way to identify the state-of-the-art among all submissions. Top ranked teams almost always employed deep learning tools, which were predominately convolutional neural networks (CNNs). Despite the great efforts, lesion outcome prediction persists challenging. The annotated data set remains publicly available and new approaches can be compared directly via the online evaluation system, serving as a continuing benchmark (www.isles-challenge.org).",

author = "Stefan Winzeck and Arsany Hakim and Richard McKinley and Pinto, \{Jos{\'e} A.A.D.S.R.\} and Victor Alves and Carlos Silva and Maxim Pisov and Egor Krivov and Mikhail Belyaev and Miguel Monteiro and Arlindo Oliveira and Youngwon Choi and Paik, \{Myunghee Cho\} and Yongchan Kwon and Hanbyul Lee and Kim, \{Beom Joon\} and Won, \{Joong Ho\} and Mobarakol Islam and Hongliang Ren and David Robben and Paul Suetens and Enhao Gong and Yilin Niu and Junshen Xu and Pauly, \{John M.\} and Christian Lucas and Heinrich, \{Mattias P.\} and Rivera, \{Luis C.\} and Castillo, \{Laura S.\} and Daza, \{Laura A.\} and Beers, \{Andrew L.\} and Pablo Arbelaezs and Oskar Maier and Ken Chang and Brown, \{James M.\} and Jayashree Kalpathy-Cramer and Greg Zaharchuk and Roland Wiest and Mauricio Reyes",

year = "2018",

month = sep,

day = "13",

doi = "10.3389/fneur.2018.00679",

language = "English",

volume = "9",

journal = "Frontiers in Neurology",

issn = "1664-2295",

number = "SEP",

}

Winzeck, S, Hakim, A, McKinley, R, Pinto, JAADSR, Alves, V, Silva, C, Pisov, M, Krivov, E, Belyaev, M, Monteiro, M, Oliveira, A, Choi, Y, Paik, MC, Kwon, Y, Lee, H, Kim, BJ, Won, JH, Islam, M, Ren, H, Robben, D, Suetens, P, Gong, E, Niu, Y, Xu, J, Pauly, JM, Lucas, C, Heinrich, MP, Rivera, LC, Castillo, LS, Daza, LA, Beers, AL, Arbelaezs, P, Maier, O, Chang, K, Brown, JM, Kalpathy-Cramer, J, Zaharchuk, G, Wiest, R & Reyes, M 2018, 'ISLES 2016 and 2017-benchmarking ischemic stroke lesion outcome prediction based on multispectral MRI', Frontiers in Neurology, vol. 9, no. SEP, 679. https://doi.org/10.3389/fneur.2018.00679

TY - JOUR

T1 - ISLES 2016 and 2017-benchmarking ischemic stroke lesion outcome prediction based on multispectral MRI

AU - Winzeck, Stefan

AU - Hakim, Arsany

AU - McKinley, Richard

AU - Pinto, José A.A.D.S.R.

AU - Alves, Victor

AU - Silva, Carlos

AU - Pisov, Maxim

AU - Krivov, Egor

AU - Belyaev, Mikhail

AU - Monteiro, Miguel

AU - Oliveira, Arlindo

AU - Choi, Youngwon

AU - Paik, Myunghee Cho

AU - Kwon, Yongchan

AU - Lee, Hanbyul

AU - Kim, Beom Joon

AU - Won, Joong Ho

AU - Islam, Mobarakol

AU - Ren, Hongliang

AU - Robben, David

AU - Suetens, Paul

AU - Gong, Enhao

AU - Niu, Yilin

AU - Xu, Junshen

AU - Pauly, John M.

AU - Lucas, Christian

AU - Heinrich, Mattias P.

AU - Rivera, Luis C.

AU - Castillo, Laura S.

AU - Daza, Laura A.

AU - Beers, Andrew L.

AU - Arbelaezs, Pablo

AU - Maier, Oskar

AU - Chang, Ken

AU - Brown, James M.

AU - Kalpathy-Cramer, Jayashree

AU - Zaharchuk, Greg

AU - Wiest, Roland

AU - Reyes, Mauricio

PY - 2018/9/13

Y1 - 2018/9/13

N2 - Performance of models highly depend not only on the used algorithm but also the data set it was applied to. This makes the comparison of newly developed tools to previously published approaches difficult. Either researchers need to implement others' algorithms first, to establish an adequate benchmark on their data, or a direct comparison of new and old techniques is infeasible. The Ischemic Stroke Lesion Segmentation (ISLES) challenge, which has ran now consecutively for 3 years, aims to address this problem of comparability. ISLES 2016 and 2017 focused on lesion outcome prediction after ischemic stroke: By providing a uniformly pre-processed data set, researchers from all over the world could apply their algorithm directly. A total of nine teams participated in ISLES 2015, and 15 teams participated in ISLES 2016. Their performance was evaluated in a fair and transparent way to identify the state-of-the-art among all submissions. Top ranked teams almost always employed deep learning tools, which were predominately convolutional neural networks (CNNs). Despite the great efforts, lesion outcome prediction persists challenging. The annotated data set remains publicly available and new approaches can be compared directly via the online evaluation system, serving as a continuing benchmark (www.isles-challenge.org).

AB - Performance of models highly depend not only on the used algorithm but also the data set it was applied to. This makes the comparison of newly developed tools to previously published approaches difficult. Either researchers need to implement others' algorithms first, to establish an adequate benchmark on their data, or a direct comparison of new and old techniques is infeasible. The Ischemic Stroke Lesion Segmentation (ISLES) challenge, which has ran now consecutively for 3 years, aims to address this problem of comparability. ISLES 2016 and 2017 focused on lesion outcome prediction after ischemic stroke: By providing a uniformly pre-processed data set, researchers from all over the world could apply their algorithm directly. A total of nine teams participated in ISLES 2015, and 15 teams participated in ISLES 2016. Their performance was evaluated in a fair and transparent way to identify the state-of-the-art among all submissions. Top ranked teams almost always employed deep learning tools, which were predominately convolutional neural networks (CNNs). Despite the great efforts, lesion outcome prediction persists challenging. The annotated data set remains publicly available and new approaches can be compared directly via the online evaluation system, serving as a continuing benchmark (www.isles-challenge.org).

UR - https://www.scopus.com/pages/publications/85055195176

U2 - 10.3389/fneur.2018.00679

DO - 10.3389/fneur.2018.00679

M3 - Journal articles

AN - SCOPUS:85055195176

SN - 1664-2295

VL - 9

JO - Frontiers in Neurology

JF - Frontiers in Neurology

IS - SEP

M1 - 679

ER -

ISLES 2016 and 2017-benchmarking ischemic stroke lesion outcome prediction based on multispectral MRI

Abstract

Funding

UN SDGs

Access to Document

Other files and links

Fingerprint

Cite this