Analysis of Variability in Manual Liver Tumor Delineation in CT Scans

Jan Hendrik Moltz; Stefan Braunewell; Jan Rühaak; Frank Heckel; Sebastiano Barbieri; Lennart Tautz; Horst Karl Hahn; Heinz-Otto Peitgen

doi:10.1109/ISBI.2011.5872797

Analysis of Variability in Manual Liver Tumor Delineation in CT Scans

Jan Hendrik Moltz, Stefan Braunewell, Jan Rühaak, Frank Heckel, Sebastiano Barbieri, Lennart Tautz, Horst Karl Hahn, Heinz-Otto Peitgen

Institute of Mathematics and Image Computing

Abstract

Manual delineations by experts are often used as reference standards for validating segmentation algorithms, although it is well known that they always show some degree of variability. Our goal is to estimate the effects of using a limited number of expert segmentations. Given ten manual delineations of 13 liver tumors, we analyzed the volume error made by randomly selecting subsets of the ten segmentations compared to the complete set. We found that when using just one segmentation the expected error was 17% with a maximum of 35%. This means that it is questionable whether a comparison with a single reference allows a reliable validation. When three segmentations are chosen, the error is halved, so this might be a reasonable compromise between accuracy and viability of evaluation studies.

Original language	English
Title of host publication	2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro
Number of pages	4
Place of Publication	Chicago, Illinois, USA
Publisher	IEEE
Publication date	01.03.2011
Pages	1974-1977
ISBN (Print)	978-1-4244-4127-3
ISBN (Electronic)	978-1-4244-4128-0
DOIs	https://doi.org/10.1109/ISBI.2011.5872797
Publication status	Published - 01.03.2011
Event	2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro - Chicago, United States Duration: 30.03.2011 → 02.04.2011

Access to Document

10.1109/ISBI.2011.5872797

https://ieeexplore.ieee.org/document/5872797/

Cite this

Moltz, J. H., Braunewell, S., Rühaak, J., Heckel, F., Barbieri, S., Tautz, L., Hahn, H. K., & Peitgen, H-O. (2011). Analysis of Variability in Manual Liver Tumor Delineation in CT Scans. In 2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro (pp. 1974-1977). (International Symposium on Biomedical Imaging: From Nano to Macro). IEEE. https://doi.org/10.1109/ISBI.2011.5872797

@inproceedings{55f132321e324b73a872e50caaa7d155,

title = "Analysis of Variability in Manual Liver Tumor Delineation in CT Scans",

abstract = "Manual delineations by experts are often used as reference standards for validating segmentation algorithms, although it is well known that they always show some degree of variability. Our goal is to estimate the effects of using a limited number of expert segmentations. Given ten manual delineations of 13 liver tumors, we analyzed the volume error made by randomly selecting subsets of the ten segmentations compared to the complete set. We found that when using just one segmentation the expected error was 17% with a maximum of 35%. This means that it is questionable whether a comparison with a single reference allows a reliable validation. When three segmentations are chosen, the error is halved, so this might be a reasonable compromise between accuracy and viability of evaluation studies.",

author = "Moltz, {Jan Hendrik} and Stefan Braunewell and Jan R{\"u}haak and Frank Heckel and Sebastiano Barbieri and Lennart Tautz and Hahn, {Horst Karl} and Heinz-Otto Peitgen",

year = "2011",

month = mar,

day = "1",

doi = "10.1109/ISBI.2011.5872797",

language = "English",

isbn = "978-1-4244-4127-3",

series = "International Symposium on Biomedical Imaging: From Nano to Macro",

publisher = "IEEE",

pages = "1974--1977",

booktitle = "2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro",

note = "2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro ; Conference date: 30-03-2011 Through 02-04-2011",

}

Moltz, JH, Braunewell, S, Rühaak, J, Heckel, F, Barbieri, S, Tautz, L, Hahn, HK & Peitgen, H-O 2011, Analysis of Variability in Manual Liver Tumor Delineation in CT Scans. in 2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro. International Symposium on Biomedical Imaging: From Nano to Macro, IEEE, Chicago, Illinois, USA, pp. 1974-1977, 2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro, Chicago, Illinois, United States, 30.03.11. https://doi.org/10.1109/ISBI.2011.5872797

Analysis of Variability in Manual Liver Tumor Delineation in CT Scans. / Moltz, Jan Hendrik; Braunewell, Stefan; Rühaak, Jan et al.

2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro. Chicago, Illinois, USA : IEEE, 2011. p. 1974-1977 (International Symposium on Biomedical Imaging: From Nano to Macro).

Research output: Chapters in Books/Reports/Conference Proceedings › Conference contribution › peer-review

TY - GEN

T1 - Analysis of Variability in Manual Liver Tumor Delineation in CT Scans

AU - Moltz, Jan Hendrik

AU - Braunewell, Stefan

AU - Rühaak, Jan

AU - Heckel, Frank

AU - Barbieri, Sebastiano

AU - Tautz, Lennart

AU - Hahn, Horst Karl

AU - Peitgen, Heinz-Otto

PY - 2011/3/1

Y1 - 2011/3/1

N2 - Manual delineations by experts are often used as reference standards for validating segmentation algorithms, although it is well known that they always show some degree of variability. Our goal is to estimate the effects of using a limited number of expert segmentations. Given ten manual delineations of 13 liver tumors, we analyzed the volume error made by randomly selecting subsets of the ten segmentations compared to the complete set. We found that when using just one segmentation the expected error was 17% with a maximum of 35%. This means that it is questionable whether a comparison with a single reference allows a reliable validation. When three segmentations are chosen, the error is halved, so this might be a reasonable compromise between accuracy and viability of evaluation studies.

AB - Manual delineations by experts are often used as reference standards for validating segmentation algorithms, although it is well known that they always show some degree of variability. Our goal is to estimate the effects of using a limited number of expert segmentations. Given ten manual delineations of 13 liver tumors, we analyzed the volume error made by randomly selecting subsets of the ten segmentations compared to the complete set. We found that when using just one segmentation the expected error was 17% with a maximum of 35%. This means that it is questionable whether a comparison with a single reference allows a reliable validation. When three segmentations are chosen, the error is halved, so this might be a reasonable compromise between accuracy and viability of evaluation studies.

U2 - 10.1109/ISBI.2011.5872797

DO - 10.1109/ISBI.2011.5872797

M3 - Conference contribution

SN - 978-1-4244-4127-3

T3 - International Symposium on Biomedical Imaging: From Nano to Macro

SP - 1974

EP - 1977

BT - 2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro

PB - IEEE

CY - Chicago, Illinois, USA

T2 - 2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro

Y2 - 30 March 2011 through 2 April 2011

ER -

Moltz JH, Braunewell S, Rühaak J, Heckel F, Barbieri S, Tautz L et al. Analysis of Variability in Manual Liver Tumor Delineation in CT Scans. In 2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro. Chicago, Illinois, USA: IEEE. 2011. p. 1974-1977. (International Symposium on Biomedical Imaging: From Nano to Macro). doi: 10.1109/ISBI.2011.5872797

Analysis of Variability in Manual Liver Tumor Delineation in CT Scans

Abstract

Access to Document

Fingerprint

Cite this