Skip to main content

Toward a standard for the evaluation of PET-Auto-Segmentation methods following the recommendations of AAPM task group No. 211: Requirements and implementation.

Publication ,  Journal Article
Berthon, B; Spezi, E; Galavis, P; Shepherd, T; Apte, A; Hatt, M; Fayad, H; De Bernardi, E; Soffientini, CD; Ross Schmidtlein, C; El Naqa, I ...
Published in: Med Phys
August 2017

PURPOSE: The aim of this paper is to define the requirements and describe the design and implementation of a standard benchmark tool for evaluation and validation of PET-auto-segmentation (PET-AS) algorithms. This work follows the recommendations of Task Group 211 (TG211) appointed by the American Association of Physicists in Medicine (AAPM). METHODS: The recommendations published in the AAPM TG211 report were used to derive a set of required features and to guide the design and structure of a benchmarking software tool. These items included the selection of appropriate representative data and reference contours obtained from established approaches and the description of available metrics. The benchmark was designed in a way that it could be extendable by inclusion of bespoke segmentation methods, while maintaining its main purpose of being a standard testing platform for newly developed PET-AS methods. An example of implementation of the proposed framework, named PETASset, was built. In this work, a selection of PET-AS methods representing common approaches to PET image segmentation was evaluated within PETASset for the purpose of testing and demonstrating the capabilities of the software as a benchmark platform. RESULTS: A selection of clinical, physical, and simulated phantom data, including "best estimates" reference contours from macroscopic specimens, simulation template, and CT scans was built into the PETASset application database. Specific metrics such as Dice Similarity Coefficient (DSC), Positive Predictive Value (PPV), and Sensitivity (S), were included to allow the user to compare the results of any given PET-AS algorithm to the reference contours. In addition, a tool to generate structured reports on the evaluation of the performance of PET-AS algorithms against the reference contours was built. The variation of the metric agreement values with the reference contours across the PET-AS methods evaluated for demonstration were between 0.51 and 0.83, 0.44 and 0.86, and 0.61 and 1.00 for DSC, PPV, and the S metric, respectively. Examples of agreement limits were provided to show how the software could be used to evaluate a new algorithm against the existing state-of-the art. CONCLUSIONS: PETASset provides a platform that allows standardizing the evaluation and comparison of different PET-AS methods on a wide range of PET datasets. The developed platform will be available to users willing to evaluate their PET-AS methods and contribute with more evaluation datasets.

Duke Scholars

Published In

Med Phys

DOI

EISSN

2473-4209

Publication Date

August 2017

Volume

44

Issue

8

Start / End Page

4098 / 4111

Location

United States

Related Subject Headings

  • Tomography, X-Ray Computed
  • Software
  • Phantoms, Imaging
  • Nuclear Medicine & Medical Imaging
  • Image Processing, Computer-Assisted
  • Humans
  • Algorithms
  • 5105 Medical and biological physics
  • 4003 Biomedical engineering
  • 1112 Oncology and Carcinogenesis
 

Citation

APA
Chicago
ICMJE
MLA
NLM
Berthon, B., Spezi, E., Galavis, P., Shepherd, T., Apte, A., Hatt, M., … Kirov, A. S. (2017). Toward a standard for the evaluation of PET-Auto-Segmentation methods following the recommendations of AAPM task group No. 211: Requirements and implementation. Med Phys, 44(8), 4098–4111. https://doi.org/10.1002/mp.12312
Berthon, Beatrice, Emiliano Spezi, Paulina Galavis, Tony Shepherd, Aditya Apte, Mathieu Hatt, Hadi Fayad, et al. “Toward a standard for the evaluation of PET-Auto-Segmentation methods following the recommendations of AAPM task group No. 211: Requirements and implementation.Med Phys 44, no. 8 (August 2017): 4098–4111. https://doi.org/10.1002/mp.12312.
Berthon, Beatrice, et al. “Toward a standard for the evaluation of PET-Auto-Segmentation methods following the recommendations of AAPM task group No. 211: Requirements and implementation.Med Phys, vol. 44, no. 8, Aug. 2017, pp. 4098–111. Pubmed, doi:10.1002/mp.12312.
Berthon B, Spezi E, Galavis P, Shepherd T, Apte A, Hatt M, Fayad H, De Bernardi E, Soffientini CD, Ross Schmidtlein C, El Naqa I, Jeraj R, Lu W, Das S, Zaidi H, Mawlawi OR, Visvikis D, Lee JA, Kirov AS. Toward a standard for the evaluation of PET-Auto-Segmentation methods following the recommendations of AAPM task group No. 211: Requirements and implementation. Med Phys. 2017 Aug;44(8):4098–4111.

Published In

Med Phys

DOI

EISSN

2473-4209

Publication Date

August 2017

Volume

44

Issue

8

Start / End Page

4098 / 4111

Location

United States

Related Subject Headings

  • Tomography, X-Ray Computed
  • Software
  • Phantoms, Imaging
  • Nuclear Medicine & Medical Imaging
  • Image Processing, Computer-Assisted
  • Humans
  • Algorithms
  • 5105 Medical and biological physics
  • 4003 Biomedical engineering
  • 1112 Oncology and Carcinogenesis