Importance of the Quality of Annotation: Impact of Simulated Inter-Observer Variability on Deep Neural Network Automated Segmentation Model Performance.

BACKGROUND: Deep neural network based prostate segmentation depends on manual annotations, yet the effect of annotation variability on model performance remains underexplored. METHODS: Prostate contours were manually delineated by an expert clinician on 119 T2-weighted MR images from the PROSTATEx Challenge 2017 training dataset, and slice-wise synthetic radial modifications of 1-10 mm were applied to create 10 modified training datasets plus an unmodified baseline. Identical SegResNet models were trained with Auto3DSeg/MONAI and evaluated against unmodified validation and test sets using the Dice similarity coefficient (DSC). RESULTS: Mean test DSC decreased from 0.917 for the baseline model to 0.856 at 10 mm modification. Models trained with small annotation perturbations of 1-5 mm maintained DSC values of at least 0.90, whereas performance declined significantly beyond 5 mm. Pairwise DSC agreement across modified annotations also fell as modification amplitude increased. CONCLUSIONS: Prostate segmentation models tolerated modest annotation variability but degraded substantially when variability exceeded 5 mm, underscoring the importance of annotation quality when training and benchmarking DNN-based automated segmentation models.

Duke Scholars

Author Dominic Angelo LaBella Radiation Oncology