OBJECTIVES/HYPOTHESIS: To determine the feasibility, validity, and reliability of an evaluation tool for the assessment of competency in mastoid surgery. This study tests the hypothesis that residents of dissimilar training levels differ in their technical performance as measured by this tool. STUDY DESIGN: Cross-sectional validation study. METHODS: Two or more faculty otolaryngologists evaluated each resident performing a cortical mastoidectomy on a cadaveric temporal bone. Performance was rated using global and checklist components of a mastoidectomy assessment tool. RESULTS: Fifteen internal and external faculty members evaluated 23 residents over 3 years resulting in 118 evaluations. Construct validity was observed as scores increased with clinical training year. These differences were greater for more complex tasks. There was a percentage agreement of 78.5% between evaluator pairs in the determination of pass (score 3-5) versus fail (score 1-2) for the checklist instrument, and an agreement of 74.4% for the global instrument. Although agreement was lower for the exact score on a scale of 1 to 5, differences of 1 or less occurred in over 80% of evaluator pairs. Regression analysis confirmed faculty perception that sharpening of the posterior external auditory canal cortex and opening of the antrum from posterior to anterior are strong predictors of overall surgical performance. CONCLUSIONS: Our results indicate that the tool we have developed is a feasible, valid, and reliable instrument for the assessment of competency in mastoidectomy. The instrument can be used to provide formative feedback and to identify procedural tasks for which additional training may be necessary.