OBJECTIVE: To develop and validate an instrument to assess surgical skills during vaginal surgery. STUDY DESIGN: Trainees from 2 institutions were directly assessed in the operating room by supervising surgeons while performing a vaginal hysterectomy using the new Vaginal Surgical Skills Index, global rating scale, and visual analogue scale. Trainees were assessed again by the same surgeons 4 weeks after the live surgery and by a blinded outside reviewer using a videotape of the case. Internal consistency, interrater and intrarater reliability, and construct validity were evaluated. RESULTS: Two hundred twelve evaluations were analyzed on 76 surgeries from 27 trainees. There was good internal consistency, interrater, and intrarater reliability. Vaginal Surgical Skills Index scores correlated with global rating score and visual analog scale scores. Increasing Vaginal Surgical Skills Index scores significantly correlated with year of training and surgical volume with an estimated increase in score of 0.3 per hysterectomy performed. CONCLUSION: The Vaginal Surgical Skills Index is a feasible, reliable, and valid instrument to assess vaginal surgical skills.