PURPOSE: To develop and determine the reliability of a novel measurement instrument assessing the quality of residents' discharge summaries. METHOD: In 2014, the authors created a discharge summary evaluation instrument based on consensus recommendations from national regulatory bodies and input from primary care providers at their institution. After a brief pilot, they used the instrument to evaluate discharge summaries written by first-year internal medicine residents (n = 24) at a single U.S. teaching hospital during the 2013-2014 academic year. They conducted a generalizability study to determine the reliability of the instrument and a series of decision studies to determine the number of discharge summaries and raters needed to achieve a reliable evaluation score. RESULTS: The generalizability study demonstrated that 37% of the variance reflected residents' ability to generate an adequate discharge summary (true score variance). The decision studies estimated that the mean score from six discharge summary reviews completed by a unique rater for each review would yield a reliability coefficient of 0.75. Because of high interrater reliability, multiple raters per discharge summary would not significantly enhance the reliability of the mean rating. CONCLUSIONS: This evaluation instrument reliably measured residents' performance writing discharge summaries. A single rating of six discharge summaries can achieve a reliable mean evaluation score. Using this instrument is feasible even for programs with a limited number of inpatient encounters and a small pool of faculty preceptors.