BACKGROUND: Pediatric critical care medicine requires the acquisition of procedural skills, but to date no criteria exist for assessing trainee competence in central venous catheter (CVC) insertion. OBJECTIVE: The goal of this study was to create and demonstrate validity evidence for a direct observation tool for assessing CVC insertion. METHODS: Ten experts used the modified Delphi technique to create a 15-item direct observation tool to assess 5 scripted and filmed simulated scenarios of CVC placement. The scenarios were hosted on a dedicated website from March to May 2013, and respondents recruited by e-mail completed the observation tool in real time while watching the scenarios. The goal was to obtain 50 respondents and a total of 250 scenario ratings. RESULTS: A total of 49 pediatrics intensive care faculty physicians (6.3% of 780 potential subjects) responded and generated 188 scenario observations. Of these, 150 (79.8%) were recorded from participants who scored 4 or more on the 5 scenarios. The tool correctly identified the expected reference standard in 96.8% of assessments with an interrater agreement kappa (standard error) = 0.94 (0.07) and receiver operating characteristic = 0.97 (95% CI 0.94-0.99). CONCLUSIONS: This direct observation assessment tool for central venous catheterization demonstrates excellent performance in identifying the reference standard with a high degree of interrater reliability. These assessments support a validity construct for a pediatric critical care medicine faculty member to assess a provider placing a CVC in a pediatrics patient.