To conduct effectiveness research in mental health, many nontrivial problems need to be addressed. A multidisciplinary expert panel designed an outcomes module for major depression and dysthymia to measure disease-specific outcomes, treatments routinely provided, and patient characteristics that influence treatment or its outcomes. The outcomes module was pilot-tested to evaluate its ability to identify a diagnostically homogeneous group of patients, to establish its reliability and validity, and to assess the feasibility of administering the module in a mental health setting. In a cohort of 40 patients, the module identified all 31 patients who had a research diagnosis of major depression or dysthymia (100% sensitivity), and 2 of 9 patients who did not have this diagnosis (77.8% specificity). The outcomes module measured key constructs accurately enough to discriminate between groups of depressed patients expected to differ, indicating positive construct validity. High rates of instrument completion supported the feasibility of using the module in specialty settings. Extensive efforts locating patients for follow-up will be needed to draw valid conclusions about treatment effectiveness.