Concerns for the reliability and validity of the National Stroke Project Stroke Severity Scale.

BACKGROUND: The National Stroke Project (NSP) was a retrospective cohort study of US Medicare beneficiaries hospitalized with stroke or transient ischemic attack (TIA). The NSP included a simple assessment of stroke severity (NSP-Stroke Scale, NSP-SS). Used for risk adjustment in outcome studies, the reliability and validity of the NSP-SS have not been assessed. We determined the reliability, concurrent and construct validity of the NSP-SS. METHODS: The initial neurologic examinations of 100 consecutive patients hospitalized with ischemic stroke/TIA in a single academic medical center were reviewed. The NSP-SS was retrospectively scored twice by the same rater and independently by a second rater to assess reliability. The National Institutes of Health Stroke Scale (NIH-SS) was also scored retrospectively and used as the criterion standard for concurrent validity. Construct validity was based on discharge status. RESULTS: The NSP-SS had moderate-substantial inter-rater (weighted kappa, κ(w) = 0.66, 95% CI 0.55-0.77) and intra-rater (κ(w) = 0.63, 95% CI 0.52-0.75) reliability. Correlation between NSP-SS and NIH-SS scores was moderate (Spearman r = 0.65, 95% CI 0.52-0.75, p < 0.0001) but some categorizations in the NSP-SS seemed inappropriate reflecting poor content validity. Each NSP-SS point was associated with a greater likelihood of poor outcome (OR = 2.1, 95% CI 1.1-3.7, p = 0.016). Based on dichotomized scores (NSP 0-2 and NIH-SS <6; mild deficits), the NSP-SS sensitivity was 70.9% (95% CI 57.9-81.2%), specificity 82.2% (95% CI 68.7-90.7%), likelihood ratio for severe stroke 4.0 (95% CI 2.1-7.6) and likelihood ratio for mild stroke 0.3 (95% CI 0.20-0.5). The dichotomized NSP-SS and NIH-SS similarly predicted poor outcome (NSP-SS >2, OR = 4.7, 95% CI 1.7-13.0, p = 0.003 vs. NIH-SS ≥6, OR = 4.4, 95% CI 1.5-13.0, p = 0.006) with excellent discrimination (C = 0.827 and 0.826, respectively). CONCLUSION: The NSP-SS has moderate-substantial reliability but poor content validity and poor to moderate concurrent validity as compared with the NIH-SS. In addition, it is not clear that the NSP-SS is easier to extract from medical records than the NIH-SS. Given this, and its other limitations, the utility of this scale for risk adjustment in future stroke outcome studies is questionable.

Duke Scholars

Author Nada Kais El Husseini Neurology, Stroke and Vascular Neurology

Author Larry Bruce Goldstein Neurology, Stroke and Vascular Neurology