Skip to main content
Journal cover image

The normality assumption on between-study random effects was questionable in a considerable number of Cochrane meta-analyses.

Publication ,  Journal Article
Liu, Z; Al Amer, FM; Xiao, M; Xu, C; Furuya-Kanamori, L; Hong, H; Siegel, L; Lin, L
Published in: BMC Med
March 29, 2023

BACKGROUND: Studies included in a meta-analysis are often heterogeneous. The traditional random-effects models assume their true effects to follow a normal distribution, while it is unclear if this critical assumption is practical. Violations of this between-study normality assumption could lead to problematic meta-analytical conclusions. We aimed to empirically examine if this assumption is valid in published meta-analyses. METHODS: In this cross-sectional study, we collected meta-analyses available in the Cochrane Library with at least 10 studies and with between-study variance estimates > 0. For each extracted meta-analysis, we performed the Shapiro-Wilk (SW) test to quantitatively assess the between-study normality assumption. For binary outcomes, we assessed between-study normality for odds ratios (ORs), relative risks (RRs), and risk differences (RDs). Subgroup analyses based on sample sizes and event rates were used to rule out the potential confounders. In addition, we obtained the quantile-quantile (Q-Q) plot of study-specific standardized residuals for visually assessing between-study normality. RESULTS: Based on 4234 eligible meta-analyses with binary outcomes and 3433 with non-binary outcomes, the proportion of meta-analyses that had statistically significant non-normality varied from 15.1 to 26.2%. RDs and non-binary outcomes led to more frequent non-normality issues than ORs and RRs. For binary outcomes, the between-study non-normality was more frequently found in meta-analyses with larger sample sizes and event rates away from 0 and 100%. The agreements of assessing the normality between two independent researchers based on Q-Q plots were fair or moderate. CONCLUSIONS: The between-study normality assumption is commonly violated in Cochrane meta-analyses. This assumption should be routinely assessed when performing a meta-analysis. When it may not hold, alternative meta-analysis methods that do not make this assumption should be considered.

Duke Scholars

Published In

BMC Med

DOI

EISSN

1741-7015

Publication Date

March 29, 2023

Volume

21

Issue

1

Start / End Page

112

Location

England

Related Subject Headings

  • Sample Size
  • Odds Ratio
  • Humans
  • General & Internal Medicine
  • Cross-Sectional Studies
  • 42 Health sciences
  • 32 Biomedical and clinical sciences
  • 11 Medical and Health Sciences
 

Citation

APA
Chicago
ICMJE
MLA
NLM
Liu, Z., Al Amer, F. M., Xiao, M., Xu, C., Furuya-Kanamori, L., Hong, H., … Lin, L. (2023). The normality assumption on between-study random effects was questionable in a considerable number of Cochrane meta-analyses. BMC Med, 21(1), 112. https://doi.org/10.1186/s12916-023-02823-9
Liu, Ziyu, Fahad M. Al Amer, Mengli Xiao, Chang Xu, Luis Furuya-Kanamori, Hwanhee Hong, Lianne Siegel, and Lifeng Lin. “The normality assumption on between-study random effects was questionable in a considerable number of Cochrane meta-analyses.BMC Med 21, no. 1 (March 29, 2023): 112. https://doi.org/10.1186/s12916-023-02823-9.
Liu Z, Al Amer FM, Xiao M, Xu C, Furuya-Kanamori L, Hong H, et al. The normality assumption on between-study random effects was questionable in a considerable number of Cochrane meta-analyses. BMC Med. 2023 Mar 29;21(1):112.
Liu, Ziyu, et al. “The normality assumption on between-study random effects was questionable in a considerable number of Cochrane meta-analyses.BMC Med, vol. 21, no. 1, Mar. 2023, p. 112. Pubmed, doi:10.1186/s12916-023-02823-9.
Liu Z, Al Amer FM, Xiao M, Xu C, Furuya-Kanamori L, Hong H, Siegel L, Lin L. The normality assumption on between-study random effects was questionable in a considerable number of Cochrane meta-analyses. BMC Med. 2023 Mar 29;21(1):112.
Journal cover image

Published In

BMC Med

DOI

EISSN

1741-7015

Publication Date

March 29, 2023

Volume

21

Issue

1

Start / End Page

112

Location

England

Related Subject Headings

  • Sample Size
  • Odds Ratio
  • Humans
  • General & Internal Medicine
  • Cross-Sectional Studies
  • 42 Health sciences
  • 32 Biomedical and clinical sciences
  • 11 Medical and Health Sciences