Skip to main content

Many analysts, one data set: Making transparent how variations in analytic choices affect results

Publication ,  Journal Article
Silberzahn, R; Uhlmann, EL; Martin, DP; Anselmi, P; Aust, F; Awtrey, E; Bahník, ; Bai, F; Bannard, C; Bonnier, E; Carlsson, R; Cheung, F ...
Published in: Advances in Methods and Practices in Psychological Science
January 1, 2018

Twenty-nine teams involving 61 analysts used the same data set to address the same research question: whether soccer referees are more likely to give red cards to dark-skin-toned players than to light-skin-toned players. Analytic approaches varied widely across the teams, and the estimated effect sizes ranged from 0.89 to 2.93 (Mdn = 1.31) in odds-ratio units. Twenty teams (69%) found a statistically significant positive effect, and 9 teams (31%) did not observe a significant relationship. Overall, the 29 different analyses used 21 unique combinations of covariates. Neither analysts’ prior beliefs about the effect of interest nor their level of expertise readily explained the variation in the outcomes of the analyses. Peer ratings of the quality of the analyses also did not account for the variability. These findings suggest that significant variation in the results of analyses of complex data may be difficult to avoid, even by experts with honest intentions. Crowdsourcing data analysis, a strategy in which numerous research teams are recruited to simultaneously investigate the same research question, makes transparent how defensible, yet subjective, analytic choices influence research results.

Duke Scholars

Altmetric Attention Stats
Dimensions Citation Stats

Published In

Advances in Methods and Practices in Psychological Science

DOI

EISSN

2515-2467

ISSN

2515-2459

Publication Date

January 1, 2018

Volume

1

Issue

3

Start / End Page

337 / 356

Related Subject Headings

  • 5204 Cognitive and computational psychology
  • 5201 Applied and developmental psychology
 

Citation

APA
Chicago
ICMJE
MLA
NLM
Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E., … Nosek, B. A. (2018). Many analysts, one data set: Making transparent how variations in analytic choices affect results. Advances in Methods and Practices in Psychological Science, 1(3), 337–356. https://doi.org/10.1177/2515245917747646
Silberzahn, R., E. L. Uhlmann, D. P. Martin, P. Anselmi, F. Aust, E. Awtrey, E. Bahník, et al. “Many analysts, one data set: Making transparent how variations in analytic choices affect results.” Advances in Methods and Practices in Psychological Science 1, no. 3 (January 1, 2018): 337–56. https://doi.org/10.1177/2515245917747646.
Silberzahn R, Uhlmann EL, Martin DP, Anselmi P, Aust F, Awtrey E, et al. Many analysts, one data set: Making transparent how variations in analytic choices affect results. Advances in Methods and Practices in Psychological Science. 2018 Jan 1;1(3):337–56.
Silberzahn, R., et al. “Many analysts, one data set: Making transparent how variations in analytic choices affect results.” Advances in Methods and Practices in Psychological Science, vol. 1, no. 3, Jan. 2018, pp. 337–56. Scopus, doi:10.1177/2515245917747646.
Silberzahn R, Uhlmann EL, Martin DP, Anselmi P, Aust F, Awtrey E, Bahník, Bai F, Bannard C, Bonnier E, Carlsson R, Cheung F, Christensen G, Clay R, Craig MA, Rosa AD, Dam L, Evans MH, Cervantes IF, Fong N, Gamez-Djokic M, Glenz A, Gordon-Mckeon S, Heaton TJ, Hederos K, Heene M, Hofelich Mohr AJ, Högden F, Hui K, Johannesson M, Kalodimos J, Kaszubowski E, Kennedy DM, Lei R, Lindsay TA, Liverani S, Madan CR, Molden D, Molleman E, Morey RD, Mulder LB, Nijstad BR, Pope NG, Pope B, Prenoveau JM, Rink F, Robusto E, Roderique H, Sandberg A, Schlüter E, Schönbrodt FD, Sherman MF, Sommer SA, Sotak K, Spain S, Spörlein C, Stafford T, Stefanutti L, Tauber S, Ullrich J, Vianello M, Wagenmakers EJ, Witkowiak M, Yoon S, Nosek BA. Many analysts, one data set: Making transparent how variations in analytic choices affect results. Advances in Methods and Practices in Psychological Science. 2018 Jan 1;1(3):337–356.

Published In

Advances in Methods and Practices in Psychological Science

DOI

EISSN

2515-2467

ISSN

2515-2459

Publication Date

January 1, 2018

Volume

1

Issue

3

Start / End Page

337 / 356

Related Subject Headings

  • 5204 Cognitive and computational psychology
  • 5201 Applied and developmental psychology