How Values and Uncertainty Shape Scientific Advance in Peer Review

Tens of thousands of scientists contribute to peer review as journal editors and reviewers of the millions of manuscripts submitted every year. How do they decide what is quality work? What values do they apply in evaluating which science merits publication and which does not? How do they respond to dissensus and uncertainty? Who has the greatest influence over the final outcome? This study combines close reading with large language models to analyze 80,000 reviews of 28,000 accepted and rejected manuscripts in engineering and the life sciences. By following reviewers’ value judgments and editorial decisions, we come to a different view of how epistemic cultures are practiced in journal science. Instead of a consensual dialogue revealing salient norms, we find reviewers differently weigh (“commensurate”) their judgments to attribute value to works. Their pluralistic viewpoints elevate uncertainty about the work, and editors respond by aligning with the most negative of reviewers. Surprisingly, we observe engineers and life scientists find the same epistemic criteria are salient, valued, and influential, with novelty and accuracy being primary. These results underscore how contingency and uncertainty are structural features of STEM peer review and essential to its effectiveness and legitimacy.

Duke Scholars

Author Daniel Scott Smith Sociology