2011-02-25 (Creation date: 2011-02-25)

Carole J. Lee

The incentives, review processes, and norms of peer review provide the foundation for sustaining the piecemeal contribution and critique of knowledge in scientific communities. However, a growing industry of empirical research challenges the normative well-functioning of expert peer review. Inter-rater reliability rates for expert reviewers is low enough to be considered “poor” by psychometric standards. Moreover, publication biases skew the results of meta-analyses and suppress the communication of results that should inform the design of future studies. Rather than adopt a normatively pessimistic interpretation of these results, I suggest that a Kuhnian perspective about the flexibility with which shared evaluative criteria can be interpreted and applied can reframe these results in ways that construe them as normatively appropriate while suggesting new empirical and normative questions. Although this Kuhnian perspective does not ameliorate problematic secondary consequences of low inter-rater reliability rates and publication biases (such as the “luck of the reviewer draw” and skewed meta-analyses), I suggest that modifications to discipline-wide communication structures can be adopted to address these concerns.

IU Workshop in Methods

Workshop in Methods

Social Science Research Commons

Performers
Dr. Leeis Assistant Professor of Philosophy at the University of Washington. She received her BA from Wellesley College and her PhD from the University of Michigan in 2006. Her primary research interests are in philosophy of psychology, epistemology, and philosophy of social science.

This is the first in a series of WIM-hosted presentations that will address the topic of "Going amiss in scientific research." Thanks to Jutta Schickore (IU - History and Philosophy of Science) for the ideas and the support.