Description
In research, peer review is considered a pillar of trust. This is problematic. A lot of research is, at best, not reproducible or, sometimes, even wrong, despite peer review. This talk will discuss the origins of peer review, problems with peer review and some things that could be better.
In this talk I will describe the research publication pipeline that is based on peer review. Peer review in some sense resembles what happens on an open source software pull request but it is also fundamentally different. For example it is often a somewhat secretive affair with ambiguous comments from referees with no room for discussion.
I will discuss the historic origins of peer review and highlight a number of problems that are prevalent as a result.
This will lead to thoughts on the difference between trust and confidence. Should we trust research which implies trusting the peer review process, or should we instead aim to have confidence in the research? And, if that's the goal, how can that be achieved?
I will include some examples of specific peer review processes from pure mathematics (my original field of research) but also discuss topics related to the wider software development industry (such as zero-trust security).
This will conclude with a hopefully optimistic answer to the question: "If we were inventing the research publication pipeline today, what would it look like?".
I hope that this talk will not only be of interest to Python users doing research but also to the wider Python community who might be interested in understanding what "trust the research" means.