Meta-research reading list for peer reviewers
Looking for ways to improve your manuscripts and peer reviews? This "science of science" reading list can help!
When reviewing papers or writing your own papers, it’s important to remember that the fact that something is standard practice for your field doesn’t mean that it’s the best way of doing things. Scientists, journals and funding agencies are increasingly recognising the limitations of many existing practices and are implementing new policies to improve transparency, rigour and reproducibility. The eLife Ambassadors meta-research group has prepared a list of meta-research articles for authors and peer reviewers. These "science of science" papers will help peer reviewers learn to identify and understand the problems with some very common practices.
This collection of articles also offers constructive solutions that make it easier for authors to improve transparency, rigour and reproducibility. Citing relevant meta-research articles in reviews is one step that peer reviewers can take to improve the quality of the scientific literature. While the articles in the list focus on practices that are common in the basic biomedical sciences, these practices are also used in many other fields that typically have small sample sizes. Each topic is clearly labelled, so it’s easy to identify articles that are relevant to your discipline.
If you participate in a pre-print or peer review journal club (such as ASAPBio), you may want to try discussing some of these articles at one of your sessions. Sharing information about best practices with your colleagues may improve the quality of future discussions and reviews.
Meta-research articles and educational articles
Meta-research articles to improve transparency & reproducibility
Why you shouldn’t use bar graphs to show continuous data (and what to do instead):
http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002128
Follow-up paper with information on how to recognise & fix other common visualisation problems:https://www.ahajournals.org/doi/10.1161/CIRCULATIONAHA.118.037777Free online tools for creating better graphics for scientific publications:
- Interactive dotplots, box plots and violin plots:
https://www.jbc.org/article/S0021-9258(20)32792-7/pdf - Interactive line graphs:
http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002484 - Other free tools for creating static graphics: https://twitter.com/T_Weissgerber/status/953334933019398145
How to create clear and informative image-based figures (photos, microscopy, electron microscopy, clinical imaging techniques):
https://journals.plos.org/plosbiology/article/metrics?id=10.1371/journal.pbio.3001161- Creating image-based figures for publications (a Fiji workflow):
https://f1000research.com/articles/9-1373 - How to report imaging methods:
https://elifesciences.org/articles/55133
The problem with underpowered studies:
https://www.nature.com/articles/nrn3475Why it’s important to report all excluded observations & the reasons for exclusion:
http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002331P-values are often reported incorrectly - Why it’s important to present the information needed to verify the test result:
https://link.springer.com/article/10.3758/s13428-015-0664-2Why we need to report more than "Data were analyzed by t-tests or ANOVA":
https://elifesciences.org/articles/36163Why you can’t conclude that two experimental effects are different because one is statistically significant, and the other is not:
https://www.nature.com/articles/nn.2886When a case-control study is not a case-control study:
https://www.medicaljournals.se/jrm/content/abstract/10.2340/16501977-0343Unblinded studies find larger effects:
http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002190Animal research - follow the ARRIVE guidelines to improve transparency and reproducibility:
- http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1001756
- https://www.biorxiv.org/content/early/2017/09/12/187245
- https://elifesciences.org/articles/13615
Animal research - multi-lab studies may improve reproducibility:
http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.2003693Check for clusters of non-independent data (replicates, mice from the same litter, etc.): Did the authors account for non-independence in their analysis?:
https://bmcneurosci.biomedcentral.com/articles/10.1186/1471-2202-11-5Beware of image manipulation:
http://mbio.asm.org/content/7/3/e00809-16.long- Interactive dotplots, box plots and violin plots:
Educational articles
Common misconceptions about data analysis and statistics:
http://jpet.aspetjournals.org/content/351/1/200.full.pdf+htmlStrategies for improving data analysis and reporting in experimental biology:
http://jpet.aspetjournals.org/content/372/1/136Small samples are more likely to give spurious results:
http://www.jstor.org/stable/27858964Transparency is the key to quality:
http://www.jbc.org/content/290/50/29692.full.pdf+htmlResearch methods - know when your numbers are significant:
http://www.nature.com/nature/journal/v492/n7428/full/492180a.htmlA shorter version of this list was originally developed by the eLife Ambassadors Meta-research team and posted on ECR Life.
