Quality Assurance
Automated screening tools
Automated screening of scientific manuscripts can help authors to identify and fix common problems, such as failing to state whether experiments were blinding or randomised, using potentially misleading bar graphs to present continuous data, or failing to acknowledge study limitations. Tools can screen a manuscript and provide authors with customised feedback in seconds. This makes automated screening a valuable strategy for improving transparency and reproducibility on a large scale, across many fields.
At QUEST, we have developed several new screening tools and are founding members of an international working group that combines many different tools into a powerful screening pipeline (ScreenIT).
Barzooka
Barzooka is a deep convolutional neural network that screens publication PDFs and checks for bar graphs of continuous data and other common graphing issues. Many different data distributions can lead to the same bar graph and the actual data may suggest different conclusions from the summary statistics alone. Barzooka also detects more informative alternatives to bar graphs, like dot plots, box plots and histograms.
Why you shouldn’t use bar graphs of continuous data, and what to use instead:
1. https://www.ahajournals.org/doi/10.1161/CIRCULATIONAHA.118.037777
2. https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002128
ScreenIT pipeline
The Automated Screening Working Group is an international group of tool creators working to improve scientific manuscripts. This group was co-founded by QUEST members. Group members have combined their tools into the ScreenIT pipeline, which screens for common problems that can affect transparency or reporting and provides feedback to authors. Throughout the COVID-19 pandemic, we have been using our automated ScreenIT pipeline to screen COVID-19 preprints on medRxiv and bioRxiv. Public reports are automatically posted via hypothes.is and tweeted out via @SciScoreReports.
For more details on the international working group and the ScreenIT pipeline, see:
The ScreenIT pipeline includes the following tools:
SciScore
Blinding, randomisation, sample-size calculations, sex/gender, ethics and consent statements, resources, RRIDs
http://sciscore.com; RRID:SCR_016251
ODDPub
Open data, open code
https://github.com/quest-bih/oddpub; RRID:SCR_018385
Limitation-Recognizer
Author-acknowledged limitations
https://github.com/kilicogluh/limitation-recognizer; RRID:SCR_018748
Barzooka
Bar graphs of continuous data
https://quest-barzooka.bihealth.org; RRID:SCR_018508
JetFighter
Rainbow colour maps (these colour maps create visual artifacts and aren’t colourblind safe)
https://jetfighter.ecrlife.org; RRID:SCR_018498
Seek and Blastn
Correct identification of nucleotide sequences
http://scigendetection.imag.fr/TPD52/; RRID:SCR_016625
TrialIdentifier
Checks clinical trial registration numbers from ClinicalTrials.gov
https://github.com/bgcarlisle/TRNscreener; RRID:SCR_019211
rtransparent
Statements on conflicts of interest, funding, or protocol registration
https://github.com/serghiou/rtransparent; RRID:SCR_019276
scite
Citations of retracted publications, or papers with erratums or corrections
http://www.scite.ai/; RRID:SCR_018568
