The QUEST Toolbox
Want to make your research more reproducible?
The QUEST Toolbox contains helpful tools, programs and online platforms aimed at facilitating the reproducibility of a research project on all stages. The tools are organized by stages in the life cycle of a research project. The QUEST toolbox is aimed specifically at biomedical researchers. While some tools are specific to that community, others would be useful to other research communities. We tried to list non-commercial tools wherever possible.
Preparation of Project
This category focuses on all the steps happening before the actual project is conducted.
Accessing & Bookmarking Literature
Tools that allow you to find and manage scholarly literature:
Unpaywall - Browser extension that searches for a free version of a publication whenever you are on a journal site and grants direct access to PDF via a single click. Behind this is the probably biggest database of open access publications that searches more than 50.000 publishers and repositories. Tool link | Further information
Lazy Scholar - Browser extension similar to unpaywall. It also searches several sources (most importantly Google Scholar) for a free version of a publication. Additional comfort functions are citation generation and additional publication and journal metrics like citation count. Tool link
PrePrint Search Engine - Scientific literature search engine that is specifically targeted at preprints. Searches many of the biggest and most used preprint repositories. Tool link
CiteULike - Web service which allows users to save and share citations to academic papers. Based on the principle of social bookmarking, the site works to promote and to develop the sharing of scientific references amongst researchers. Tool link
Zotero - Open-source reference manager that comes with a browser extension for easy reference import. Tool link
Tools that help you design your experiment. To make your research more reproducible it is important to clarify certain design features of your experiment beforehand. One of them is power analysis to estimate what sample size is appropriate for your experiment. Others are implementing measures to reduce the risk of subjective bias in your experiment, like blinding and randomization.
Experimental Design Assistant - Free online tool that helps you designing your experiment in the best possible way with measures to reduce subjective bias and appropriate statistical analysis. Allows to visualize your experimental design and share and discuss it with others. Tool link | Video
LabWorm - Database of scientific online tools that is constantly filled and voted upon by the scientific community. Find the most used and liked tools for your specific field of research. Tool link
Protocols.io - Open access repository for experimental methods. Search for experimental protocols designed by others or share your own protocol to increase your visibility. Tool link | Further information
Quick Randomization Calculator - Tool to randomly assign subjects to treatment groups. Tool link
How to integrate sex and gender into research – Collection of guidelines, tools and resources to help researchers and reviewers better account for sex and gender in health research. Tool Link
When the project design (including experimental design and statistical analysis) is set up, it is time to preregister your work. When you preregister your experiment, including your statistical analysis, it clarifies and makes it transparent which analyses were planned ahead of time and which new, exploratory analysis steps were added only later. Preregistration can help to reduce publication bias and outcome-switching by deciding on an analysis plan before the data were obtained. To get an idea on how a possible preregistration might look like, you can have a look at: Further information
Preregistration on OSF - Upload your project protocol and invite all your project collaborators. Either keep the protocol private and for project-internal use only at first or share it openly so everybody can see what you are working on. Tool link
AsPredicted - Very slim and easy tool for preregistration that asks 9 questions and produces a standardized preregistration document out of this. Tool link
preclinicaltrials.eu - PreclinicalTrials is an online register of protocols for preclinical animal studies. Increase the transparency and robustness of your research by recording the measures used to reduce the risk of bias. Tool link
Animal Study Registry – Online registry specifically focused on the preregistration of animal studies. You can set an embargo period to keep your registration private in the beginning and your registration gets a DOI. Tool link
Execution of Project
This category focuses on the steps that happen after data collection, especially statistical analysis and visualization.
Many modern scientific projects require a complex processing and statistical analysis of the data obtained during the experimental phase. To make the methods used in the analysis more transparent and reproducible (both for yourself as well as others) you can use the following programming tools:
Git - Version control system that allows you to log your process during the generation of the analysis code. Save functioning versions of your code and build your analysis pipeline step by step. Tool link
Notebooks with R and RStudio - Integrate your R analysis code into a notebook that contains the analysis code, the output (e.g. figures) as well as text to explain your analysis methods. Use the notebook interactively or generate pdf or html documents with a single click. Tool link | Further information
Jupyter Notebooks - Interactive notebooks to make your analysis code reproducible. Originally developed for Python, but nowadays it can be used for a variety of other programming languages, including R, Julia, C++ and many others. Tool link | Installation link via Anaconda | Further information
These tools allow investigators with no programming expertise to make static and interactive graphics for scientific papers. You interactively try out different ways of visualizing your data and produce publication quality plots. Later, readers can upload the datafile for the interactive graphic into the tool to explore the entire dataset, or examine a particular plot in more detail. Each tool allows you to create different types of graphs.
Interactive Dotplot - Create dot plots, box plots, violin plots and combinations of these plots. Bar graphs are included for educational purposes, but are not recommended. You can also examine subgoups (i.e. male vs. female) or show clusters of non-independent data (i.e. replicates, mice from the same litter). Tool link | Further information
Interactive Line Graph - This tool is designed for small datasets and makes it easier to examine the amount of overlap between groups and determine whether all individuals follow the same response pattern. You can view different summary statistics, focus on groups, time points or conditions of interest, examine lines for any individual in the dataset, or view change scores for any two time points or conditions. Tool link | Further information
Interactive Figures for Repeated Independent Experiments - This tool allows you to display data for repeated independent experiments. For example, you might use it to examine data for three independent experiments that compare cell counts each day for 5 days in cells exposed to drug vs. placebo treatment. Tool link
Electronic Lab Notebook (eLN)
We offer an eLN for interested BIH researchers from the Charité and perspectively the MDC. The eLN meets the criteria of Good Laboratory Practice (GLP), respective DIN EN ISO regulations as well as 21CFR11 of the FDA.
While we have decided to offer the labfolder eLN, we generally support the idea of a transition from paper to eLN, regardless of the choice of a particular system. A comprehensive list of different eLN maintained by the Harward Medical School can be found here.
Follow the Links for further information
Managing the risk of re-identification
In biomedical research, it is important to design data management workflows in a way that ensures that sensitive data can only be attributed to patients or study participants when necessary. Re-identification risk management is a process in which quantitative or qualitative methods are used to estimate, document and potentially reduce the risk of unintended attribution. It can support with assessing the privacy impact of data processing, reducing risks to patients or participants and with applying data minimization principles and anonymization methods to meet legal requirements.
Several high-profile re-identification attacks have demonstrated that simply removing or hiding directly identifying data, such as names, adresses or social security numbers, will often not be enough to achieve high levels of protection. To this, formal approaches, which employ mathematical and statistical models for quantifying risks, transforming data and assessing the impact of modifications on the usefulness of output data, must be employed. For this purpose, tool support is needed.
Different methods and tools have been developed for different types of data (e.g. structured data, textual data or medical images). For structured tabular data the following list of free and open source software tools can serve as a starting point:
- ARX Data Anonymization Tool - Tool link | Further information
- sdcMicro – Tool link | Further information
(Text kindly provided by Fabian Prasser, BIH)
Publication of Project Results
There are different ways to make your research easily accessible to others. You can post a preprint of your publication before going through peer review or you can choose an open access journal for publication. You can post your publication in a repository even if you did not publish in an open access journal (often after an embargo period). Additionally, you can make your research data open and citable using a suitable repository.
Open Access Journals
Open Access Journal positive list - A list of biomedical open access journals that obey certain quality standards. A good starting point if you want to look for suitable open access journals for your research field. Contains additional information for each journal like publication costs or typical time until publication. Tool link
Directory of Open Access Journals (DOAJ) - A more in-depth and comprehensive database of open access journals for all research fields. Journals have to go through a quality assessment before they are listed on DOAJ. Tool link
Publication Fund Charité - For Charité-researchers only. Many open access journals charge an article processing charge (APC). These costs can be covered by the publication fund if the total publication costs including VAT do not exceed €2000 per article. Tool link
FAQ Open Data
Open sharing of research data allows reanalysis and synthesis of data and supports reproducibility. Thus, it is strongly encouraged by governments, journals and scientific institutions. The FAQ Open Data answers the “why”, “where”, and “how” of Open Data, provides further useful information and gives contact details for support on the topic.
Repositories for Open Data
Repositories allow you to upload all material connected to your research project - so not only the publication but also associated data or analysis scripts.
Zenodo - A repository that supports DOI versioning. This makes your repository submits citable. It has no size limit. Tool link
Figshare - Another all-purpose repository. Can upload files up to 5GB. Tool link
Preprint servers are used to post a preliminary version of your publication already before submitting it to a peer reviewed journal. The advantage is that you can make your work visible early on and get comments from others to improve your paper further.
bioRxiv - Preprint server for biomedical science. Tool link