Qualitätssicherung
QUEST Toolbox (für die eigene Forschung)
Want to make your research more reproducible?
The QUEST Toolbox contains helpful tools, programmes and online platforms aimed at facilitating the reproducibility of a research project on all stages. The tools are organised by stages in the life cycle of a research project. The QUEST toolbox is aimed specifically at biomedical researchers. While some tools are specific to that community, others would be useful to other research communities. We tried to list non-commercial tools wherever possible.
Preparation of project
This category focusses on all the steps happening before the actual project is conducted.
Preparation of project
Tools that allow you to find and manage scholarly literature:
Unpaywall - Browser extension that searches for a free version of a publication whenever you are on a journal site and grants direct access to PDF via a single click. Behind this is the probably largest database of open access publications that searches more than 50.000 publishers and repositories. link | Video
Open Knowledge Maps - A visual search engine for scientific knowledge based on the principles of open science. Recently, the search engine was used to develop CoVis: a curated knowledge map of seminal works on COVID-19 from eight critical areas of biomedical research. link
Lazy Scholar - Browser extension similar to Unpaywall. It also searches several sources (most importantly Google Scholar) for a free version of a publication. Additional comfort functions are citation generation and additional publication and journal metrics like citation count. link
PrePrint Search Engine - Scientific literature search engine that is specifically targeted at preprints. Searches many of the largest and most used preprint repositories. link
Zotero - Open-source reference manager that comes with a browser extension for easy reference import. link
Tools that help you design your experiment. To make your research more reproducible, it is important to clarify certain design features of your experiment beforehand. One of them is power analysis to estimate what sample size is appropriate for your experiment. Others are implementing measures to reduce the risk of subjective bias in your experiment, like blinding and randomisation.
Experimental Design Assistant - Free online tool that helps you designing your experiment in the best possible way with measures to reduce subjective bias and appropriate statistical analysis. Allows to visualise your experimental design and share and discuss it with others. link | Video
G*Power - Tool that helps you conducting your statistical power analysis. link | Further information | Video
bio.tools - Database of scientific online tools that facilitates researchers from across the spectrum of biological and biomedical science to find, understand, utilise and cite the resources they need in their day-to-day work. link
protocols.io - Open access repository for experimental methods. Search for experimental protocols designed by others or share your own protocol to increase your visibility. link | Further information
Bio-Protocols - Peer reviewed journal for research protocols (free and open access). link
Research Randomizer - A tool to generate random numbers or assign participants to experimental conditions. link | Further information
Quick Randomization Calculator - Tool to randomly assign subjects to treatment groups. link
How to integrate sex and gender into research – Collection of guidelines, tools and resources to help researchers and reviewers better account for sex and gender in health research. link
When the project design (including experimental design and statistical analysis) is set up, it is time to preregister your work. When you preregister your experiment, including your statistical analysis, it clarifies and makes it transparent which analyses were planned ahead of time and which new, exploratory analysis steps were added only later. Preregistration can help to reduce publication bias and outcome-switching by deciding on an analysis plan before the data were obtained. To get an idea on how a possible preregistration might look like, you can have a look here.
Preregistration on OSF - Upload your project protocol and invite all your project collaborators. Either keep the protocol private and for project-internal use only at first or share it openly so everybody can see what you are working on. link
AsPredicted - Very slim and easy tool for preregistration that asks 9 questions and produces a standardised preregistration document out of this. link
preclinicaltrials.eu - PreclinicalTrials is an online register of protocols for preclinical animal studies. Increase the transparency and robustness of your research by recording the measures used to reduce the risk of bias. link
Animal Study Registry – Online registry specifically focussed on the preregistration of animal studies. You can set an embargo period to keep your registration private in the beginning and your registration gets a DOI. link
GOT-IT– Guidelines on target assessment and validation for innovative therapeutics – was the accompanying project in the BMBF funded consortium Target validation for pharmaceutical drug compound development. It aimed at aligning quality criteria for the assessment of potential disease-modulating molecules (so called targets) in pre-clinical research, before they can be considered for use in clinical studies (for more information, see here).
Additionally to the Guidelines, the GOT-IT project developed the Critical Path Generator, the Educational Tool and the Experts Platform, freely accessible tools that support scientists during planning and execution of target assessments and validation projects. With these, data robustness and reproducibility as well as rigour in pre-clinical research may be improved, increasing the effectiveness of translating biomedical research into drug discovery.
Critical Path Generator
The Critical Path Generator tool will assist scientists in planning and structuring a target assessment project. The CP Generator will help to arrange important target assessment blocks into a project-specific Critical Path. Using a query form and different sets of guiding questions (Critical Path Questions (CPQs) & Experimental Approach Questions (EAQs)), the CP Generator identifies strengths and weaknesses of a translational project and provides support how to invest critical resources in an optimal way. link
Educational Tool
The Educational Tool for young scientists informs about important aspects of target assessment. The target assessment process is depicted as a train journey. In the Central Station, information is provided for planning a target assessment project. Being on the journey, you will be taken through different train stations that provide additional information about important core aspects of target assessment. In addition, illustrative examples for potential roadblocks, delays and project goals are provided. link
GOT-IT Expert Platform
The GOT-IT recommendations consider the complexity of tasks associated with bridging the gap between basic science research findings and the identification of clinically effective products. The guidelines suggest the need for an environment where academic research teams focussing on newly identified drug targets can connect with experienced and qualified expert industry teams in support of successful drug discovery and development. The GOT-IT Expert Platform is designed to facilitate these interactions. Based on an innovative match-making algorithm, the platform helps academic scientists recruit drug discovery experts, receive timely and well-informed advice, and develop a more extended network of partnerships to ultimately increase the exchange between Industry and Academia. A set of onboarding questions ensures that project-specific goals and individual needs are taken into account so that productive collaborations can provide the most relevant and meaningful support for early-phase translational drug discovery programmes. link
The PREMIER Wiki platform was developed to transparently share, store and further develop knowledge within a department / laboratory / institute.
This structure of an open Wiki system guarantees a continuous exchange of knowledge. All information and documents about processes and internal regulations of a laboratory can be stored here and are available to all employees. Document control, e.g. writing and updating SOPs, can also be done via the PREMIER Wiki. The Wiki is password protected so that each employee has his or her own Wiki account. Reading and writing rights can be assigned variably, depending on existing needs. This means that the Wiki can be adapted precisely to any organisation.
On this PREMIER Wiki platform, you will additionally find the modular clickable PREMIER QMS (the QM house) with all contents. These contents are not editable, they should rather serve as an orientation for your laboratory and help with the introduction of the individual modules. Additionally you will find on this platform the PREMIER template for the experimental design of your research project.
All other content of the Wiki platform is up to you. You decide which content should be shared, maintained and passed on to your colleagues.
The PREMIER toolbox can be found here.
Execution of project
This category focusses on the steps that happen after data collection, especially statistical analysis and visualisation.
Tools execution
Many modern scientific projects require a complex processing and statistical analysis of the data obtained during the experimental phase. To make the methods used in the analysis more transparent and reproducible (both for yourself as well as others) you can use the following programming tools:
Git - Version control system that allows you to log your process during the generation of the analysis code. Save functioning versions of your code and build your analysis pipeline step by step. link
Notebooks with R and RStudio - Integrate your R analysis code into a notebook that contains the analysis code, the output (e.g. figures) as well as text to explain your analysis methods. Use the notebook interactively or generate pdf or html documents with a single click. link | Further information
Jupyter Notebooks - Interactive notebooks to make your analysis code reproducible. Originally developed for Python, but nowadays it can be used for a variety of other programming languages, including R, Julia, C++ and many others. link | Installation link via Anaconda | Further information
We offer an eLN for interested BIH researchers from the Charité and perspectively the MDC. The eLN meets the criteria of Good Laboratory Practice (GLP), respective DIN EN ISO regulations as well as 21CFR11 of the FDA.
Further information about registration and the use of the labfolder eLN you will find here.
While we have decided to offer the labfolder eLN, we generally support the idea of a transition from paper to eLN, regardless of the choice of a particular system. A comprehensive list of different eLN maintained by the Harvard Medical School can be found here.
These tools allow investigators with no programming expertise to make static and interactive graphics for scientific papers. You interactively try out different ways of visualising your data and produce publication quality plots. Later, readers can upload the datafile for the interactive graphic into the tool to explore the entire dataset, or examine a particular plot in more detail. Each tool allows you to create different types of graphs.
Interactive Dotplot - Create dot plots, box plots, violin plots and combinations of these plots. Bar graphs are included for educational purposes, but are not recommended. You can also examine subgroups (i.e. male vs. female) or show clusters of non-independent data (i.e. replicates, mice from the same litter). link | Further information
Interactive Line Graph - This tool is designed for small datasets and makes it easier to examine the amount of overlap between groups and determine whether all individuals follow the same response pattern. You can view different summary statistics, focus on groups, time points or conditions of interest, examine lines for any individual in the dataset, or view change scores for any two time points or conditions. link | Further information
Interactive Figures for Repeated Independent Experiments - This tool allows you to display data for repeated independent experiments. For example, you might use it to examine data for three independent experiments that compare cell counts each day for 5 days in cells exposed to drug vs. placebo treatment. link
In biomedical research, it is important to design data management workflows in a way that ensures that sensitive data can only be attributed to patients or study participants when necessary. Re-identification risk management is a process in which quantitative or qualitative methods are used to estimate, document and potentially reduce the risk of unintended attribution. It can support with assessing the privacy impact of data processing, reducing risks to patients or participants and with applying data minimisation principles and anonymisation methods to meet legal requirements.
Several high-profile re-identification attacks have demonstrated that simply removing or hiding directly identifying data, such as names, addresses or social security numbers, will often not be enough to achieve high levels of protection. To this, formal approaches, which employ mathematical and statistical models for quantifying risks, transforming data and assessing the impact of modifications on the usefulness of output data, must be employed. For this purpose, tool support is needed.
Different methods and tools have been developed for different types of data (e.g. structured data, textual data or medical images). For structured tabular data, the following list of free and open source software tools can serve as a starting point:
- ARX Data Anonymisation Tool, link |Further information
- sdcMicro, link | Further information
(Text kindly provided by Fabian Prasser, BIH)
Publication of project results
There are different ways to make your research easily accessible to others. You can post a preprint of your publication before going through peer review or you can choose an open access journal for publication. You can post your publication in a repository even if you did not publish in an open access journal (often after an embargo period). Additionally, you can make your research data open and citable using a suitable repository.
Publication of project results
Open Access Journal positive list - A list of biomedical open access journals that obey certain quality standards. A good starting point if you want to look for suitable open access journals for your research field. Contains additional information for each journal like publication costs or typical time until publication. link
Directory of Open Access Journals (DOAJ) - A more in-depth and comprehensive database of open access journals for all research fields. Journals have to go through a quality assessment before they are listed on DOAJ. link
Publication Fund Charité - For Charité/ BIH researchers only. Many open access journals charge an article processing charge (APC). These costs can be covered by the publication fund. link
Open sharing of research data allows reanalysis and synthesis of data and supports reproducibility. Thus, it is strongly encouraged by governments, journals and scientific institutions. The FAQ Open Data answers the "why", "where" and "how" of Open Data, provides further useful information and gives contact details for support on the topic.
Repositories allow you to upload all material connected to your research project - not only the publication but also associated data or analysis scripts.
Zenodo - A repository that supports DOI versioning. This makes your repository submits citable. It has no size limit. link
Figshare - Another all-purpose repository. Can upload files up to 5GB. link
Open Science Framework - Manage and save your research project documents and data before and after publication. link | Video
Preprint servers are used to post a preliminary version of your publication already before submitting it to a peer reviewed journal. The advantage is that you can make your work visible early on and get comments from others to improve your paper further.
bioRxiv - Preprint server for biology. link
medRxiv - Preprint server for health sciences. link
FIDDLE – You would like to publish research outcomes which do not fit into the regular publication pipeline? FIDDLE provides guidance on publishing outcomes like null results or unanalysed datasets, which all too often end up in the file drawer. link
See the publication: fiddle: a tool to combat publication bias by getting research out of the file drawer and into the scientific community, 2020 doi.org/10.1042/CS20201125
CRediT - Clearly assign the credit for the different contributions of the researchers to the project. link | Further information
