WT BIH MERIT

Implementing mechanisms of robustness and transparency into the assessments of translational research

WT BIH MERIT is a science-based, iterative policy implementation project that aims to improve the assessment of research by introducing Robustness and reproducibility indicators ("QUEST criteria") and Robust and transparent evaluations (good evaluation practice guidelines, GEP) into the institutional research assessment for funding, recruitment and doctoral programmes at the BIH/Charité. The robustness and reproducibility indicators are sets of criteria used to assess research, while the good evaluation practice guidelines aim to improve the way in which the reviewers select individuals or projects.

The overarching aim is to incentivise and reward robust and reproducible research through structured research assessments in order to increase the probability of patient-oriented translational research at the BIH/Charité.

The underlying assumptions are that robustness and transparency in biomedical research precede translation. Both are multifactorial and are often implicitly applied to research and its assessment of funding and recruitment. Making them explicit in a structured manner for institutional incentive systems supports the internal preoccupation with robustness and transparency of both the applying researcher and the reviewer. However, the steering potential is only fully realised if the selection processes themselves are robust and reproducible. A good evaluation practice aims to increase transparency and accountability of the governance bodies throughout the whole application and selection process. The application of the QUEST criteria as an additional pillar of research assessment in combination with robust and reproducible assessment procedures might alter the selection processes towards more robust and reproducible research.

The so-called QUEST criteria relate to the mechanisms of robust and reproducible research, such as: 1) priority setting, 2) strategies for establishing scientific rigor, 3) strategies for ensuring transparency and disseminating results, and 4) stakeholder engagement. They are a new, additional tool for assessing projects proposals and publications. The GEP guidelines aim to improve the selection process through more transparency, accountability and robustness. Both refer to the QUEST framework, build on the QUEST existing literature, and are further developed for application in various settings within the institutional context of the BIH/Charité.

The areas of implementation include:

1. Intramural funding schemes, such as the (Junior) Clinician Scientist Program, the Validation Fund or the Gender Equality Fund
2. Recruitment committees
3. Doctoral programmes

The target groups of MERIT are institutional research governance bodies (e.g., reviewers, committees and boards).

The implementation process comprises the following steps:

• Field assessment and access to the field, as well as integration in existing workflows in the different areas of implementation 
• The development of the evaluation tool (QUEST criteria and GEP guidelines)
  • Theory validation (science-based, policy-based)
  • Piloting
  • Adaptation
  • Implementation
• Capacity building (teaching tools, workshops, etc. for internal applicants)
• Project evaluation and analysis (infrastructure, process, outcome)

The projects deliverables include the development of different sets of robustness and reproducibility indicators ("QUEST criteria"), and their successful integration into the application/review forms of the institutional funding lines, recruitment committees and doctoral programmes (years 1 to 2). Another project output is the development of evaluation guides for reviewers and board members in the respective areas of implementation (years 1 to 2).

This research is funded by the Wellcome Trust under the framework of the Charité/BIH Translational Partnership.