COMPASS

Background

Open Science (OS) and Responsible Research and Innovation (RRI) represent key concepts of the European research funding programme Horizon Europe. To foster an OS & RRI transformation at Charité – Universitätsmedizin Berlin, one of the largest university hospitals in Europe, BIH QUEST Center designs and implements OS & RRI programmes to ensure trustworthiness, usefulness, and ethics of biomedical research. To continuously evaluate its programme activities, QUEST Center aims to establish the Monitoring & Evaluation System COMPASS.

Objectives

In COMPASS, a complex multi-level [1] OS & RRI policy and programme implementation is evaluated based on a participatory [2] impact-oriented evaluation design [3] using pre-defined evaluation criteria and indicators. Importantly, conceptual and empirical evaluation elements are examined based on a theory of action (ToA) and theory of change (ToC) programme model [4]. While conceptual elements relate to the programme’s working rationale, empirical elements refer to programme achievements and intended (or unintended) outcomes. Iterative evaluation cycle facilitate the direct utilisation of evaluation results to further improve and align QUEST programme activities with the requirements of Charité researchers on the one hand, and organisational goals of the strategy Charité2030 on the other. Thus, COMPASS will enable evidence-based programme improvement and reconciliation. To ensure high evaluation quality, COMPASS activities are externally advised by CEval, Center for Evaluation (www.ceval.de/).

Methodology

In our work, we benefit from the interdisciplinary composition of our team and the diversity in methodological training. Building on this diverse expertise, we apply mixed & multi method approaches with the aim to create multi-level perspectives. In this, our research questions and research designs address real-life contexts in terms of their particular conditions and challenges with regards to the implementation of OS and RRI.

Primarily, we apply and integrate rigorous quantitative methods and qualitative methods in order to achieve a deeper understanding of these conditions and challenges. Here, the integration of methods means that we either apply both methods in parallel and work with a comparative approach that leads to different types of data from which we can draw multi-level conclusions. In a different research project, we may decide to conduct quantitative research that with its first results can point towards particular aspects that may require further contextualisation and at that point can be investigated in more detail through qualitative methods, and vice versa.

Accordingly, the methodological focus is defined either within a fixed mixed method framework, where the method is derived from the research question, or within emerging mixed method, where the method is selected and constantly adapted according to first research results.

Literature

[1] Widmer, T. & Frey, K. (2006) doi.org/10.5167/uzh-66042;

[2] Bryson, J.M. et al. (2010) doi.org/10.1016/j.evalprogplan.2010.07.001;

[3] Gerstler, P.J. et al. (2016) doi.org/10.1596/978-1-4648-0779-4;

[4] Weiss, C.H. (1997) doi.org/10.1177/0193841X9702100405;

[5] Tashakkori, A. & Teddlie, C. (2010) doi.org/10.4135/9781506335193;

[6] Goertz, G. (2018) doi.org/10.1515/srsr-2018-0081

Implications / Perspectives

COMPASS is a key tool to identify influencing factors that contribute to or counteract the success of QUEST programmes. Based on the (i) COMPASS OS & RRI indicator system, (ii) Output & outcome maps on QUEST OS & RRI activities, (iii) Data collection instruments measuring OS & RRI implementation in biomedical research, and (iv) Evaluation reports incl. recommendations COMPASS provides institutional decision-makers with an evidence base for strategic alignment of organizational governance.