COMPASS

Background

Open Science (OS) and Responsible Research and Innovation (RRI) represent key concepts of the European research funding programme Horizon Europe. To foster an OS & RRI transformation at Charité – Universitätsmedizin Berlin, one of the largest university hospitals in Europe, BIH QUEST Center designs and implements OS & RRI programmes to ensure trustworthiness, usefulness, and ethics of biomedical research. To continuously evaluate its programme activities, QUEST Center aims to establish the Monitoring & Evaluation System COMPASS.

Objectives

In COMPASS, a complex multi-level [1] OS & RRI policy and programme implementation is evaluated based on a participatory [2] impact-oriented evaluation design [3] using pre-defined evaluation criteria and indicators. Importantly, conceptual and empirical evaluation elements are examined based on a theory of action (ToA) and theory of change (ToC) programme model [4]. While conceptual elements relate to the programme’s working rationale, empirical elements refer to programme achievements and intended (or unintended) outcomes. Iterative evaluation cycle facilitate the direct utilisation of evaluation results to further improve and align QUEST programme activities with the requirements of Charité researchers on the one hand, and organisational goals of the strategy Charité2030 on the other. Thus, COMPASS will enable evidence-based programme improvement and reconciliation. To ensure high evaluation quality, COMPASS activities are externally advised by CEval, Center for Evaluation (www.ceval.de/).

Team

A joint interdisciplinary research process is crucial for our work as complex issues require multiple disciplinary and methodological perspectives, for example when developing theoretical frameworks addressing contemporary challenges related to OS & RRI cultural changes in biomedical research, but also in academia in general. Hence, team members bring in a variety of academic backgrounds from the natural and social sciences which enables a highly interdisciplinary and dynamic approach to diverse evaluation projects.

We pursue a vision of transdisciplinary research which includes joint conceptualisation of research questions and tasks throughout the research process.

Christiane Wetzel, as head of QUEST Program Evaluation, is a neuroscientist, science manager, and evaluator. She was trained in Biochemistry, received her PhD in Neuroscience from Freie Universität Berlin, and in 2019 a MSc in Science Management from Technische Universität Berlin. Conducting pre-clinical research at Max Delbrück Center for Molecular Medicine, Berlin, and at the Hospital for Sick Children and University of Toronto, Canada, she has many years of experience in technology transfer, cooperated with industry on an early drug development research project, and holds a patent. At QUEST, she expanded her research field to science policy and management, focusing on the development of concepts such as Open Science and Responsible Research and Innovation in research-performing organisations.

Aurelie Vasanta studies Philosophy and Economics at Humboldt-Universität zu Berlin. Her research interests are in the field of analytical philosophy and ethics. At QUEST, she is part of the ELN implementation team, where she consults research groups.

Team Website

Methodology

In our work, we benefit from the interdisciplinary composition of our team and the diversity in methodological training. Building on this diverse expertise, we apply mixed & multi method approaches with the aim to create multi-level perspectives. In this, our research questions and research designs address real-life contexts in terms of their particular conditions and challenges with regards to the implementation of OS and RRI.

Primarily, we apply and integrate rigorous quantitative methods and qualitative methods in order to achieve a deeper understanding of these conditions and challenges. Here, the integration of methods means that we either apply both methods in parallel and work with a comparative approach that leads to different types of data from which we can draw multi-level conclusions. In a different research project, we may decide to conduct quantitative research that with its first results can point towards particular aspects that may require further contextualisation and at that point can be investigated in more detail through qualitative methods, and vice versa.

Accordingly, the methodological focus is defined either within a fixed mixed method framework, where the method is derived from the research question, or within emerging mixed method, where the method is selected and constantly adapted according to first research results.

Literature

[1] Widmer, T. & Frey, K. (2006) doi.org/10.5167/uzh-66042;

[2] Bryson, J.M. et al. (2010) doi.org/10.1016/j.evalprogplan.2010.07.001;

[3] Gerstler, P.J. et al. (2016) doi.org/10.1596/978-1-4648-0779-4;

[4] Weiss, C.H. (1997) doi.org/10.1177/0193841X9702100405;

[5] Tashakkori, A. & Teddlie, C. (2010) doi.org/10.4135/9781506335193;

[6] Goertz, G. (2018) doi.org/10.1515/srsr-2018-0081

Implications / Perspectives

COMPASS is a key tool to identify influencing factors that contribute to or counteract the success of QUEST programmes. Based on the (i) COMPASS OS & RRI indicator system, (ii) Output & outcome maps on QUEST OS & RRI activities, (iii) Data collection instruments measuring OS & RRI implementation in biomedical research, and (iv) Evaluation reports incl. recommendations COMPASS provides institutional decision-makers with an evidence base for strategic alignment of organizational governance.