COMPASS

Background

Open Science (OS) and Responsible Research and Innovation (RRI) represent key concepts of the European research funding programme Horizon Europe. To foster an OS & RRI transformation at Charité – Universitätsmedizin Berlin, one of the largest university hospitals in Europe, BIH QUEST Center designs and implements OS & RRI programmes to ensure trustworthiness, usefulness, and ethics of biomedical research. To continuously evaluate its programme activities, QUEST Center aims to establish the Monitoring & Evaluation System COMPASS.

Objectives

In COMPASS, a complex multi-level [1] OS & RRI policy and programme implementation is evaluated based on a participatory [2] impact-oriented evaluation design [3] using pre-defined evaluation criteria and indicators. Importantly, conceptual and empirical evaluation elements are examined based on a theory of action (ToA) and theory of change (ToC) programme model [4]. While conceptual elements relate to the programme’s working rationale, empirical elements refer to programme achievements and intended (or unintended) outcomes. Iterative evaluation cycle facilitate the direct utilisation of evaluation results to further improve and align QUEST programme activities with the requirements of Charité researchers on the one hand, and organisational goals of the strategy Charité2030 on the other. Thus, COMPASS will enable evidence-based programme improvement and reconciliation. To ensure high evaluation quality, COMPASS activities are externally advised by CEval, Center for Evaluation (www.ceval.de/).

Team

A joint interdisciplinary research process is crucial for our work as complex issues require multiple disciplinary and methodological perspectives, for example when developing theoretical frameworks addressing contemporary challenges related to OS & RRI cultural changes in biomedical research, but also in academia in general. Hence, team members bring in a variety of academic backgrounds from the natural and social sciences which enables a highly interdisciplinary and dynamic approach to diverse evaluation projects.

We pursue a vision of transdisciplinary research which includes joint conceptualisation of research questions and tasks throughout the research process.

Christiane Wetzel, as head of QUEST Program Evaluation, is a neuroscientist, science manager, and evaluator. She was trained in Biochemistry, received her PhD in Neuroscience from Freie Universität Berlin, and in 2019 a MSc in Science Management from Technische Universität Berlin. Conducting pre-clinical research at Max Delbrück Center for Molecular Medicine, Berlin, and at the Hospital for Sick Children and University of Toronto, Canada, she has many years of experience in technology transfer, cooperated with industry on an early drug development research project, and holds a patent. At QUEST, she expanded her research field to science policy and management, focusing on the development of concepts such as Open Science and Responsible Research and Innovation in research-performing organisations.

Ina Frenzel studied Sociology-Technical direction at Technische Universität Berlin. She is a senior researcher and empiricist focusing on technology research. Applying mainly quantitative research approaches, in the team QUEST Program Evaluation she studies the acceptance and adoption of novel technology, such as electronic labbooks (ELN), and its underlying social innovation.

Sarah Berndt studied Social Sciences and received her PhD in Sociology. Her research focus in the team of Philipp Pohlenz (Prof. for Higher Education Research and Professionalisation of Academic Teaching) at Otto-von-Guericke-University Magdeburg is on science and higher education research and methodology in empirical social science research. At BIH QUEST Center, Sarah is part of the BMBF-funded research project RE-PLACE. She examines social innovation processes associated with new technologies, particularly the electronic lab notebook currently implemented at Charité.

Daniela Schirmer studied Sociology and Political Sciences and has previously worked in training and education. In her PhD project at QUEST and Otto-von-Guericke-Universität Magdeburg (with Prof. Philipp Pohlenz, Higher Education Research and Professionalisation of Academic Teaching), she examines the social challenges of the transformation process sparked by changing research documentation from analogue to digital documentation in electronic labbooks, mainly using qualitative research methodology.

Aurelie Vasanta studies Philosophy and Economics at Humboldt-Universität zu Berlin. Her research interests are in the field of analytical philosophy and ethics. At QUEST, she is part of the ELN implementation team, where she consults research groups.

Sarah Wendt studied Global Studies at Humboldt-Universität zu Berlin, University of Pretoria, Jawaharlal Nehru University New Delhi and FLACSO Argentina with a focus on qualitative social research and global power structures. In the team QUEST Program Evaluation, she conducts qualitative analyses to examine support factors and barriers to the transfer of OS & RRI training and funding measures into research practice.

Team Website

Methodology

In our work, we benefit from the interdisciplinary composition of our team and the diversity in methodological training. Building on this diverse expertise, we apply mixed & multi method approaches with the aim to create multi-level perspectives. In this, our research questions and research designs address real-life contexts in terms of their particular conditions and challenges with regards to the implementation of OS and RRI.

Primarily, we apply and integrate rigorous quantitative methods and qualitative methods in order to achieve a deeper understanding of these conditions and challenges. Here, the integration of methods means that we either apply both methods in parallel and work with a comparative approach that leads to different types of data from which we can draw multi-level conclusions. In a different research project, we may decide to conduct quantitative research that with its first results can point towards particular aspects that may require further contextualisation and at that point can be investigated in more detail through qualitative methods, and vice versa.

Accordingly, the methodological focus is defined either within a fixed mixed method framework, where the method is derived from the research question, or within emerging mixed method, where the method is selected and constantly adapted according to first research results.

Literature

[1] Widmer, T. & Frey, K. (2006) doi.org/10.5167/uzh-66042;

[2] Bryson, J.M. et al. (2010) doi.org/10.1016/j.evalprogplan.2010.07.001;

[3] Gerstler, P.J. et al. (2016) doi.org/10.1596/978-1-4648-0779-4;

[4] Weiss, C.H. (1997) doi.org/10.1177/0193841X9702100405;

[5] Tashakkori, A. & Teddlie, C. (2010) doi.org/10.4135/9781506335193;

[6] Goertz, G. (2018) doi.org/10.1515/srsr-2018-0081

Implications / Perspectives

COMPASS is a key tool to identify influencing factors that contribute to or counteract the success of QUEST programmes. Based on the (i) COMPASS OS & RRI indicator system, (ii) Output & outcome maps on QUEST OS & RRI activities, (iii) Data collection instruments measuring OS & RRI implementation in biomedical research, and (iv) Evaluation reports incl. recommendations COMPASS provides institutional decision-makers with an evidence base for strategic alignment of organizational governance.