BUA Open Science Dashboards

The project "BUA Open Science Dashboards – Developing Indicators and Screening Tools for their Prototypical Implementation" is conducted in collaboration with the Open Access Office Berlin (OABB), and funded through an Open Call of the BUA Objective 3 "Advancing Research Quality and Value" (see here for an overview of funded projects). It focuses on the systematic monitoring of open science practices, predominantly using semi-automated screening tools, and on the presentation of indicators in dashboards.

The QUEST work package has two objectives, the first of which is largely complete (as of June 2022).

First, we extended the Charité Dashboard on Responsible Research developed at QUEST to include indicators of data reusability based on the FAIR data principles. For this, we screened research datasets published by Charité researchers using existing, externally developed tools, and where necessary we adjusted these tools to our needs. In the next stage of the project, some criteria will also be evaluated manually. For the identification of research data object identifiers and for the manual verification and screening of data sharing we have published a protocol on protocols.io that describes this workflow.

Second, we will take up ideas for open science indicators for other fields of scholarship, as developed by the OABB and disciplinary stakeholders [for more information, see this blog post]. We will use – and possibly adopt - available screening tools to also collect these indicators. Ultimately, we will support the OABB with the generation of pilot dashboards for the two participating disciplines, based on the indicators thus collected.

Third, during the winter semester of 2021/22, the Berlin Science Survey (BSS) was conducted, which is also funded by the BUA Objective 3 “Advancing Research Quality and Value” and located at the Robert K. Merton Zentrum (RMZ). Among other things, there was a special focus on dealing with Open Science. As part of the project “BUA Open Science Dashboards”, the survey results were presented in a separate BSS dashboard for the subset of researchers with Charité affiliation. In this dashboard, the survey results can be broken down by status groups and various characteristics of their research, such as empirical vs. theoretical research or competition in their research field. The first thing presented is the tension that researchers face between external expectations and their own values. This is followed by analyzes of the application of Open Science practices that have already been implemented and the difficulties encountered in implementing them. Finally, the perception of the Berlin research environment in general and the implementation of Open Science in the BUA in particular were evaluated. Overall, the Open Science questions provided by the BSS were presented interactively, allowing for a more in-depth discussion and better utilization of the results for strategy development and communication.

FAIR is a framework for describing the findability and reusability of research data. It is widely claimed that research data should be FAIR (and thus, reusable) for both humans and machines, and there are multiple projects, initiatives and standards, as well as FAIR screening tools and workflows. However, so far none of these tools and workflows have been comprehensively applied to datasets produced by researchers of a particular research institution. Indeed, there are hardly any collections of datasets for which information on their FAIRness is available at all. Our goal is to assess FAIR compliance (‘FAIRness’) for the whole body of datasets shared by Charité researchers, thus giving an overview of data reusability and supporting development by infrastructure providers. Correspondingly, the prime goal of the project BUA Open Science Dashboards is the application of screening tools, which output the compliance of datasets with FAIR criteria. In this case, FAIRness and thus data reusability is understood from the point of view of automated or programmatic access, i.e., “FAIR for machines”. Where necessary, these screening tools can be adopted to our needs, which might make manual steps necessary.

At the same time, the project also builds on the piloting ungeFAIR project, which started in 2020. It was the goal of ungeFAIR to define a subset of FAIR criteria sufficiently unspecific and generic (hence, ungeFAIR) to allow a person without disciplinary knowledge to assess reusability according to a subset of criteria. In this case, FAIRness is understood as reusability for humans, which directly access individual datasets. To include criteria for the FAIRness of datasets for humans in the Charité dashboard, it is necessary to define a smaller subset of FAIR criteria which does not require disciplinary knowledge to be assessed, and for these to develop and validate assessment workflows. These will be ultimately implemented as browser-based extraction forms using Numbat, used to assess datasets shared by Charité researchers (see also Open Data LOM), and shared with others for adoption and reuse.

In this process, we want to overcome the following obstacles, which have so far impeded the development of reliable, validated FAIR assessments by humans:

• FAIR criteria were originally only loosely defined, and the convergence onto definitions which can be operationalized is still ongoing; also, in the process, individual criteria can be broken up further
• As long as criteria are not of high granularity, their practical assessment requires multiple design and prioritization decisions
• For multiple FAIR criteria, only researchers from the respective field can reasonably assess compliance

Importantly, where we will have both automated and manual detection of FAIR criteria at our disposal, we will undertake to gauge the quality of automated screening tools. A mutual validation of machine-readability and human-readability is conceptually impossible (see footnote), but nevertheless, given certain borderline conditions, manual assessment can help to gauge the quality of automated assessments.

Footnote:

Validation can be conceptualized as high agreement between human raters for human-readability. However, strictly speaking, data from a human check cannot serve to validate machine-readability. Nevertheless, the human readout can for the moment be used to assess, if not formally validate, the FAIRness of data for machines at least for the specific screening tool applied.

This holds true given two conditions:

• human extraction of information on FAIRness is more complete than by machines, and
• datasets are typically selected for reuse by humans directly and not through algorithms. These two are currently fulfilled, but this might change in the future.