Quality AssuranceClosed
BRAVO
Benchmarking Academic Institutions' Contributions to Increasing Value and Reducing Waste in Biomedical Research
Under the heading of "Increasing Value, Reducing Waste in Biomedical Research", the concept of quality in biomedical research is currently being critically debated worldwide. The main lines of criticism relate to the validity and reproducibility of experimental studies and the tendency to publish only desirable study results (publication bias, distorted publication).
As a cross-cutting topics, BRAVO addresses both 1) quality assurance and 2) the acceptance and cultural embedding of quality criteria in biomedical research at German medical schools.
The aim of the project is to promote and improve internal university standards and incentive systems for quality assurance.
BRAVOS subprojects
Research institutions such as University Medical Centers (UMCs) can foster responsible research practices by providing education, infrastructure, and incentives. However, research assessment typically relies on traditional metrics that do not reflect research quality, such as number of publications, journal impact factors, and third-party grant funding. Research institutions are often unaware of their performance on responsible research practices. Baseline and comparative assessments of these practices would highlight where there is room for improvement and where to allocate resources, and thereby empower UMCs to support improvement. We performed a status quo analysis of established clinical trial registration and reporting practices at German UMCs and developed a dashboard to communicate these baseline assessments with UMC leadership and the wider research community. While the uptake of certain practices has increased over time (e.g. prospective registration and Open Access), there is much room for improvement. The dashboard was shaped by expert interviews with relevant stakeholders and contextualises UMC performance to established requirements and guidelines. Beyond helping institutions assess how they perform in relation to mandates or their institutional policy, the dashboard may inform interventions to increase the uptake of responsible research practices as well as evaluate the impact of these interventions.
Publication
Franzen, D. L., Carlisle, B. G., Salholz-Hillel, M., Riedel, N., & Strech, D. (2023). Institutional dashboards on clinical trial transparency for University Medical Centers: A case study. PLOS Medicine, 20(3), e1004175. https://doi.org/10.1371/journal.pmed.1004175
Contact
Prof. Dr. Dr. Daniel Strech, Head of research group
Delwen Franzen, PhD, Research fellow
Maia Salholz-Hillel, MSc, Research fellow, doctoral candidate
We interviewed 28 different stakeholders (60% German) to get their views on a proof-of-principle dashboard. Stakeholders included UMC leadership, support staff (e.g., policy makers or librarians), funders and experts in responsible research. The proof-of-principle dashboard consisted of different quantitative metrics that showed the uptake of a responsible research practice (e.g., registration and reporting of clinical trials, or Open Access and open data on a UMC-level). Overall, stakeholders thought the dashboard was helpful and appreciated the fact that the metrics were based on concrete behaviours. Main weaknesses included the lack of an overall narrative underlying this way of assessing research institutions. Stakeholders considered the dashboard a good opportunity to initiate change and hoped that the dashboard could be extended with other metrics in the future.They feared that making the dashboard public might risk incorrect interpretation of the metrics and put UMCs in a bad light.
Contact
Prof. Dr. Dr. Daniel Strech, Head of research group
Martin Holst, research fellow
This cross-sectional study aimed to describe whether and how relevant policies of university medical centres in Germany support the robust and transparent conduct of research, and how prevalent traditional metrics are. We searched institutional policies for academic degrees and academic appointments as well as websites of core facilities and research in general, of all 38 German university medical centres. We screened these documents for mentions of indicators of robust and transparent research (study registration; reporting of results; sharing of research data, code, and protocols; Open Access, and measures to increase robustness) and for mentions of more traditional metrics of career progression (number of publications, number and value of awarded grants, impact factors, and authorship order). We found that references to robust and transparent research practices are, with a few exceptions, generally uncommon in institutional policies at German university medical centres, while traditional criteria for academic promotion and tenure still prevail.
Contact
Prof. Dr. Dr. Daniel Strech, Head of research group
Martin Holst, research fellow
Inadequate links between trial publications and registrations make each less findable and compromise evidence synthesis and health policy decision making. The World Health Organization (WHO) and others call for better adoption of this straightforward transparency practice. In this cross-sectional bibliographic study, we aimed to evaluate links between registration and publication across clinical trials led by German university medical centers (UMCs) and registered in either ClinicalTrials.gov or the German Clinical Trials Registry (DRKS).
We used automated strategies to download and extract data from trial registries, PubMed, and trial publications for a cohort of all registered, published clinical trials conducted across German UMCs and completed between 2009 and 2017. We implemented regular expressions to detect and classify publication identifiers (DOI and PMID) in registrations, and trial registrations numbers (TRNs) in publication metadata, abstract, and full-text.
We found that only 20% (373/1,895) of trials have a linked publication (DOI or PMID) in the registration as well as the TRN in the publication full-text, abstract, and metadata, and only 25% (477) of trials met the CONSORT and ICMJE guidelines to include TRNs in both the full-text and the abstract. 17% (327) of trials had no links. The most common link was TRN reported in the full-text (60%, 1,137). ClinicalTrials.gov trials were overall better linked than DRKS trials, and this difference appears to be driven by PubMed and registry infrastructure, rather than by trialists. Of trials reporting a TRN in the abstract, trials in ClinicalTrials.gov were more likely than trials in DRKS to have the TRN captured in the PubMed metadata. Most (78%, 662/849) ClinicalTrials.gov registrations with a publication link were automatically indexed from PubMed metadata, which is not possible in DRKS.
In conclusion, German UMCs have not comprehensively linked trial registrations and publications by both including a structured reference to the publication in the registration, and reporting TRNs in results publications. In addition to improved linking by trialists, changes in the PubMed TRN capturing process (such as automated strategies like those developed in this study) and automated indexing of publications in DRKS would make trial registrations and results more findable.
Publication
Maia Salholz-Hillel, Daniel Strech, Benjamin Gregory Carlisle, Results publications are inadequately linked to trial registrations: An automated pipeline and evaluation of German university medical centers. SAGE Journals, Published April 1, 2022, doi.org/10.1177/17407745221087456
Contact
Prof. Dr. Dr. Daniel Strech, Head of research group
Maia Salholz-Hillel, MSc, Research fellow, doctoral candidate
Open Access (OA) broadens the dissemination of research discoveries and allows others to build on existing knowledge. While there is evidence that Open Access is growing, many publications remain behind a paywall. In many cases, journal or publisher self-archiving policies allow researchers to make a version of their publication openly accessible in a repository, sometimes after an embargo period (green OA). In this study, we investigated the realised potential of green OA in a sample of ~2000 publications from interventional clinical trials published in 2010 – 2020. Included trials were registered in ClinicalTrials.gov or DRKS, conducted at a German UMC, and reported as complete in 2009 – 2017. We queried Unpaywall (OurResearch) via its API to assess the OA status of publications in this sample. We then queried Shareyourpaper (OA.Works) via its API to obtain article-level self-archiving permissions for paywalled publications in this sample. We identified 881 paywalled publications, 86% (n=760) of which had a current "best permission" for self-archiving the accepted or published version in an institutional repository (query date: 03/12/2021). The analysis was limited to publications resolving in Unpaywall and focused on the "best permission" in Shareyourpaper for archiving in an institutional repository. These findings are informing part of an ongoing intervention with trialists at the Charité to increase clinical trial transparency. Our case study provides a concrete example of how available tools may be used to obtain actionable information related to self-archiving at scale, and thereby empower efforts to increase science discoverability.
Publication
Franzen, D. L. (2023). Leveraging Open Tools to Realize the Potential of Self-Archiving: A Cohort Study in Clinical Trials. Publications, 11(1), Article 1. https://doi.org/10.3390/publications11010004
Contact
Prof. Dr. Dr. Daniel Strech, Head of research group
Delwen Franzen, PhD, Research fellow
