Using Meta-Research to Improve Science: A Participant-Guided “Learn by Doing” Course (in Englisch)

Concerned about the reproducibility crisis in scientific research?

Wondering what you can do to improve scientific rigor & reproducibility?

Looking for a course that offers hands on experience?

Want to learn the skills needed to identify practices that contribute to poor reproducibility in your own field, and develop solutions?

Apply for the QUEST Center’s meta-research course, where students will learn about meta-research by working as a team to design, conduct and publish a meta-research study.

What is meta-research?

Meta-research is research on research. Meta-research, or science of science studies, can help us to improve science by identifying common problems with the design, conduct, analysis and reporting of scientific studies. This allows us to develop targeted solutions to improve scientific research. Meta-research may also examine other topics that influence research practices, including hiring and promotion practices, educational programs, journal guidelines and funding agency policies.

Course Overview

In this participant guided learn by doing course, students will work as a team to complete a meta-research study. All participants will be listed as authors on a meta-research paper, which will be submitted to a peer-reviewed journal for publication at the conclusion of the course. After completing the course, students will be able to apply the skills that they have learned to address problems in their own fields.

Course language: English

Eligibility

This course is funded by the Berlin University Alliance and is open to students in the biomedical or biological sciences at Charité - Universitätsmedizin Berlin, Humboldt-Universität zu Berlin, Freie Universität Berlin or Technische Universität Berlin. The course is open to PhD students, medical students and MD/PhD students who actively participate in research. Masters students who have significant research experience may also apply.

Instructor

Tracey L. Weissgerber, Ph.D. Meta-researcher QUEST – Quality | Ethics | Open Science | Translation BIH Center for Transforming Biomedical Research Charité - Universitätsmedizin Berlin Berlin Institute of Health (BIH) Email: tracey.weissgerber@charite.de

Dr. Weissgerber is a meta-researcher at QUEST (Quality | Ethics | Open Science | Translation) in Berlin. She completed her graduate work in physiology at Queen’s University (Canada), a post-doctoral fellowship at the Magee-Women’s Research Institute (USA), and was an Assistant Professor at Mayo Clinic (USA). Her 2015 paper on bar graphs has been viewed more than 350,000 times. This paper contributed to policy changes in many journals that encourage authors to replace bar graphs of continuous data with more informative graphics (dot plots, box plots, violin plots).

Student Assistant: TBD

ECTS & Time commitment

Interested in receiving ECTS for this course? We will provide a packet (signed certificate/information on the course etc.) to help students obtain ECTS credit from their institution or department (estimated ECTS: 5-6 credits). As this course is open for students from various programs, obtaining formal approval of the program is not feasible and students themselves are responsible for ensuring that they will receive credit for the course.

This elective course will run from late June to the end of November.
Students will participate in a 90-minute class video conference each week, which will combine a seminar to teach new material with active discussions about how to proceed with each phase of the meta-research project. Students will interact virtually through Microsoft Teams throughout the course. It is estimated that students will spend, on average, one day per week throughout the course working on course activities.  

Learning Objectives

After completing the course, students will be able to:

  1. Define meta-research, and explain how scientists can use meta-research to improve science
  2. Critique study designs for systematic review/literature survey style meta-research studies by evaluating scientific rigor and feasibility
  3. Write screening, abstraction and analysis protocols for a meta-research study
  4. Implement procedures to train the study team in data screening and analysis
  5. Conduct a systematic review/literature survey style meta-research study
  6. Create a public repository containing the results of the meta-research study, including protocols, data and meta-data
  7. Use the results of the meta-research study to develop targeted solutions to the problems identified
  8. Prepare a communication strategy to explain the results of the meta-research study to active researchers
  9. Write a manuscript detailing the results of the meta-research study

Course Outline

 

Week

Topic

1

June 29 to July 5

What is meta-research? How can we use meta-research to change science?

Preparation: Select and read at least three articles from the science of science reading list.

2

July 6

Designing a meta-research study

Preparation: Review detailed protocols for two meta-research studies

2 & 3

July 6-19

Virtual brainstorming: We will hold one Virtual Brainstorming Day to discuss each of the ideas proposed by the instructors, and an additional Virtual Brainstorming Day to discuss proposals from the class and select “wild card” ideas in weeks 2 and 3 of the course. Class members will divide into groups to draft protocols for each proposed project.

You can find information on Virtual Brainstorming Days here: https://osf.io/c5gyz/. You’ll be asked to check in on Teams 2-3 times over the course of each brainstorming day to share your thoughts and respond to others’ comments.

Assessing feasibility: Students will complete feasibility assessments for each proposed project.

4

July 20-26

Selecting a project

Preparation: Rank the project ideas in order, and provide reasons for your ranking. Consider the following criteria:

1.     Quality of the study design

2.     Feasibility (time & resources needed to complete the project)

3.     Is the project a good fit for the expertise of the team?

4.     Potential to change scientific practice (Are solutions to the problem available? How might the project lead to new solutions?)

5.     Personal interest

5

July 27-Aug 2

Creating screening, abstraction & analysis protocols

Class members will divide into three groups to work on the screening, data abstraction and data analysis protocols. Each group will develop and test their protocol, revise their protocol, and create a training set of 25 articles.

Deliverable: Screening and abstraction protocols

6

Aug 3-9

Screening & selection

Preparation: Log onto Rayyan and complete screening for the training articles

7

Aug 10-16

Why you need a dissemination strategy, and how to design one that works

Deliverable: During weeks 6 & 7, the team will complete screening, the study flow chart, and the final study protocol

8

Aug 17-23

Abstractor training 1: Testing & revising a data abstraction protocol

9

Aug 24-30

Creating a repository, Abstractor training 2

Deliverable: Final protocol deposited on OSF, all abstractors completed training, data abstraction plan & timeline

10-18

Aug 31-Oct 25

Data abstraction, resolving discrepancies

Weekly call to discuss progress, resolve issues and discrepancies, deal with challenges that arise during data abstraction

19

Oct 26 – Nov 1

Data analysis

20

Nov 2-8

Communication strategies – introduction, methods, visualizations

Deliverable: Draft introduction, methods, educational visualizations

21

Nov 9-15

Data figures, writing results

Deliverables:

·       Revised drafts of introduction, methods, educational visualizations

·       First draft of data figures, results

22

Nov 16-22

Discussion, Revising the manuscript

Deliverables: Final draft of manuscript

Leadership – Career Development

Students will work in teams to lead different sections of the research project. The leadership team for each phase of the project will be responsible for designing the strategy for completing that phase, and working with all class participants to complete the required tasks.

Phases include the following:

  1. Organizing Virtual Brainstorming Days for each project
  2. Project selection
  3. Protocol development
  4. Screening
  5. Data abstraction
  6. Data analysis
  7. Depositing data and protocols 8. Manuscript preparation

Grading Scheme

Course marks will be assigned based on the following criteria:

  1. Protocol: 10% (same mark assigned to all course participants)
  2. Manuscript: 30% (same mark assigned to all course participants)
  3. Repository: 10% (same mark assigned to all course participants)
  4. Performance on tasks led by the student: 15%
  5. Contribution to tasks led by others: 35%