Pause, Persist, Pivot: Key Decisions Health Professions Education Researchers Must Make About Conducting Studies During Extreme Events.

When extreme events occur, some research becomes a clear priority, but what becomes of all other research? Does it stop indefinitely, or can it be paused with plans to resume, persist with modifications, or pivot to address new priorities? Facing this dilemma and witnessing it among their fellow health professions education researchers, the authors recognized a need for guidance. This Invited Commentary presents a framework, organized as key questions related to the research stage and process, to assist health professions education researchers in making decisions about how to proceed with research that was planned or in progress when an extreme event occurred. Although at the time of this writing, the COVID-19 pandemic was the extreme event at hand, the authors intentionally created questions and discussed considerations that can be helpful for thinking through decisions in a variety of disruptions in health professions education research-many of which require similar difficult decisions and creative solutions to carry important research forward and maintain high quality.

During this pandemic, many faculty members have found that their mental energy is redirected toward efforts to adapt and adjust their clinical, educational, and research work to a rapidly shifting reality of uncertain duration.[1,2] Clinicians are delivering care in new ways. Educators are teaching in virtual worlds. Researchers have stopped studies or are creating innovative work-arounds. The literature is replete with suggestions for these various audiences.[3,4] For researchers, national and local institutions focus much of their guidance on policies, principles, and technical details that pertain primarily to clinical and lab-based research.[5-7] Meanwhile, health professions education (HPE) researchers need guidance about whether and how to proceed with education studies they have either begun or were about to begin. Should such research simply stop or at least wait for the “new normal”? We believe there are viable options for education research to continue.

Extreme Events and the Researcher’s Dilemma

Extreme events such as earthquakes, hurricanes, political conflicts, and pandemics are “occurrences that, relative to some class of related occurrences, are either notable, rare, unique, profound, or otherwise significant in terms of their impacts, effects, or outcomes.”[8](p407) While disruptions to researchers’ plans occur frequently and often can be addressed through contingency plans for typical occurrences,[9] extreme events are so unanticipated and thoroughly disruptive that they necessitate review and reconsideration of the research project.

When these events arise, the researcher’s dilemma is “what to do now?” Some researchers will abandon or reduce their studies to data already collected, perhaps to pursue new studies. Others will continue their studies.[10-15] For those who continue, options include pausing the research for a period of time with criteria for when to resume, persisting with the research through the event—often with modifications to the study procedures, or pivoting the focus of their research. These choices are not mutually exclusive, as some research may continue with the original question (pause or persist) and add new questions or modify existing questions (pivot). In this Invited Commentary, we offer a framework, presented as guiding questions, to assist HPE researchers in deciding how to proceed with their research. The questions address the stage of research and the main components of the research process (study purpose, design or approach, procedures, and findings). These are summarized in Chart 1. Chart 2 demonstrates using the framework with an example. We acknowledge that extreme events will spawn their own set of novel education studies, and researchers will follow traditional guidance for quality research.

Chart 1

Questions, Options, and Considerations for Health Professions Education Researchers Conducting Studies During Extreme Events

Chart 2

Example of an Exploratory, Qualitative Study Initiated Before the COVID-19 Pandemic and Discussions of Considerations Leading to a Decision on How to Proceed With This Study During the Pandemic and Discussions of Considerations Leading to a Decision on How to Proceed With This Study During the PandemicIn fall 2019, a research team launched a study of physicians’ willingness to share vulnerability with trainees. The team approached the study from a constructivist orientation, beginning analysis of data as soon as data collection began. The team completed interviews with 30 physicians from multiple specialties and career stages by January 2020 and had completed the first phase of data analysis in mid-March when concerns about the COVID-19 pandemic escalated. The team had just begun preparing for the second phase of data collection in which they planned to interview trainees about their perceptions and observations of faculty members. Initially, the team shifted attention to clinical and personal responsibilities and concerns related to the pandemic, but eventually they returned to regular research meetings (virtually) and discussed how to proceed with the study. The discussion posed concerns aligned with research components as detailed below.

Key Guiding Questions

What is the study’s stage?

If a study is still in the design stage, researchers can adjust their plans depending on how intertwined the topic is with the extreme event and whether they can study the topic in other ways or at another time. For example, if the event occurs in a limited region, can the research be conducted elsewhere? Is it worthwhile to adjust the design to compare affected and unaffected regions or populations? If the study is already underway, researchers will need to decide how to manage the event’s impact on the study. Some researchers may decide to pivot, particularly if new funding sources support research in their topic areas with a focus on how the extreme event affects education. Funding may also provide access to new data sources related to the extreme event and encourage collaboration across institutions, disciplines, and fields. Which stage the study is in, along with considerations of the research process, will help with pause, persist, or pivot decisions.

How will the extreme event affect the purpose of the study?

Studies serve a variety of purposes, including description of a phenomenon (e.g., prevalence of burnout in a population), exploration for deeper understanding of a phenomenon (e.g., how do learners perceive and experience inequity in clinical learning environments), testing hypotheses or interventions (e.g., is team-based learning as effective for teaching diversity, equity, inclusion topics as it is for biochemistry topics), and translating theory or knowledge from research to educational practices (e.g., how best to apply retrieval practice in clinical education).[16] Extreme events may affect these purposes in different ways. Consider, for example, a descriptive research study examining how students use online modules to study for exams. When the study started, modules were optional or supplemental to in-person lectures. Now, as a result of the pandemic, online modules are the primary option for learning course material. Does this new situation still yield a valid description of the phenomenon? Probably not. Is it still useful to know what patterns looked like before the pandemic? Perhaps—particularly if the purpose of the study could be revised to compare use pre–post or to document trends over time.

Considerations for the pause, persist, or pivot decision relate to the usefulness and credibility of a study. First, is the purpose of the study still relevant and useful? In many cases, the answer is yes, and the study should persist on this basis but may require modifications. Then, the decision will depend on whether these modifications are possible (e.g., ethically, feasibly). Questions about students’ use of online modules are still relevant, but the new data collection may expand to better represent the impact of the pandemic or capitalize on the opportunity to ask about different, yet relevant, outcomes. For example, researchers might incorporate into their study faculty development needs for successful teaching online. Second, can the purpose of the study and research questions still be answered with adequate validity, credibility, or trustworthiness if the study pauses or persists? If not, should it pivot? The answer depends on how the extreme event will affect the phenomenon of interest or intervention, both in the short and long terms. Descriptive research might pivot to focus on trends or comparisons (as in the study of online learning). Exploratory research might persist and remain attuned to shifts or patterns emerging over time as data collection and analysis continue. Research testing a hypothesis or intervention may not be particularly affected or the modified intervention can serve as an additional intervention arm, particularly if conducted in a controlled environment.

How is the study design or approach affected?

Study design relies on the researcher’s philosophical orientation.[17] Qualitative research conducted from a constructivist orientation needs to reflect contextual changes. Correspondingly, this research may incorporate changing circumstances into the original research question. Quantitative descriptive studies aim to represent a phenomenon or state of affairs in a designated population and time frame, often from a positivist or postpositivist stance that strives for statistically generalizable claims. How events threaten the external validity of the study outcomes will be important to determine.

How should study procedures change?

Extreme events can affect many aspects of study procedures, including recruitment, intervention, data collection, and data analysis. There are numerous procedural considerations, so we mention a few general points here and acknowledge that many require exploration in the context of a particular study.

Recruitment.

Recruitment may be difficult or impossible during or after the extreme event. Certain groups may be disproportionately affected by the event and harder to access than others. Researchers may use statistical techniques to address this issue or recruit in an unaffected area. Recruitment strategies could change. For example, trying to recruit residents may be challenging using email, but announcements at pandemic-generated, virtual social events may work effectively.

Interventions.

Interventions may need to be paused either until delivery of the original intervention can be resumed or, if that is not possible, until the intervention has been reconceptualized. A key deciding factor is how much the intent, fidelity, or integrity of the intervention would change if altered. A complete redesign may be necessary as educational systems change or content is deemed more or less important during or after an extreme event.

Data collection.

Data collection methods could be scrutinized. New questions or scales might be added to surveys as the study design either pivots or additional information is required for context. Interviews may include new questions to collect information about participants’ current circumstances and experiences of the extreme event or to reflect on changes in their thinking, perspectives, and beliefs. Researchers should examine data collection procedures such as format (e.g., shifting from in-person to video interviews or focus groups or from paper to electronic surveys), consent procedures (e.g., if signature cannot be obtained in person, if additional risks or benefits need to be identified), and researcher training (e.g., to collect data in a new way, to ensure safety, to recognize signs of distress). Any changes made partway through data collection require careful review for changes in the quality and integrity of the data. If initial data sources are limited or no longer available, researchers could consider entirely different data sources and techniques that require flexible or no recruitment, such as the use of text mining methods to study social media content to answer research questions.

Data analysis.

Data analysis may require checks for differences in responses, ratings, patterns, or themes before and after the event. Additional focus groups or interviews could check resonance and nonresonance of findings. The analysis in qualitative studies should be attentive to interpretations related to the extreme event. Researchers might turn to realist framing, which invokes the primacy of context.[18] Studies involving data collection before and after the event can use techniques such as interrupted time series and the unexpected event during survey design to determine the significance of the event on the outcome.[19] Missing data analysis techniques, such as intent to treat or multiple imputations, should be considered when the data collection has been disrupted or impeded by the event.[20] Changes or modifications in the analytic techniques may also require additional collaborators or team members to appropriately address these new challenges.

How should findings address the extreme event?

When researchers report their findings, they need to consider how design and process decisions may affect their findings. We suggest paying careful attention in 2 areas.

Representativeness/validity/trustworthiness.

The researchers will need to ponder whether the data they report on still represent the phenomenon of interest, the population intended, and the perspectives of that population. The researchers may want to describe their participants in phases and clarify any changes in representation in the sample based on the extreme events. Discrepancies may need to be reported as findings or limitations.

Framing/interpretation.

The experience and effects of the extreme event may bring new insights to include in the findings. Researchers should consider how to clarify the additional interpretations that come to light in the aftermath of the extreme event. For example, education studies that examine learning during extreme events such as war and forced migration will need to be interpreted carefully, underscoring the context, politics, and policies that circumscribe the findings from the study.

How should the impact of the extreme event on the study be acknowledged in the manuscript?

The researchers’ decisions have implications for the writing of the manuscript. The researchers will have to decide where to best position the extreme event—introduction, method, discussion, or all 3? If the study pivoted, the extreme event may be best positioned up front, as part of the framing in the introduction. The method section likely will include details explaining the decision to pause, persist, or pivot and associated changes. In qualitative studies, researcher reflexivity will require careful attention to how members of the research team experienced the extreme event and how their experiences influence their interpretations.

To enrich the discussion, researchers might think through the relevance and implications of their findings in light of the extreme event. Some questions include the following: Can we still learn something from the findings, especially if all data were collected before the extreme event? Are data useful for comparison or benchmarking in future studies? Are the research questions still meaningful and important despite changes that have occurred and will likely continue to occur after the event (e.g., if a technology studied before the event becomes obsolete or a policy changes)? The limitations section may elaborate on context and explain modifications, particularly for studies that paused or persisted.

Final Reflections

Extreme events are likely to disrupt all aspects of education[1,21] and, correspondingly, education research. Researchers may question the relevance of topics and studies chosen before an extreme event. Such reservations stem from concern about pressing social issues, guilt for using limited resources or burdening participants, and observed shifts in funding and publication priorities.[22] For example, several funding agencies, journals, and academic outlets have shifted attention to pandemic issues (the “covidisation” of research)[23] and may distract researchers from focus on important educational issues that are fundamental to education even in a time of crisis.

While some of the challenges faced in education research are similar to those of clinical and other social and behavioral science research, the lack of funding for and small-scale and local focus of most HPE research can bring these studies to a halt, especially during times of scarcity and heightened competition for limited resources. We provided questions as reminders of issues that we encourage those who continue their work to consider when determining how to proceed and how to do so in a scholarly manner. These questions should provide a way to think through an educational research study during any unexpected event. They should provide ways forward when such movement initially feels overwhelming or impossible.