On Schrödinger's Cat and Evaluation of Trials Disrupted by the Covid19 Pandemic: A Critical Appraisal.

From the beginning of 2020, the world has been fighting the SARS-Cov-2 outbreak. The life of each one of us has profoundly changed. Unavoidably, our clinical routine has drastically modified in its priorities and methodologies (1). The COVID-19 pandemic has also raised significant issues in the field of research. The investigators’ responsibility has increased with the need to thoughtfully weigh the risk-benefit ratio for each protocol in an emergency and evolving scenario (2).

Many projects that had started in the pre-COVID-19 era were halted in the past months, mainly because of 1) the restrictions that governments put in place to limit the spread of the Coronavirus, 2) the inadequacy/inappropriateness of the research activities as planned before the pandemic, and/or 3) the participants’ reluctancy at attending the scheduled visits in facilities usually located within the hospital premises. Some investigators managed to redesign their study methods and make them compatible with the new situation. Someone also proposed possible solutions to tackle the most common issues of the “new” research routine (3). Others faced challenging (if not insurmountable) stalls to solve and eventually decided to stop their studies. Indeed, it was not rare the closure of research activities in some countries, especially during the early period of the pandemic.

How should we judge the findings generated by studies that suffered the effects of the COVID-19 pandemic? In principle, the modification of a protocol “on the way” or the premature ending of a study are deviations that, in “normal” times, would be flagged as major methodological flaws. Purists of research methodology may indeed find these deviations hard to accept. But, can we judge today’s research activities without considering the new context of COVID-19 and the global consequences we all have been experiencing? What degree of flexibility should we apply when reviewing an article containing issues clearly due to the pandemic? In this context, it is also essential to consider the increasing interest around pragmatic research for achieving a wise compromise between the potentially decontextualized findings from rigid methodologies and the heterogenous “real life” scenarios where the study results will be applied (4). Perhaps, the pandemic may lead us to introduce sufficient doses of pragmatism and creativity as core elements of future research. For sure, scientific projects will have to include backup plans considering the possibility of postponing/modifying recruitment strategies, maintaining protocol adherence despite environmental changes, and designing alternative strategies for intervention delivery.

In this issue of The Journal of Frailty & Aging, Tavoian and colleagues (5) present the results of a pilot study initiated before the COVID-19 outbreak. The trial was focused on exploring whether stationary bicycle high-intensity interval training was a more efficient standalone exercise strategy to improve cardiovascular and muscular function compared with resistance or aerobic training in older adults. Unfortunately, the restrictions imposed by the emergent pandemic situation led to a premature stop of the program, precluding the achievement of the expected sample size and leaving the researchers with partial results.

The data could appear insufficient to convey a complete and clear message (at least coherent with the original study aims). For example, the unbalanced representation of women in the study sample (11 vs. only 3 men) might already render the findings of difficult interpretation. On the other hand, we might say that something is always better than nothing! Those partial results could still feed the discussion in the scientific community and serve to support further steps in the field (6). The Chinese philosopher Lao Tzu once said: “The journey of a thousand miles begins with one step”.

The study by Tavoian and colleagues was well-presented and conducted rigorously, methodologically speaking. It also had its record on clinicaltrials.gov, having been registered before the initiation of recruitment. The topic was of interest and novel. Last but not least, the trial was framed as a pilot study with the primary aim of collecting preliminary data for future larger-scale activities. Under these premises, it could have been unfair to reject it simply because an external event (i.e., the COVID-19 pandemic) had disrupted the researchers’ original plans. Are we sure that what is (partially) presented by Tavoian and colleagues may not be sufficient to address questions raised by other researchers? After all, when a study is well-conducted, it is always important to publish it, independently of issues that might have affected its conduct. Of course, the expected impact of an unforeseen event disrupting a pilot study is different from that occurring in a phase 3 trial. This implies that a case-by-case approach is crucial but needs to be structured on specific criteria. Otherwise, the risk could be to fall into a dangerous level of subjectivity or to reject everything in search of a utopic perfection.

No study is perfect, and every research has an own value. It is a matter of clearly and fairly presenting the experience, constructively discussing the strengths and weaknesses. At the same time, the readers should be transparently informed of the deviations imposed by the COVID-19 outbreak and provided with all the information necessary to judge the value of findings and discern potentially relevant results from “background noise”. The need for clarity and details is important for discriminating vague justifications (e.g., “due to the pandemic”) from founded decisions based on political, sanitary, and/or scientific choices (2).

It is also noteworthy that the numbers we report in our tables are persons who have trusted in our research and devoted part of their time to it. The restitution of the study results has to be considered an ethical duty of the researcher in front of the participant. Consistently, many investigations benefit from funds allocated by private or public institutions for specific research purposes and potentially taken from other priorities. It is, thus, also a matter of financial clarity and accountability behavior.

With the aim of providing the correct legitimacy to those valuable research activities critically affected by the COVID-19 pandemic, it has been felt that a set of standardizing criteria could be helpful. Table 1 presents aspects that might support clinicians, researchers and policymakers in better judging the quality of the information provided by partial scientific results.

Table 1

Specific criteria to additionally consider when judging if an article is still worth being published despite COVID-19-related deviations from original plans

Criterion	How to potentially verify
A solid and innovative rationale for the study	• The introduction section should present the scientific rationale and provide sufficient evidence to support the study
Certified design of the study	• When applicable and as appropriate, the study protocol is registered (e.g., a trial has a clinicatriasl.gov ID, a systematic review has its record in PROSPERO) •The study was originally of adequate statistical power to answer the research question(s) • The main methodological and logistical complexities encountered in the study and the adopted countermeasures are discussed. This information could be helpful for researchers preparing to conduct similar studies
The methodological deviations due to COVID-19 are presented and justified	• The Methods and Discussion sections should clearly describe the difference(s) between the original plans and the final output of the study •The deviation from the protocol due to the COVID-19 pandemic is adequately justified (e.g., remote assessments were conducted to limit attrition) • An interim statistical analysis for futility or efficacy (also adopting less stringent alpha and beta values) is performed • The Discussion section should clearly and transparently explain how the deviations from the original protocol might have affected the study findings • The study findings are presented and discussed concisely (e.g., in the form of brief communications/short reports rather than as full research papers) and cautiously (i.e., as hypothesis-generating rather than as confirmatory evidence)
“Point of no return”	• No possibility of temporary suspension of the protocol (i.e., modification of the study timeline) • No possibility of adapting the planned activities to the emergent COVID-19 scenario (i.e., adaptation of the methodology)

The post-COVID-19 world we live in is putting us in the position of reconsidering many aspects of our (professional) lives. Different priorities are today modifying our decisions. How we conduct research today is not the same as before COVID-19. The population we are studying is not the same either, and its values and needs might be different (7). Some adaptations of research to the new scenario (e.g., more extensive use of technologies, higher involvement of participants in collecting their data) are necessary to continue advancing in research and avoiding that what was initiated before COVID-19 could go lost.