Development of the Verbal Autopsy Instrument for COVID-19 (VAIC).

BACKGROUND: Improving accuracy of identification of COVID-19-related deaths is essential to public health surveillance and research. The verbal autopsy, an established strategy involving an interview with a decedent's caregiver or witness using a semi-structured questionnaire, may improve accurate counting of COVID-19-related deaths.
OBJECTIVE: To develop and pilot-test the Verbal Autopsy Instrument for COVID-19 (VAIC) and a death adjudication protocol using it. METHODS/KEY
RESULTS: We used a multi-step process to design the VAIC and a protocol for its use. We developed a preliminary version of a verbal autopsy instrument specifically for COVID. We then pilot-tested this instrument by interviewing respondents about the deaths of 15 adults aged ≥65 during the initial COVID-19 surge in New York City. We modified it after the first 5 interviews. We then reviewed the VAIC and clinical information for the 15 deaths and developed a death adjudication process/algorithm to determine whether the underlying cause of death was definitely (40% of these pilot cases), probably (33%), possibly (13%), or unlikely/definitely not (13%) COVID-19-related. We noted differences between the adjudicated cause of death and a death certificate.
CONCLUSIONS: The VAIC and a death adjudication protocol using it may improve accuracy in identifying COVID-19-related deaths.

INTRODUCTION

Accurately ascertaining disease-specific deaths is a critical element of understanding the pathophysiology, natural history, prognosis, treatment, and epidemiology of any illness.[1] This is particularly important during a pandemic of a novel infectious disease, when this information can impact public health as well as treatment strategies. Identifying coronavirus disease 2019 (COVID-19)–related deaths has posed several challenges, however. Early in the pandemic, testing was not widely available, especially in the most affected communities.[2] The pandemic also blunted health care–seeking behaviors, with many sick people not going to a physician or ED/hospital due to fear of being exposed to the virus or concerns about an over-burdened health care system. Thus, many decedents who likely died from COVID-19 never had a medical evaluation before death. The protean clinical manifestations of the disease can also make attribution difficult if decedents did not have “classic” presentations of COVID-19 including dyspnea, cough, and fever. Finally, in many communities, the majority of deaths occurred in nursing homes or assisted living facilities, where multiple comorbidities, the virtual absence of testing at the beginning of the pandemic, and regulatory and reporting issues all conspired to make the accurate identification of COVID-19 deaths even more difficult.[3]

Taken together, all of these issues create misclassification biases that work in the same direction: undercounting of COVID-19 deaths. As a result, current reports on COVID-19-related deaths under-estimate the actual numbers.[1] This has been confirmed by emerging research examining excess deaths during the COVID-19 pandemic.[4] Death certificates, a data source for surveillance and research which already had issues with inaccuracy before this pandemic, likely dramatically under-report COVID-19-related deaths.[5]

The implications are profound; virtually all clinical and population health studies will be susceptible to misclassification due to inability to accurately determine whose death is a result of COVID-19. This will significantly undermine the ability of public health professionals and researchers to accurately learn from COVID-19-related deaths to improve understanding of this infectious illness in ways that may impact interventions and prevention efforts. Strategies to mitigate this problem are urgently needed.

The verbal autopsy is an approach with significant potential to improve accurate counting of COVID-19-related deaths.[6,7] This method employs interviews with decedents’ caregivers or witnesses using semi-structured questionnaires to elicit pertinent information on signs, symptoms, and circumstances leading to their death.[6,7] Used by the World Health Organization (WHO), epidemiologists, and researchers for more than 50 years, this increasingly standardized technique has been a critical tool in disease surveillance, measuring the impact of public health interventions, and outbreak investigation.[6-9] It is particularly useful in developing countries without robust vital statistics programs.[6] To our knowledge, a disease-specific verbal autopsy has not been previously proposed as a tool to examine whether a death is COVID-related. We believe that it may be a powerful method to more accurately count COVID deaths. Interviews with family caregivers or other proxies in the community and with certified nursing assistants in long-term care have the potential to provide likely highly reliable information to determine which deaths are caused by COVID-19.

We report on the development and pilot testing of a Verbal Autopsy Instrument for COVID-19 (VAIC), which can serve as a critical component of a death adjudication strategy for public health officials, researchers, and clinicians during and after this pandemic.

METHODS

We used a multi-step process to design and optimize the VAIC, a verbal autopsy instrument to assess for COVID-19-related deaths, and a protocol for its use in research and surveillance. This process, shown in Fig. 1, included developing a verbal autopsy instrument, pilot testing and revising the instrument, and creating a death adjudication process using it.

Fig. 1

Multi-step process to design and optimize the Verbal Autopsy Instrument for COVID-19 (VAIC).

Developing the Verbal Autopsy Instrument for COVID-19

We developed a structured questionnaire modeled on existing verbal autopsy instruments and verbal autopsy standards.[6,7] Models included the WHO’s Instrument for Cause of Death During Ebola Outbreaks,[10] which has questions about exposure, signs, and symptoms of an infectious disease and the REasons for Geographic and Racial Differences in Stroke (REGARDS), a large ongoing cohort study, which includes interviews with next of kin or witnesses on learning of the death of a cohort member, using questions about signs, symptoms, and circumstances surrounding the demise modeled on national consensus recommendations.[11-13] An important component of the verbal autopsy instrument is an open-ended opportunity for the respondent to describe, in their own words “about illness that led to death?” We incorporated information specific to COVID-19-related illness based on existing knowledge of the virus’ clinical characteristics. Members of our research team (PG, MRS, MMS) have made important contributions to this knowledge.[14] Our team has broad extensive clinical experience caring for many patients with COVID-19 in New York City, one of the first US epicenters of the disease, in a variety of clinical settings including outpatient clinics, the Emergency Department, hospital wards, and the Intensive Care Unit.

An initial draft of the instrument was designed by three of the authors (TR, MMS, MSL), including one (MMS) with experience designing the verbal autopsy tool for the REGARDS cohort study.[11,12] The instrument was reviewed and revised by the entire team. We also met as a team to develop a standardized process for conducting the interview using the tool, including instructions and a script for the interviewer.

Pilot Testing and Further Revising the Instrument

We pilot-tested the verbal autopsy instrument using deaths of adults aged ≥65 who received primary care from the Weill Cornell Center on Aging, an urban, academic Geriatrics clinic in New York City at which several of the authors (MP, CA, MB, TD, VML, BR, MSL) provide care. The clinic was already closely tracking established patients who died for any reason during the initial New York City COVID-19 surge (4/7/20–5/19/20). We planned to use 15 cases to pilot-test and revise this instrument. These cases were selected from among the 39 patients receiving care from this clinic who had died during this time period. Cases were selected for inclusion based on the recommendation of the decedent’s primary care provider, with the key criterion that an appropriate respondent to contact to participate in the interview could be identified. In this convenience sample, we were unable to contact one potential respondent, but all other potential respondents approached consented to participate. Interviews were conducted by authors who were also clinicians at the WCM Center on Aging (MP, CA, MB, TD, VML, BLR, MSL). The Weill Cornell Institutional Review Board reviewed our project and determined it was exempt.

We conducted the 15 interviews in three groups of five. After the completion of each group of interviews, we met as a team to discuss issues with the instrument and instructions/script. We made changes after the completion of the first group of 5, which improved the flow of the interview and eliminated respondent confusion. We confirmed the impact of these changes based on experience in subsequent interviews, and we believed additional refinements were not needed after the second or third groups of 5. The final version of the VAIC Instrument and Instructions are included as Supplemental Figs. 1 and 2.

All interviews were conducted via telephone. The median number of days between the death and the interview was 33 (range 8–58 days, IQR 19–45 days). During each interview, the interviewer recorded responses on the VAIC interview form and documented the time that the interview took to complete (range 14–45 min, median 20 min, IQR 16–30 min). The completed VAIC form was scanned and added to the deceased patient’s clinic electronic medical record.

COVID Death Adjudication Using the Instrument

A team of six of the authors (TR, MMS, MRS, PG, MSL) carefully reviewed the VAIC and other available clinical information including outpatient and inpatient medical records with laboratory results, imaging reports, and chart notes for each of the 15 deceased patients. We assessed the utility and value of the instrument and developed a death adjudication protocol that might be used in the future. Members of this protocol development team represented several disciplines: general internal medicine, primary care, geriatrics, and emergency medicine. This team included one author (MMS) with extensive experience leading and overseeing the death adjudication process for the REGARDS cohort study.[11,12] For each death adjudication, adjudicators independently reviewed the VAIC as well as any relevant outpatient or inpatient medical chart information available within the Weill Cornell Medicine/NewYork-Presbyterian Hospital system, including provider documentation and test results near the end of life. The goal of adjudication was to determine whether the underlying cause of death was definitely, probably, possibly, or unlikely/definitely not COVID-19-related. This is a standard approach to categorization of likelihood of a specific underlying cause of death to quantify the degree of certainty within the adjudication process to facilitate analysis.[11,12] We utilized established principles used in previous studies, including the Women’s Health Initiative and the REGARDS study, to conceptualize the underlying cause of death as the disease or injury that, in the days to weeks before death, interrupted the steady-state and initiated events resulting in death.[11,12,15]

RESULTS

The 15 decedents whose deaths we used to pilot-test the verbal autopsy instrument ranged in age from 72 to 96 years (median 89 years, inter-quartile range (IQR) 83–91 years), and 8 (53%) decedents were male. Most decedents (14, 93%) were non-Hispanic white with 1 (7%) Hispanic. Eleven (73%) were living in the community at the time of death, 3 (20%) in a skilled nursing facility, and 1 (7%) in assisted living. Six (40%) died in the hospital, 3 (20%) at home but not in hospice, 2 (13%) in home hospice, 2 (13%) in a skilled nursing facility, 1 (7%) in inpatient hospice, and 1 (7%) at assisted living. Among the 11 decedents who lived at home, 9 (82%) received regular home health care services (including aides, physical therapy, and hospice care). Respondents included adult children (8, 53%), spouses/partners (3, 20%), other family (2, 13%), and friends/associates (2, 13%).

We used a multi-step process to effectively calibrate our adjudication team and develop a reproducible algorithm. Calibration is critical to ensure that each adjudicator is consistent and approaches the process in the same manner. We initially reviewed 5 of the cases (including one from each of the three groups of interviews conducted chronologically and described above) and discussed them as a group to develop rules/principles. We then reviewed another set of 5 cases and made adjudication decisions independently. We discussed these adjudication judgments as a team, reached a consensus, and expanded our principles/algorithm. We then reviewed and made determinations on the remaining 5 and further expanding our rules/principles into an algorithm after reaching consensus. We noted that the response to the open-ended question typically provided the most useful information about the underlying cause of death. The adjudication results for each case, including the location of the death, past medical history, a brief summary of circumstances surrounding the patient’s death (symptoms, evaluation, testing), possible exposures, and our adjudication decision are shown in Table 1.

Table 1

Cases Reviewed and COVID-19-Related Death Adjudication Determinations

Case #.	Details of case/circumstances surrounding death	COVD-19-related death adjudication determination
1	Long-term nursing home resident—with advanced dementia but without medical comorbidities    • Had fever, cough    • Chest x-ray with interstitial changes consistent with viral pneumonia    • Primary care physician diagnosed patient with presumed COVID-19    • Several residents on floor had recently confirmed or presumed COVID-19    • Never received a COVID-19 test    • Died in nursing home a few days after onset of symptoms	Probably
2	Lived at home with family—with dementia, aortic valve replacement on anti-coagulation, paroxysmal atrial fibrillation    • Few weeks of cough, which worsened with fever, shortness of breath (SOB)    • Presented to hospital hypoxic    • Chest x-ray with multifocal pneumonia    • Positive COVID-19 test during hospitalization    • Died during short hospitalization on recently launched hospice unit	Definitely
3	Lived at home with hospice care—with hepatocellular carcinoma and recently discovered with suspicious pancreatic mass    • Had non-productive cough for few weeks, developed fever    • Presented to hospital for these symptoms, not hypoxic    • Chest x-ray clear    • Positive COVID-19 test during hospitalization    • Improved clinically transiently during hospitalization but developed urinary incontinence, urosepsis and died during hospitalization on recently launched hospice unit	Probably
4	Lived at home with home health aides and wife—with Parkinson’s disease, atrial fibrillation, congestive heart failure    • Developed SOB/tachypnea, fever    • Found to be hypoxic on evaluation in home    • Chest x-ray showed vascular congestion    • Never received a COVID-19 test    • Home health aide had a positive COVID-19 test, symptoms before decedent’s illness and wife had positive antibody test soon after his death    • Died at home	Probably
5	Lived at home on hospice—with coronary artery disease, diabetes mellitus type 2, chronic kidney disease, congestive heart failure    • No fever or cough    • SOB at very end of life thought due to fluid overload    • Receiving antibiotic for urinary tract infection before death    • Never received a COVID-19 test    • Died at home on hospice	Unlikely/no
6	Lived at home with home health aide—with prostate and colon cancer, severe aortic stenosis, pacemaker, atrial fibrillation, congestive heart failure    • Had 4 days of cough, SOB, and fell, triggering presentation to hospital    • Presented to hospital severely hypoxic    • Chest x-ray with bi-lateral lung opacities    • Positive COVID-19 test during hospitalization    • Experienced worsened congestive heart failure and kidney failure during hospitalization    • Died during hospitalization on recently launched hospice unit	Definitely
7	Lived at home with home health aide—with dementia, coronary artery disease atrial fibrillation on anti-coagulation, aortic stenosis    • Fever, worsening confusion so brought to hospital    • Chest x-ray with potential early pneumonia    • Positive COVID-19 test during hospitalization    • Delirious throughout hospitalization    • Died during hospitalization on recently launched hospice unit	Definitely
8	Lived at home with wife and home health aide—with dementia, diabetes mellitus    • Had congestion in chest, fever, sleep disturbances    • Chest x-ray showed mild congestion, no overt edema or focal pneumonia    • Exposures included home health aide, wife, physical therapist who came to home, all of whom tested positive    • Became lethargic and hypotensive, brought to hospital    • Positive COVID-19 test during hospitalization    • Worsened pneumonia, pleural effusions    • Died during hospitalization	Definitely
9	Lived at home with family—with atrial fibrillation, congestive heart failure, obstructive sleep apnea, chronic obstructive pulmonary disease    • Multiple falls and fever    • Outpatient chest x-ray showed possible sternal fracture and viral pneumonia    • Primary care physician diagnosed patient with presumed COVID-19    • Severe hypoxia on home monitoring    • Home hospice initiated because aggressive interventions not desired    • Never received a COVID-19 test    • Died in nursing home a few days after onset of symptoms	Probably
10	Longtime nursing home resident—with advanced dementia coronary artery disease, atrial fibrillation    • Low-grade fever for 5 days in nursing home    • Negative COVID-19 test in nursing home    • Brought to hospital with severe hypoxic respiratory failure, SOB    • Positive COVID-19 test during hospitalization    • Died during brief hospitalization	Definitely
11	Lived at home with home health aide—with coronary artery disease, chronic kidney disease, hypothyroidism    • Presented to hospital from home in mixed cardiogenic and septic shock, made comfort care while hospitalized    • Negative COVID-19 test during hospitalization    • Died in hospital	Unlikely/no
12	Lived at home with home health aide—with advanced dementia, diabetes mellitus type 2    • Referred to inpatient hospice due to failure to thrive and pneumonia, which primary care provider thought was possibly COVID-19    • Never received a COVID-19 test    • With respiratory illness with SOB at inpatient hospice, where died	Possibly
13	Lived at home with home health aide—with lung cancer, tongue cancer, uterine cancer, chronic obstructive pulmonary dieses    • Developed SOB which daughter attributed to aspiration    • Not evaluated in person by a treating clinician during final illness    • Never received a COVID-19 test    • Died at home	Possibly
14	Assisted living facility resident—with advanced dementia, diabetes mellitus type 2    • Developed SOB, cough    • Was hypoxic, with acute respiratory distress syndrome    • Several residents in facility had recently had confirmed or presumed COVID-19    • Never received a COVID-19 test    • Died in nursing home a few days after onset of symptoms	Probably
15	Long-term nursing home resident—with dementia, breast mass    • Received positive COVID-19 test during surveillance in nursing home when was asymptomatic    • Due to positive test, was relocated to dedicated wing within the facility for COVID-19 residents    • Several days later, developed SOB and cough    • Died in nursing a few days after onset of symptoms	Definitely

Overall, we determined that 40% of deaths were definitely COVID-19-related, 33% probably, 13% possibly, and 13% unlikely/definitely not (Table 1). Notably, for case #6, determined to be “definitely” COVID-19-related by the adjudication team, the death certificate listed “heart failure” as the underlying cause of death. The algorithm for death adjudication our team developed to categorize deaths as definitely, probably, possibly, or unlikely/definitely not COVID-19-related is shown in Fig. 2. The US National Center for Health Statistics (NCHS) released guidelines on how to appropriately certify deaths related to COVID-19. This guidance recommends considering deaths in two groups: (1) deaths from direct complications of laboratory-confirmed COVID-19 and (2) compelling clinical history of COVID-19 but not tested or with a negative test.[16] These groups are analogous to our definitely and probably categories.

Fig. 2

Algorithm for COVID-related death adjudication determinations.

DISCUSSION

Accurately counting and carefully studying deaths related to COVID-19 is a critical component of appreciating the prevalence and impact of this pandemic. The VAIC verbal autopsy instrument and adjudication process we have rigorously developed offer a useful tool to assist in this process. Similar to other disease-specific verbal autopsy instruments, we found it feasible to use with a variety of respondents and to complete within a reasonable timeframe. We were able to create an algorithm to adjudicate potential COVID-19-related deaths that we successfully applied to categorize cases using the verbal autopsy and clinical information. We focused particularly on developing rules and principles for the probable and possible categories, using clinical syndrome, impressions of treating clinicians, and exposures.

We think that a verbal autopsy approach using this tool improves on relying exclusively on death certificates to ascertain COVID-19-related deaths. Though death certificates are important, inaccuracies have been a long-standing issue[5,17-19] that has reduced their utility in surveillance and research even before COVID-19. In the USA, death certification is typically completed by treating clinicians, most of whom have not been trained on this task. NCHS uses sophisticated algorithms to attempt to overcome these inaccuracies entering the revised underlying cause of death in the National Death Index (NDI).[20] Death adjudication by researchers using all available information, however, frequently disagrees with the NDI.[12] Death certification inaccuracies are likely an even larger issue with the outbreak of a novel disease.[5] While NCHS released guidelines on how to appropriately certify COVID-19-related deaths,[16] the release occurred in June 2020, after the pandemic had caused many deaths. Also, most clinicians are likely not aware of the guidelines, and, as a result, practices surrounding COVID-19-related death certificate completion vary widely. Even in the small sample of cases we examined, COVID-19 was not mentioned in the death certificate of a decedent that we adjudicated to have had a probably COVID-19-related death.

Our protocol may be modified as appropriate and easily incorporated into existing community-based surveillance. It may be useful for public health officials investigating deaths in local communities with surges or clusters. Accountable care organizations and individual medical clinics may use it to explore deaths of patients in their care. It may be helpful to smaller institutions unaccustomed to disease tracking including skilled nursing facilities, assisted living facilities, and group homes. This tool may be particularly useful in the resource-poor countries where healthcare system limitations have made verbal autopsy strategies a particularly important part of mortality tracking.

This approach may also be useful for research by integrating it into existing follow-ups in large cohort studies to explore COVID-19-related deaths among subjects. It may also have utility in assessment of adverse events/deaths in clinical trials, particularly when a patient receives care at an institution external to the clinical trial home institution. Similar to other death adjudication protocols for research, we would recommend that, in practice, two independent adjudicators assess each case, with disagreements resolved by consensus. Additionally, we anticipate that larger-scale analyses using the approach we propose will combine definitely and probably COVID-19-related into a single category for analyses and would conduct a sensitivity analysis adding possibly to assess the impact of doing so, as has been done in other studies.[13,21]

The tool and strategy we present here have important limitations. A verbal autopsy strategy relies on identifying a respondent who is willing to be interviewed and is a reliable reporter, which may be challenging soon after a death, particularly during a pandemic. Large cohort studies commonly offer to conduct the interview at a later time to be respectful of the family’s grieving. Recall/reporting bias is always a potential issue with verbal autopsies and may have impacted our study. Though current recommendations suggest it is reasonable to collect data for verbal autopsies up to 1 year after death, research suggests that the probability of a correct diagnosis is likely to decline month by month during this period.[22] We attempted to minimize the impact of this bias by also including medical records for decedents, and we did not encounter cases where the information we received from informant interviews differed substantively from the medical records. This instrument and protocol have been tested on a small number of cases. It is likely that the rules/principles for death adjudication have not included all possible scenarios and will need further expansion. The likely numerous scenarios not accounted for by rules and algorithms are offset by the common practice of resolving disagreements by committee, which partly overcomes this challenge. These cases were selected from a larger group using convenience sampling, so selection bias is possible, with the potential contribution of COVID-19 to death in unselected cases potentially harder to characterize than in the cases we examined. Also, the VAIC was developed in English and in a single urban geriatric clinic in the eastern USA. Changes may be required for it to be useful in a different population, and a rigorous process is needed if it is translated into a different language. Another limitation is that our study included primarily non-Hispanic white older adults. Recent research suggests that patients of different races and ethnicities may have different presenting and outcomes,[23-25] though these differences may be due to social structures and racism rather than biology and physiology. Further research is needed to examine its performance among various races/ethnicities. We are currently planning to conduct a larger study to validate and use the VAIC in a larger, more diverse population. Furthermore, our understanding of the pathophysiology, presenting symptoms, and natural course of COVID-19, a new disease, continues to evolve, informed by clinical experience and research.[26-28] For example, recent literature identified additional symptoms suggestive of COVID-19,[26-28] with the diagnostic specificity of anosmia and dysgeusia increasingly recognized.[29] Also, the disease itself may change, with newer variants arising and changes to presentation and illness course as new treatments and prevention/management strategies become available. As a result, we plan to review and revise this tool and strategy periodically to make appropriate modifications. Nevertheless, given the urgency of accurate mortality data during this pandemic, we believe that releasing this tool now for use is essential; we are hopeful that it will improve counting and understanding of COVID-19-related deaths.

CONCLUSIONS

Improving the accuracy of identification of COVID-19-related deaths is essential to public health surveillance and research during and after this pandemic. Limitations in death certification processes and quality as well as shortages in testing availability, under-resourced public health/surveillance structures, overwhelmed healthcare systems, and changes in health care–seeking behavior of patients have made accurate counting of cases and deaths related to this novel disease challenging. The verbal autopsy instrument and death adjudication protocol we describe here may serve as a critical element of a strategy to accurately ascertain COVID-19 mortality. This is vital to understand the impact of COVID-19 on different populations and for the development of intervention and prevention strategies.

(PDF 42.3 kb)

(DOCX 23.4 kb)