Verbal autopsy can consistently measure AIDS mortality: a validation study in Tanzania and Zimbabwe.

BACKGROUND: Verbal autopsy is currently the only option for obtaining cause of death information in most populations with a widespread HIV/AIDS epidemic.
METHODS: With the use of a data-driven algorithm, a set of criteria for classifying AIDS mortality was trained. Data from two longitudinal community studies in Tanzania and Zimbabwe were used, both of which have collected information on the HIV status of the population over a prolonged period and maintained a demographic surveillance system that collects information on cause of death through verbal autopsy. The algorithm was then tested in different times (two phases of the Zimbabwe study) and different places (Tanzania and Zimbabwe).
RESULTS: The trained algorithm, including nine signs and symptoms, performed consistently based on sensitivity and specificity on verbal autopsy data for deaths in 15-44-year-olds from Zimbabwe phase I (sensitivity 79%; specificity 79%), phase II (sensitivity 83%; specificity 75%) and Tanzania (sensitivity 75%; specificity 74%) studies. The sensitivity dropped markedly for classifying deaths in 45-59-year-olds.
CONCLUSIONS: Verbal autopsy can consistently measure AIDS mortality with a set of nine criteria. Surveillance should focus on deaths that occur in the 15-44-year age group for which the method performs reliably. Addition of a handful of questions related to opportunistic infections would enable other widely used verbal autopsy tools to apply this validated method in areas for which HIV testing and hospital records are unavailable or incomplete.

Verbal autopsy, in which care givers to the deceased report information on signs, symptoms and circumstances preceding death,1–3 is currently the best option for obtaining cause of death information in populations without comprehensive civil registration systems. WHO, through the Health Metrics Network, is co-ordinating an effort to develop a common tool for verbal autopsy, since there is an urgent need to validate currently used questions for different populations, settings and causes of mortality.4

There is particular need to develop methods to measure AIDS mortality, which is the leading cause of death among young adults in virtually all countries with generalised HIV epidemics. Improving measurement of AIDS mortality is crucial since the ultimate goal of interventions—including the scale up of antiretroviral treatment—is to reduce AIDS mortality.5 Monitoring the success of such programmes therefore relies on accurate measurement of AIDS deaths in the community.6 7 Hospital records and vital registration of deaths are especially inadequate for estimating the level of AIDS as a cause of mortality because of bias, underreporting and stigma associated with the disease.8

Two longitudinal community studies in Tanzania and Zimbabwe provide a unique opportunity to investigate the most sensitive and specific set of questions to ascertain HIV/AIDS-associated mortality. Both studies have collected information on the HIV status of the population over a prolonged period and maintained a demographic surveillance system (DSS) that collects information on cause of death through verbal autopsy. At the time of the verbal autopsy studies, HIV prevalence was approximately 20% in the Zimbabwe cohort and 7% in the Tanzania cohort.9 10 With data from the Zimbabwe study, we developed a computer algorithm to classify AIDS deaths from verbal autopsy data11 12 validated using serological data from deaths that occurred from 1998 to 2003. Before the algorithm can be widely used, it must be demonstrated that it performs similarly and predictably in different settings of high HIV-associated mortality (validation in place) and can perform well at different phases of the epidemic when levels of AIDS mortality differ (validation in time).

Methods

All participants of both cohorts were followed as part of demographic surveillance and were tested for HIV at each sero-survey. If an individual died between follow-ups, an attempt was made to perform verbal autopsy. An individual's HIV status at death was assumed to be the same as at his/her most recent test, which was a maximum of 3 years prior.

The Manicaland Project includes a population-based open cohort study in the rural province of Manicaland in eastern Zimbabwe.9 The study population were residents in two forested small towns, four tea and coffee estates and six rural areas (including four subsistence farming and two roadside trading centres). A baseline survey took place from 1998 to 2000, with two follow-ups occurring 3 and 5 years later (the intersurvey periods are referred to here as phase I and phase II). Of the households, 8376 and 7102 identified in the survey areas at phase I and phase II, respectively, were enumerated. Male and female participation rates in the individual cohort study survey were 78% (4320/5561) and 80% (5134/6419) at phase I and 77% (3047/3958) and 80% (3972/4936) at phase II, respectively. Verbal autopsies were conducted for 94% of all deaths,9 with 53% of the verbal autopsy reports obtained from close relatives (spouse 19%, child 7%, parent 16% and sibling 10%). At each follow-up of the study, testing for the presence of HIV antibody was performed.13

The Kisesa HIV cohort is located in Magu District, Mwanza Region, in northwestern Tanzania. The cohort was established in 1994 (when baseline studies were conducted) and data collection is based on a biannual DSS that had conducted 14 phases by 2002 and sero-surveys repeated approximately every 3 years, with three testing surveys completed before 2002.10 14–17 The population in the DSS area grew from 19 354 in 1994 to 24 403 by 2002. Participation in the DSS is more than 98%, with proxy reporting accepted for absent household members. The average participation rate in sero-surveys was 72% in the first three surveys. Deaths identified in the DSS are followed up with a verbal autopsy interview between 6 weeks and 6 months later, if a reliable informant can be identified who cared for the deceased during the final illness. Verbal autopsy interviews were completed for 67% (420/629) of the adult male deaths and 64% (424/667) of the adult female deaths recorded in the DSS between 1994 and 2002; 94% of the verbal autopsy reports were obtained from close relatives (spouse 30%, child 28%, parent 21% and sibling 15%).

The verbal autopsy tool

The study teams identified deaths through the use of checklists of all individuals interviewed at the previous phase and discussions with village health workers, employers and surviving household members present at follow-up. Data were collected on the signs, symptoms and circumstances preceding death using a structured, closed, interviewer-led questionnaire. The verbal autopsy questionnaire was originally developed in Kisesa and contained specific questions related to symptoms of late-stage HIV and opportunistic infections14 and was used in that site from 1994 to 2002. A nearly identical questionnaire was adapted in Manicaland in both phase I and phase II. The questionnaires were administered in local languages (Shona, the predominant local language in Zimbabwe, and Swahili, in Tanzania). Interviewers were clinical officers/certified nurses who received special training on how to administer the verbal autopsy questionnaire. Verbal autopsy informants were parents, spouse, other relatives or a neighbour (in rare circumstances when close relatives were not available). The interviews were conducted after the recognised mourning period, in a respectful and unhurried manner. In Kisesa, the interviewer gave a small, culturally appropriate gift (a bar of soap) to the person with whom the interview was conducted, as a token of appreciation of the time devoted to answering the lengthy questionnaire. Ethical approval for the DSS and all related procedures (such as the verbal autopsy interviews) was granted by the Tanzanian Medical Research Co-ordinating Committee for the Kisesa study and by the Zimbabwe Medical Research Council for the Manicaland Study.

Validation procedures

We developed a computer algorithm that creates a set of criteria for classifying AIDS deaths based on verbal autopsy data.12 Seventy-five per cent of deaths from Manicaland phase I were randomly assigned to a training dataset. From this training dataset, all signs/symptoms with a likelihood ratio >1.92 in univariate analyses were considered as potential identifiers of AIDS death (as defined below). Signs/symptoms were added to a list of criteria one at a time, based on the highest specificity. Verbal autopsy deaths with that sign/symptom were then removed from the dataset, and specificities of the remaining signs/symptoms were recalculated. Sensitivity was plotted against 1–specificity in a receiver operator characteristic (ROC) plot. (The ROC is a tool used to select an optimum cut-off for a diagnostics test based on the trade-off of sensitivity and specificity.) These steps were repeated until the (equally weighted) sensitivity and specificity of the list of symptoms was maximised (ie, the point closest to the top left hand corner of the ROC plot). Deaths were classified as HIV/AIDS associated if the deceased had one or more of the criteria on the list. The sensitivity and specificity of the list was then tested on the Manicaland phase I test data (the remaining 25% of deaths), all phase II and all Kisesa data. AIDS death in the gold standard was defined as an individual who was (a) HIV positive at baseline survey based on antibody testing; (b) was not reported to have suffered major injury from motor vehicle accident, self-inflicted (suicide), or accidentally (accident) or deliberately inflicted by another person (homicide) in the 2 weeks before death; and (c) did not die from direct obstetric causes (death during labour).12 Preliminary analyses highlighted that HIV prevalence among the deceased was markedly lower among the relatively older adults, so analyses were stratified at age 45 years.

Results

There were a total of 376 and 219 deaths in phase I (1998–2003) and phase II (2003–2005), respectively, of the Manicaland study and 197 in Kisesa (1994–2002), among 19–59-year-olds for whom there was a verbal autopsy and a conclusive HIV test done within 3 years of death. A minority of deaths occurred in the 45–59-year age group (13%, 17% and 19%) in Manicaland phase I, phase II and Kisesa cohorts, respectively (table 1). In Manicaland, approximately 75% of deaths were caused by AIDS, compared with 51% in Kisesa 15–44-year-olds and 33% among Kisesa 45–59-year-olds. Herpes zoster, acute respiratory tract infections, abscesses and sores, acute diarrhoea and tuberculosis were all less commonly reported in Kisesa than in Manicaland (table 2). For deaths under the age of 45 years, weight loss, jaundice, tumours, respiratory tract infections and tuberculosis were less common among HIV-positive deaths in Kisesa compared with Manicaland.

Table 1

Prevalence of AIDS mortality in Kisesa and Manicaland verbal autopsy subjects

	Manicaland					Kisesa
	Phase I train	Phase I test	Phase I test	Phase II test	Phase II test	Test	Test
Age range (years)	15–44	15–44	45–59	15–44	45–59	15–44	45–59
Total with HIV test and VA (n)	237	88	51	181	38	158	39
AIDS deaths in gold standard (n (%))	173 (73%)	64 (73%)	40 (78%)	137 (76%)	28 (73%)	81 (51%)	13 (33%)

VA, verbal autopsy.

Table 2

Sensitivity and specificity of individual signs and symptoms for AIDS deaths in Manicaland and Kisesa cohorts, stratified at age 45 years

Symptom	Train		Test
	Manicaland phase I		Manicaland phase II		Manicaland phase II				Kisesa
	15–44 years (n=237)		15–59 years (n=102)		15–44 years (n=180)		45–59 years (n=38)		15–44 years (n=158)		45–59 years (n=39)
	Se	Sp	Se	Sp	Se	Sp	Se	Sp	Se	Sp	Se	Sp
Weight loss	9	100	10	99	12	98	0	91	10	96	23	84
Herpes zoster	12	93	18	95	24	98	19	100	14	97	8	100
Jaundice	19	93	4	97	4	100	0	100	5	99	8	96
Vaginal tumours	6	94	6	94	3	100	0	100	0	100	0	96
Wasting	19	93	19	95	15	95	13	100	16	95	8	100
ARTI	11	95	10	95	8	95	13	100	5	96	0	88
Abscesses or sores	24	90	24	92	29	98	32	100	19	96	8	100
Oral candidiasis	38	86	37	89	45	86	26	91	40	92	15	100
Diarrhoea	17	95	17	92	29	93	23	100	10	92	8	88
Recent TB*	27	86	28	88	47	82	32	100	5	99	8	96

ARTI, acute respiratory tract infections; Se, sensitivity; Sp, specificity; TB, tuberculosis.

Recent tuberculosis is shown for information but was not included in the final algorithm due poor specificity.

In applying the previously developed algorithm to Kisesa data, the sensitivity in classifying AIDS deaths was low (67%), mainly because of poor sensitivity (46%) in the 45–59-year age group. Based on this observation, the algorithm was re-trained on Manicaland phase I data, restricted to 15–44-year-olds. This resulted in a different ordering of symptoms compared with the original algorithm and in the inclusion of a ninth symptom since the addition of diarrhoea gave a slightly higher mean sensitivity and specificity (nine symptoms: 75.1% compared with eight symptoms: 74.7%). Using this new algorithm, a set of nine criteria with a sensitivity of 75.0% and specificity of 75.2% was produced (figure 1).

Figure 1

Receiver operator characteristic (ROC) curve of algorithm on deaths of 15–44-year-olds from Manicaland phase I.

The newly trained algorithm performed consistently based on sensitivity and specificity of data for 15–44-year-olds from Manicaland phase I (sensitivity 79%; sensitivity 79%), Manicaland phase II (sensitivity 83%; specificity 75%) and Kisesa (sensitivity 75%; specificity 74%) tests. Although a reasonable specificity was maintained on 45–59-year-olds, the sensitivity dropped markedly in Manicaland phase I (sensitivity 73%; specificity 73%), Manicaland phase II (sensitivity 68%; specificity 80%) and Kisesa (sensitivity 54%; specificity 62%) tests in this older age group (figure 2).

Figure 2

Sensitivity and specificity of the trained algorithm. The dashed line represents the performance of the algorithm on the training dataset. Panels A to D use the full set of nine signs and symptoms. Panels E and F use a smaller set of five signs/symptoms available from the current INDEPTH tool.

The INDEPTH network is an association of health and DSSs in African, Asian and Oceania countries (http://www.indepth-network.org/). The network has developed a widely used verbal autopsy questionnaire, although the tool does not collect information on herpes zoster, abscesses or sores, vaginal tumours or oral candidiasis (table 3).18 We measured the value of using the five signs/symptoms that are available from the INDEPTH questionnaire. In the 15–44-year-old age group, using only these five signs/symptoms resulted in a sensitivity and specificity of 64% and 82%, respectively (figure 2). The reduction in sensitivity was less than expected, using the same five criteria in phase II Manicaland data and earlier Kisesa data sensitivity had decreased to 50% and 44% respectively.

Table 3

Signs and symptoms for surveillance of AIDS mortality, and the availability in INDEPTH—another widely used verbal autopsy questionnaire

Sign/symptom	Definition based on verbal autopsy question	Equivalent in INDEPTH VA questionnaire
Weight loss	Moderate or severe weight loss with no other symptoms of malnutrition	Yes
Herpes zoster	Ever suffered from zoster	Not specifically mentioned (shingles, zoster, herpes) but questions on rash, including where the rash is located (not if it is one-sided) and if the rash has blisters. No mention of pain during or after rash
Jaundice	Acute jaundice (yellowing of the whites of the eyes during the disease that lead to death) with fever and/or itching but without history of alcohol abuse	Yes
Vaginal tumours	Vaginal tumour for at least 1 month with or without bleeding	No
Wasting	Moderate or severe weight loss with at least four of the following symptoms: paleness, changing hair colour, oedema of legs, burning sensations of the feet, dry scaly skin	Two out of the five symptoms (but different phrasing): paleness oedema of ankles. No mention of changing hair colour, burning feet dry scaly skin (might be identified as rash)
Acute respiratory tract illness	Trouble breathing, cough lasting 3–27 days with fever but not recent TB, weight loss or wasting, as above	Partial cough, with duration fever shortness of breathing noisy breathing. TB—does not specify when
Abscesses or sores	Had abscesses or sores	No sores not mentioned, “other swellings or ulcers”
Oral candidiasis	Had two or three of the following: ulcers in the mouth, difficulty swallowing, white patches inside the mouth and tongue	No difficulty swallowing
Diarrhoeal disease	Loose stools lasting 3–99 days, with or without dehydration	Yes

TB, tuberculosis; VA, verbal autopsy.

Discussion

We developed a tool that consistently measures AIDS mortality using verbal autopsy. Through slight modification to our previously proposed criteria, the algorithm performs similarly in Zimbabwe and Tanzania—settings with different HIV prevalence (approximately 20% and 7%, respectively9 10) and AIDS mortality and different distribution of other causes of death. The algorithm is robust in that it performs consistently when prevalence is above approximately 5%.

This method of measuring AIDS mortality produced reliable estimates only in the 15–44-year age group. This is due to increasing levels of other-cause mortality in older ages. In Manicaland, where the proportion of deaths due to HIV in the older age groups remained high,11 the methods worked well. But in Kisesa, where HIV prevalence is lower, AIDS mortality begins to drop off after 35 years of age, especially in women.6 14 19 20 and other causes also increase in the 45–59-year age groups. Tuberculosis, in particular, reduces the specificity marginally and the sensitivity markedly 21 22 because tuberculosis symptoms overlap substantially with HIV symptoms resulting in misclassification.

Given the sensitivity and specificity of the method from the training data, we would predict that 88% and 53% of deaths of 15–44-year-olds in Manicaland and Kisesa were AIDS deaths, respectively. (The formulae for this calculation are described by Lopman et al.12) This compares with directly measured values of 76% and 51%. The overestimate in Manicaland is a result the algorithm actually performing better in phase II (higher specificity) than it did on the training dataset. To calculate the prevalence of AIDS death, the level of misclassification is corrected; however, the level of misclassification was actually smaller than calculated on the training data. This approach to estimation of AIDS deaths can be applied to other verbal autopsy data for which a gold standard is not available; however, the accuracy of the prediction will not be known. Therefore, further validation of the algorithm is recommended.

The INDEPTH verbal autopsy is a widely used tool that used a shorter symptom list than ours and does not collect information to identify herpes zoster, oral candidiasis, abscesses/sores or vaginal tumours.21 These symptoms were commonly reported in our studies and (excluding vaginal tumours) were prevalent in AIDS deaths with a sensitivity of approximately 20% for zoster and abscesses/sores and 40% for oral candidiasis. We found that the INDEPTH shortlist would perform less well and have higher levels of misclassification of AIDS deaths.23 The newly released WHO instrument includes signs and symptoms associated with herpes zoster, abscesses/sores and oral candidiasis.4 In general, verbal autopsy have only proved accurate enough to assign cause of death in very broad categories in adults. However, statistical or algorithmic approaches, such as the method used here, have been shown to perform adequately for determining prevalence of a specific cause, such as HIV. Unfortunately, causes of death other than HIV could not be validated, as neither physician assessment nor diagnostics data were routinely available.

Our analyses show that in areas of generalised epidemics, verbal autopsy can consistently measure AIDS mortality. Both studies had HIV prevalence levels exceeding 5%, which is commonly found in eastern and southern Africa. In settings where prevalence is less than 5%, getting AIDS mortality from verbal autopsy may be difficult and will require further validation studies. Even in higher prevalence settings, analyses should be restricted to age groups in which competing causes of mortality, especially other infectious causes, are relatively low. Especially for the Tanzanian population, the verbal autopsy method was much less accurate for deaths over the age of 45 years. Based on our analyses, we recommend surveillance of deaths in populations with severe HIV epidemics be undertaken in the 15–44-year age group. Addition of a handful of questions related to opportunistic infections would enable other widely used verbal autopsy tools (WHO and INDEPTH) to apply this validated method in areas where HIV testing and hospital records are unavailable or incomplete.

Verbal autopsy is currently the best option for obtaining cause of death information in populations without comprehensive civil registration systems. Verbal autopsy is generally used to assign deaths to broad categories and estimate cause-specific mortality.

We demonstrate that verbal autopsy can consistently measure AIDS mortality with a simple set of nine criteria. Surveillance should focus on deaths that occur in the 15–44-year age group for which the method performs reliably. Adding just a handful of questions related to opportunistic infections would enable other widely used verbal autopsy tools to apply this validated method to asses AIDS mortality in areas where HIV testing and hospital records are unavailable or incomplete.