Are cause of death data for Shanghai fit for purpose? A retrospective study of medical records.

OBJECTIVES: To assess the quality of cause of death reporting in Shanghai for both hospital and home deaths. DESIGN AND
SETTING: Medical records review (MRR) to independently establish a reference data set against which to compare original and adjusted diagnoses from a sample of three tertiary hospitals, one secondary level hospital and nine community health centres in Shanghai. PARTICIPANTS: 1757 medical records (61% males, 39% females) of deaths that occurred in these sample sites in 2017 were reviewed using established diagnostic standards.
INTERVENTIONS: None. PRIMARY OUTCOME: Original underlying cause of death (UCOD) from medical facilities. SECONDARY OUTCOME: Routine UCOD assigned from the Shanghai Civil Registration and Vital Statistics (CRVS) system and MRR UCODs from MRR.
RESULTS: The original UCODs as assigned by doctors in the study facilities were of relatively low quality, reduced to 31% of deaths assigned to garbage codes, reduced to 2.3% following data quality and follow back procedures routinely applied by the Shanghai CRVS system. The original UCOD had lower chance-corrected concordance and cause-specific mortality fraction accuracy of 0.57 (0.44, 0.70) and 0.66, respectively, compared with 0.75 (0.66, 0.85) and 0.96, respectively, after routine data checking procedures had been applied.
CONCLUSIONS: Training in correct death certification for clinical doctors, especially tertiary hospital doctors, is essential to improve UCOD quality in Shanghai. A routine quality control system should be established to actively track diagnostic performance and provide feedback to individual doctors or facilities as needed. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Assessment of diagnostic accuracy at the individual level for over 1700 deaths.

Established ex-ante diagnostic criteria used to develop reference diagnosis for assessing data quality, thus reducing subjectivity in choice of reference diagnoses.

Medical history documentation for the top 25 cause of deaths was reviewed by one trained doctor, thus increasing risk of potential diagnostic bias for reference diagnoses.

Introduction

Accurate and complete cause of death (COD) information, particularly as-reported in the civil registration and vital statistics (CRVS) system, is essential for health decision making.1 2 Despite its development, this is equally applicable for Shanghai, a city of 14 million registered permanent residents,3 which although experiencing a relatively low mortality rate and high life expectancy (80.2 years),4 faces a number of health-related challenges stemming from an ageing population, including a significant non-communicable disease burden, and unequal access to healthcare resources for different populations subgroups, depending on income and residence.5–10 Accurate COD data are critical for reliably informing policies and programmes to address these challenges and better allocate available healthcare resources, yet no formal scientific evaluation of the completeness and quality of data generated by the existing vital registration system in Shanghai has been undertaken.

There are more than 300 hospitals in Shanghai, with over 70 000 doctors reporting causes of death. In addition, 246 community health centres (CHC) also report causes of death, particularly for those who die at home. Although CHC doctors have received training in diagnosing causes of death, the correct certification of home deaths’ remains challenging in cases where there is limited or no documentation of the previous healthcare experience of the deceased.

It is likely that the diagnostic accuracy of deaths in Shanghai is high given the standard quality assurance process that is routinely applied (see figure 1). First, death certificates with the original underlying causes of deaths (UCODS), that is, the last mentioned COD in Part 1 of the death certificate, are collected from the hospitals or CHCs. Next, trained physicia—coders at the District Center for Diseases Control and Prevention (CDC) bureau apply the rules of the International Classification of Diseases, 10th Revision to code the certificates. In cases where the morbid sequence is unclear or improbable, further investigation is undertaken to collect more information on the deceased’s medical history or from family members. Physicians at the Shanghai Municipal CDC then review all UCODs and in cases where the UCOD is inconsistent with the sequence specified on the death certificate, the district CDC doctor is contacted with suggested changes. Finally, every 6 months a quality control meeting is convened by Shanghai CDC where certifying physicians from each of the district CDC offices meet to review and discuss those death certificates with inconsistent UCODs. These certificates, following review as described above, comprise what is known as the ‘Routine UCOD’ in the Shanghai CRVS system (see online supplemental appendix S1 for a detailed description of the quality assurance process).

Figure 1

Routine CRVS procedure in Shanghai. CDC, Center for Diseases Control and Prevention; CHC, community health centres; CRVS, Civil Registration and Vital Statistics; UCOD, underlying cause of death.

In this study, we compare both the original UCOD and the routine UCOD to a reference UCOD derived from an independent medical records review (MRR) to assess the overall quality of reported CODs in Shanghai. Specifically, our objectives were to assess:

The quality of COD reporting in Shanghai for both hospital and home deaths.

Differences in the UCOD pattern in Shanghai suggested by the MRR reference diagnosis.

To our knowledge, this is the first ever empirical evaluation of the quality of COD data in Shanghai. We expect that the findings will be useful for those responsible for health policy formulation and evaluation since they provide a scientific basis for deciding how much confidence can be attributed to local mortality data that underlie health policy and programme decisions.

Methods

Data sources

Cases for the MRR were selected to be broadly representative of the distribution of deaths across facilities, socioeconomic level and location of the facility. Thirteen health facilities were selected, including three tertiary hospitals, one secondary level hospital and nine CHCs, chosen to be representative of these types of health facilities in Shanghai.

Data inclusion

To ensure that the reference diagnoses for the MRR were as accurate and comparable as possible, we applied the ‘gold standard’ (GS) criteria for the classification of COD developed by the Population Health Metrics Research Consortium (PHMRC).11 Specifying the classification criteria for causes of death ex-ante reduces the amount of subjectivity potentially introduced by the case reviewers. The degree of certainty for each of the MRR reference diagnoses was classified as follows:

GS1: highest level of certainty—MRR diagnosis was supported by either an appropriate laboratory test or X-ray/imaging with positive findings and/or medically observed and documented appropriate illness sign(s) to a predetermined standard

GS2A: high level of certainty—diagnosis supported by appropriate laboratory/X-ray with positive findings and/or medically observed and documented appropriate illness or signs to a predetermined standard

GS2B: high level of certainty—presumed initial diagnosis of a particular condition with high certainty; this category was only used for cancer and HIV patients on long-term treatment where initial data had been lost.

GS3: Reasonable level of certainty—medical diagnoses not supported by appropriate level of laboratory investigations but which meet established clinical criteria.

GS4: Unsupported—medical diagnoses not supported by adequate observed and documented clinical evidence/criteria.

The MRR was conducted using the Shanghai adaption of the Medical Data Audit Form (MDAF) (online supplemental appendix S2), originally designed for other MRR studies,12 translated into Chinese for the data collection. All study physicians were trained on how to review a medical record using the MDAF, and how to apply the standard diagnostic criteria and GS levels. Data collection was conducted by the study physicians using the modified MDAF. Four clinicians representing four major clinical streams (medicine, surgery, paediatrics and obstetrics and gynaecology) were trained to provide a further level of quality assurance of the MRR carried out by the study physicians.

Death certificates from the selected facilities where the routine UCOD was among the leading 25 causes of death for Shanghai in 2017 were reviewed and classified into different GS levels based on the information obtained from the medical records. To ensure that, the MRR UCOD was as accurate as possible, only GS1, GS2A and GS2B cases were included in the final evaluation; GS3 and GS4 deaths were discarded due to lack of adequate evidence from the medical records.

Sample size and selection

A total of 1192 GS1 and GS2 cases were collected in the four hospitals (from a total of 2378 deaths reviewed), along with 565 GS1 and GS2 cases from the CHCs (out of 751 deaths reviewed). Figure 2 presents a flow chart of the case selection: 1350 cases were discarded because their UCOD was not in the top 25 COD for Shanghai, and 22 (1.2%) of cases were discarded because they were classified as GS3 or GS4.

Figure 2

Flow chart for selection of cases. CHC, community health centres; GS1, gold standard; sMRR, medical records review; UCOD, underlying cause of death.

Data analysis

First, we reclassified the recorded UCODs to the Global Burden of Disease (GBD) cause list, comprising 290 causes, including a code for ‘garbage’ codes.13 14

Based on the GBD cause list, we then compared:

The original UCOD with the MRR UCOD.

The routine UCOD with the MRR UCOD.

To assess COD data quality, we calculated the percentage of deaths in the original and routine UCOD data sets that had been assigned a ‘garbage-code’, namely, a cause that has limited public health utility.15 16 While this analysis is a useful and easily replicable first step in assessing data quality, it does not provide any insight into the potential misclassification of specific COD, which was one of the key aims of our study. To do so requires a MRR study, where trained, independent physicians review the medical records of a deceased individual using pre-set clinical diagnostic criteria and assign a UCOD which is compared with the previously assigned UCOD. This provides a ‘GS’ against which the quality of the original and routine COD data can be measured.17–19 In this sense, our study forms part of a collective of, several MRR studies have been conducted using the PHMRC ‘GS’ diagnostic definitions as ex-ante criteria to identify true cases of a disease or injury.11 12 20–24

A misclassification matrix was developed for each comparison in order to identify the pattern and extent of certification errors. Standard diagnostic validation metrics of sensitivity, positive predictive value (PPV), Cohen’s kappa, chance-corrected concordance (CCC), cause-specific mortality fraction (CSMF) accuracy and leading CSMFs were calculated to assess the concordance of the original and routine UCODs with the MRR UCOD.11 25 All analyses were done using R software (The R Foundation for statistical computing, V.3.6.1).

For the misclassification matrices, only the 16 leading causes of death based on MRR UCODs have been included to facilitate interpretation of findings; all other diseases were merged into residual group, labelled ‘others’.

Results

Description of the study data

A total of 1757 deaths were included in the study, 61% male and 39% female (table 1). There were only two deaths at ages less than 15 years, while 70% of deaths were among people aged 70 years and above. The tertiary hospitals accounted for 43% of deaths, the secondary hospitals 25% and the CHCs 32%. Cases from the tertiary and secondary hospitals had a similar age and sex distribution, while cases from CHCs were older (see table 1).

Table 1

Deaths (number and %), by type of facility, sex and age

Facility	Sex	Age group (years)				Total (N)	%
		0–14	15–49	50–69	70+
Tertiary	Male	0.2	4.0	39.5	56.3	494	65.7
	Female	0.4	5.0	27.5	67.1	258	34.3
	Both	0.3	4.4	35.4	60.0	752	100.0
Secondary	Male	0.0	0.7	30.7	68.7	300	68.2
	Female	0.0	1.4	22.9	75.7	140	31.8
	Both	0.0	0.9	28.2	70.9	440	100.0
CHCs	Male	0.0	0.7	19.8	79.5	278	49.2
	Female	0.0	0.3	16.4	83.3	287	50.8
	Both	0.0	0.5	18.1	81.4	565	100.0
Total	Male	0.1	2.2	31.9	65.8	1072	61.0
	Female	0.1	2.3	21.9	75.6	685	39.0
	Both	0.1	2.3	28.0	69.6	1757	100.0

CHC, community health centres.

GS1 deaths comprised 62% of all deaths and GS2 38%. Tertiary hospitals had the highest percentage of cases that were GS1 (79%), followed by secondary hospitals (60%), being least in the CHCs (40%). This gradation in diagnostic capacity is as expected since tertiary hospitals would have had the most advanced and complete medical and diagnostic facilities, followed by secondary hospitals. GS1 cases were less common among the oldest age groups, compared with younger ages, as would be expected given that the elderly typically suffers from multiple comorbidities at or around the time of death, making diagnosis of the UCOD more difficult (see table 2).

Table 2

Gold standard levels by type of health facility and age (number and %)

	GS1	GS2	Total (N)	%
Facility
Tertiary hospital	79.3	20.7	752	42.8
Secondary hospital	60.0	40.0	440	25.0
CHC	40.0	60.0	565	32.2
Age group
0–49	81.0	19.0	42	2.4
50–69	74.0	26.0	492	28.0
70+	56.3	43.7	1223	69.6
Total	61.8	38.2	1757	100.0

CHC, community health centres; GS, gold standard.

Validation of the original and routine UCOD

We first compared the ranking of the leading CODs from the original UCOD, as well as the routine UCOD, to the reference UCOD determined from MRR. Interestingly, the leading COD as assessed from the original (ie, prequality checking) UCOD was the collective of ‘garbage’ codes, which were assigned to almost one-third (31%) of deaths—over one-quarter (27.7%) of these garbage codes were coded to essential hypertension, followed by pneumonia, ill-defined deaths and unknown causes, unspecified heart failure and unspecified respiratory failure (see from online supplemental table 1). Both ischaemic stroke (sixth to second) and intracerebral haemorrhage (ninth to sixth) rose in the rankings of original UCOD after MRR compared with the original UCOD, suggesting that these two conditions are being systematically underdiagnosed by healthcare facilities in Shanghai. For all other leading CODs, there were only minor changes in rankings suggested by MRR compared with the original UCOD, although there was a generalised increase in CSMFs due to reallocation of the garbage codes. Conversely, for the routine UCODs, garbage codes were only assigned to 2.3% of deaths, suggesting that the quality control process established in Shanghai for mortality data are working well. Indeed, most of the rankings for leading COD in the routine data were identical to those from the MRR UCOD, except for an increased in the ranking of falls after MRR (13–11th), indicating a high consistency for the leading CODs between the routine UCOD and MRR UCOD (see from online supplemental table 2).

We develop misclassification matrices to assess the accuracy of cause-specific diagnosis in both the original and routine dada sets by comparing individual diagnosis of the original UCOD to the MRR UCOD (see table 3). Overall, 59.9% (1053/1757) of the original death certificates were assigned the correct UCOD by doctors. For the 96 cases where the original UCOD was classified to a cause belonging to the residual or ‘other’ category, 84 (87.5%) of them were reclassified to specific leading causes after MRR (see table 3). Almost one-third (31%, 542/1757) of the original UCOD were assigned to garbage codes. Moreover, of theses, over one-third (34.5%) were categorised as having the most severe implications for policy, including such vague diagnosis as ill-defined and unknown cause of mortality (R99), unspecified heart failure (I50.9), and unspecified Respiratory failure (J96.9) (see online supplemental table 3). Falls (71.1%), ischaemic stroke (50.9%), chronic kidney disease (CKD) due to diabetes mellitus type 2 (48.0%), intracerebral haemorrhage (44.2%) and other diseases (58.3%) were frequently misassigned to garbage codes such as essential hypertension and pneumonia in the original data. Aside from garbage codes, many other diseases were also frequently misdiagnosed. CKD due to diabetes mellitus type 2 was often misdiagnosed as diabetes mellitus type 2, while many deaths due to ischaemic heart disease (IHD) were often misassigned to chronic obstructive pulmonary disease (COPD) and ischaemic stroke.

Table 3

Misclassification matrix between original UCOD and MRR UCOD

Original death certificate	MRR UCOD
	Ischaemic heart disease	Ischaemic stroke	Tracheal, bronchus, and lung cancer	Chronic obstructive pulmonary disease	Stomach cancer	Intracerebral haemorrhage	Colon and rectum cancer	Liver cancer	Pancreatic cancer	Diabetes mellitus type 2	Falls	Breast cancer	Chronic kidney disease due to diabetes mellitus type 2	Oesophageal cancer	Prostate cancer	Garbage code	Others	Total
Ischaemic heart disease	199	5	3		2	1			2	3		1					1	217
Ischaemic stroke	8	90				1				4								103
Tracheal, bronchus, and lung cancer			149	1				1										151
Chronic obstructive pulmonary disease	5	4	4	113		1				1								128
Stomach cancer				1	107		1											109
Intracerebral haemorrhage	2			1		61				1							1	66
Colon and rectum cancer		1	1	1	1		96											100
Liver cancer								62	1									63
Pancreatic cancer		1			1				74									76
Diabetes mellitus type 2	2	3	1			1				21			6				1	35
Falls											7							7
Breast cancer	1											25						26
Chronic kidney disease due to diabetes mellitus type 2										2			5					7
Oesophageal cancer				1										19				20
Prostate cancer								‘							11			11
Garbage Code	136	119	21	48	19	53	18	11	10	23	27	2	12	3	5	14	21	542
Others	16	11	1	3	2	2	2	28	8	3	4		2		2		12	96
Total	369	234	180	169	132	120	117	102	95	58	38	28	25	22	18	14	36	1757

The bold values are the CODs of the original death certificate that consistent with that of MRR

MRR, medical records review; UCOD, underlying cause of death.

Misdiagnoses were much less frequent, when comparing the routine UCOD to the MRR UCOD, as shown in table 4. In particular, garbage codes, on further investigation, were found to be primarily deaths due to IHD, ischaemic stroke, colorectum cancer, diabetes mellitus type 2 and COPD. The rigour of the data quality processes in Shanghai is clear from the fact that only 40/1757 deaths in the routine dataset were assigned to garbage codes, compared with 542, or 13.5 times as much, in the original data. (After MRR, only 14/1757 deaths were assigned garbage codes. These were cases where the reviewers could not identify more specific UCODs even after going through all available medical records.)

Table 4

Misclassification matrix between routine UCOD and MRR UCOD

Routine UCOD	MRR UCOD
	Ischaemic heart disease	Ischaemic stroke	Tracheal, bronchus, and lung cancer	Chronic obstructive pulmonary disease	Stomach cancer	Intracerebral haemorrhage	Colon and rectum cancer	Liver cancer	Pancreatic cancer	Diabetes mellitus type 2	Falls	Breast cancer	Chronic kidney disease due to diabetes mellitus type 2	Oesophageal cancer	Prostate cancer	Garbage code	Others	Total
Ischaemic heart disease	314	9		12	2	3	2		1	9	3						3	358
Ischaemic stroke	25	196	1	1		2				5	1					2	3	236
Tracheal, bronchus, and lung cancer		1	171		1			2										175
Chronic obstructive pulmonary disease	4	4	1	140	1					1					1		1	153
Stomach cancer				1	124											1		126
Intracerebral haemorrhage	2	3		2		108				1						1	1	118
Colon and rectum cancer		1	2	3	1		109			1			1					118
Liver cancer								97			1							98
Pancreatic cancer		1			1				89							1		92
Diabetes mellitus type 2	6	3		1		2				32	2		11				1	58
Falls		1		1						1	22					1		26
Breast cancer	1											28						29
Chronic kidney disease due to diabetes mellitus type 2										1			10				1	12
Oesophageal cancer				1										21				22
Prostate cancer			1							1	1				16			19
Garbage Code	6	6	2	3			5		2	4	1					7	4	40
Others	11	9	2	4	2	5	1	3	3	2	7		3	1	1	1	22	77
Total	369	234	180	169	132	120	117	102	95	58	38	28	25	22	18	14	36	1757

The bold values are the CODs of the routine CRVS system that consistent with that of MRR

MRR, medical records review; UCOD, underlying cause of death.

Overall, only 14.3% of causes in the routine dataset were reallocated on MRR. In particular, CKD due to diabetes mellitus type 2, diabetes mellitus type 2, and falls, were often misassigned to other diseases; CKD due to diabetes mellitus type 2 to diabetes mellitus type 2, and COPD; and diabetes mellitus type 2 to IHD. Overall, though, the resulting CSMFs emerging from the routine data closely approximate what the MRR suggests is the true distribution of UCOD in these facilities.

Table 5 provides the summary metrics that assess the concordance of the original UCOD as well as the routine UCOD with the MRR UCOD. CCC measures the probability that a given cause is correctly diagnosed. CSMF accuracy is the overall accuracy of the COD distribution in a population, ranging from 0 to 1, with a value of one implying perfect concordance.21 25

Table 5

Validation metrics comparing original UCOD or routine UCOD with MRR UCOD (top 16 specific UCOD)

	Cause of death	Original UCOD						Routine UCOD
		Sensitivity	PPV	Kappa	CCC	CSMF	CSMF accuracy	Sensitivity	PPV	Kappa	CCC	CSMF	CSMF accuracy
1	Ischaemic heart disease	0.54	0.92	0.62	0.51	12.4		0.85	0.88	0.83	0.84	20.4
2	Ischaemic stroke	0.38	0.87	0.49	0.35	5.9		0.84	0.83	0.81	0.83	13.4
3	Tracheal, bronchus and lung cancer	0.83	0.99	0.89	0.82	8.6		0.95	0.98	0.96	0.95	10.0
4	Chronic obstructive pulmonary disease	0.67	0.88	0.74	0.65	7.3		0.83	0.92	0.86	0.82	8.7
5	Stomach cancer	0.81	0.98	0.88	0.80	6.2		0.94	0.98	0.96	0.94	7.2
6	Intracerebral haemorrhage	0.51	0.92	0.64	0.48	3.8		0.90	0.92	0.90	0.89	6.7
7	Colon and rectum cancer	0.82	0.96	0.88	0.81	5.7		0.93	0.92	0.92	0.93	6.7
8	Liver cancer	0.61	0.98	0.74	0.58	3.6		0.95	0.99	0.97	0.95	5.6
9	Pancreatic cancer	0.78	0.97	0.86	0.77	4.3		0.94	0.97	0.95	0.93	5.2
10	Diabetes mellitus type 2	0.36	0.60	0.44	0.32	2.0		0.55	0.55	0.54	0.52	3.3
11	Falls	0.18	1.00	0.31	0.13	0.4		0.58	0.85	0.68	0.55	1.5
12	Breast cancer	0.89	0.96	0.92	0.89	1.5		1.00	0.97	0.98	1.00	1.7
13	Chronic kidney disease due to diabetes mellitus type 2	0.20	0.71	0.31	0.15	0.4		0.40	0.83	0.54	0.36	0.7
14	Oesophageal cancer	0.86	0.95	0.90	0.86	1.1		0.95	0.95	0.95	0.95	1.3
15	Prostate cancer	0.61	1.00	0.76	0.59	0.6		0.89	0.84	0.86	0.88	1.1
16	Garbage code	1.00	0.03	0.04	1.00	30.8		0.50	0.18	0.25	0.47	2.3
17	Others	0.33	0.13	0.16	0.29	5.5		0.61	0.29	0.37	0.59	4.4
	Average				0.57(0.44,0.70)		0.66				0.75 (0.66,0.85)		0.96

CCC, chance-corrected concordance; CSMF, cause-specific mortality fraction; MRR, medical records review; PPV, positive predictive value; UCOD, underlying cause of death.

Overall, the original UCOD had a CCC and CSMF accuracy of 0.57 (0.44, 0.70) and 0.66 (on a scale from 0 to 1), respectively, meaning that, on average, only 57% of all the UCODs reported by health facilities were correctly diagnosed, and that, overall, causes of death are only about two-thirds as accurate as they should be for guiding policy. The sensitivity and CCC were highest for garbage codes and cancers (breast, colon, lung, oesophagus and stomach) and lowest for falls, and diabetes mellitus type 2 (including CKD due to diabetes).

PPV was high for all causes, except for garbage codes and diabetes (including CKD due to diabetes), indicating that for cases when specific causes (instead of garbage codes) were reported as original UCODs, they were usually consistent with the MRR evaluation.

The validation metrics for the routine UCOD demonstrated a high level of concordance with the reference MRR diagnoses, with CCC and CSMF accuracy of 0.75 (0.66, 0.85) and 0.96, respectively (table 5 and online supplemental figure 1), confirming the impression based on the misclassification matrix. The only causes with relatively low concordance were falls and diabetes mellitus (including diabetes mellitus type 2 and CKD due to diabetes mellitus), which were commonly assigned to other causes such as garbage codes and cardiovascular diseases in the routine UCOD. In addition, CKD due to diabetes mellitus type 2 tended to be misclassified as diabetes type 2 (see tables 4 and 5).

The overall accuracy of mortality data, in Shanghai, as measured by CSMF and CCC, was substantially higher for the routine data than the original death certificates for all three types of health facilities. Interestingly, COD diagnoses for home deaths were more accurate than hospital deaths, with tertiary hospitals assigning less accurate CODs than the other two facilities (see from online supplemental tables 4–6).

Discussion

The quality of COD data reported by the Shanghai CRVS system varies greatly between the original and routine UCOD. The original UCOD data, based solely on medical certification by doctors and public health physicians, is of only moderate quality, with 31% of deaths being assigned to garbage codes (ranked first among all causes) and an overall CSMF accuracy of 0.66 when compared with a much more reliable reference data set (MRR UCOD). The routine UCOD data, following extensive quality control and rigorous review with follow-back as necessary, are however, much more reliable and highly concordant with the MRR UCOD, with only 2.3% of deaths being assigned to garbage codes and overall CSMF accuracy of 0.96, exceeding that found in hospitals in the Philippines and Mexico.21 23 Introducing rigorous review procedures for COD data can therefore greatly improve, the quality of COD data, as has also been demonstrated in Brazil.26

Our study has identified a substantial diagnostic deficit in the quality of the original UCODS, with a high proportion of CODs incorrectly classified as garbage codes. Potentially more concerning, however, is that cardiovascular diseases were often misclassified as COPD and diabetes. MRR studies from Mexico and the Philippines suggested that deaths due to falls, pneumonia and cirrhosis were often wrongly assigned to cardiovascular disease,21 23 whereas these UCODs were often assigned to garbage codes in our data. Further, CKD due to diabetes mellitus tended to be misclassified as diabetes, although this may be less grave from a public health perspective as they both highlight the importance of diabetes control. On further investigation, we found that among all cases where diabetes as the UCOD had been misclassified to CKD due to diabetes, there was clear evidence from the death sequence that diabetes was leading to chronic renal insufficiency, and even to uraemia. It is unlikely that this distinction would have significant public health implications, although the GBD classification separates the two causes of deaths, possibly to facilitate more in-depth epidemiological analyses. It is also worth mentioning that death due to ‘falls’, which was not in the top 15 CSMFs based on the original UCOD, increased in rank based on the MRR. In terms of the original UCOD, when there is a fall, the clinician filling in the death certificate tends to describe the symptoms such as fracture, haematoma or multiple organ failure rather than the fall itself as the underlying COD.

The poor quality of the original UCOD reported by the certifying physicians can likely be attributed to insufficient trainings in correct death certification in the undergraduate medical curriculum or during their residency period, a lack of understanding of the public health utility of COD, and misunderstanding the concept of underlying COD. These problems are, somewhat surprisingly, worse in tertiary hospitals compared with secondary hospitals and especially CHCs. Possible explanations could in part be due to lack of training resources and sensitisation for doctors in large hospitals in the importance of correct COD certification. The workloads and responsibilities of doctors vary across different departments and different hospitals, undoubtedly affecting the quality of death certification. This situation is similar to many other countries.27 In addition, the public health sector (CDC) has no executive leadership capacity for secondary or tertiary hospitals in China, which makes the CDC requirements harder for hospitals to follow. Conversely, for home deaths reported through the CHCs, the terminal disease process is typically less complex, with likely greater compliance among doctors.

Even though the reporting of garbage codes is low in the routine Shanghai CRVS system, our study revealed that a certain degree of misclassification still exists. Deaths due to diabetes mellitus were misclassified to cardiovascular disease, undoubtedly reflecting the difficulties in deciding the clinical sequence and origin of the diseases from clinical judgement. Or because the determination of the UCOD in such cases is often no more than the certifiers medical opinion, which may differ from one to another, even when based on the same information. The misclassification of diabetes to CKD may have arisen, either because CKD may be recognised as simply a signal of physical failure by physicians, or because the certifier is not familiar with the relevance of the distinction for public health. Few deaths were assigned to falls, which usually occur among older people, who often present with many non-communicable diseases such as IHD, diabetes mellitus type 2, etc. The existence of possible directional misclassification might lead to unpredictable impacts on the true distribution of causes of death in Shanghai, potentially reducing their policy value. These more common misclassification errors should be specifically addressed in future efforts to train doctors in correct medical certification of COD.

There are some limitations of our study. Among the nearly 700 UCOD categories recorded in the database, the top 25 causes typically accounted for 75%–79% of all deaths. To reduce the workload of doctors and remove causes with too few cases expected in the selected facilities to provide reliable comparison with GS causes, as well as considerations of the timeline and representativeness of the sample, only the top 25 UCODs were included in our study. In addition, all the results and conclusions were deduced from records with adequate documentation, that are often not applicable for uncommon UCODs or deaths with insufficient medical records.

Another limitation was that the medical histories of deaths included in the study were only reviewed by one doctor. It is likely that a parallel review by two doctors may have revealed further insights from the medical histories, potentially leading to a different ‘GS’ diagnosis than that applied in this study. While it is difficult to assess the impact of one reviewer on diagnostic accuracy, the requirement of adhering to standardised ex-ante diagnostic criteria applied by the PHMRC should, in principle, have reduced the effect of this risk.

In conclusion, our study has highlighted that the quality control procedures implemented by district CDCs and the Shanghai municipal CDC as part of the routine CRVS system, where all deaths reported as garbage codes and other implausible causes of deaths are investigated and corrected, substantially increased the cause-specific diagnostic accuracy and greatly reduces the percentage of garbage codes in the data.

Our study also suggests that, proper training for clinical doctors in death certification would be the most important strategy for improving UCOD data quality in Shanghai. Compulsory training for doctors is likely to be a cost-effective means for improving diagnostic accuracy compared with the more time-consuming and labour-intensive quality control process currently applied. Multiple forms of training including face-to-face or online methods are available and should be considered.28 Shanghai CDC has developed the adapted e-learning curriculums provided by the University of Melbourne with the intention of making the training on correct certification part of the regular curriculum for medical students. In addition, Shanghai CDC is currently in the process of compiling training materials from actual case examples in Shanghai with systematic and high frequency errors. However, knowledge gained through a training course does not always guarantee improvement in certification. Hence, a clear implication of our study is the need to improve the information exchange mechanism between the district CDC doctors responsible for correcting the medical certificates of UCOD and the hospitals, to ensure that feedback is effective and contributes to preventing diagnostic errors at source, as recognised elsewhere.29–31 Making it a requirement that physicians show competency in COD certification in order to complete their residency training would also enhance certification competency, as is now being piloted by Shanghai CDC.