Validity of discharge ICD-10 codes in detecting the etiologies of endogenous Cushing's syndrome.

OBJECTIVE: To investigate the validity of discharge ICD-10 codes in detecting the etiology of endogenous Cushing's syndrome (CS) in hospitalized patients.
METHODS: We evaluated the sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) of CS etiology-related ICD-10 codes or code combinations by comparing hospital discharge administrative data (DAD) with established diagnoses from medical records.
RESULTS: Coding for patients with adrenocortical adenoma (ACA) and those with bilateral macronodular adrenal hyperplasia (BMAH) demonstrated disappointingly low sensitivity at 78.8% (95% CI: 70.1-85.6%) and 83.9% (95% CI: 65.5-93.9%), respectively. BMAH had the lowest PPV of 74.3% (95% CI: 56.4-86.9%). In confirmed ACA patients, the sensitivity for ACA code combinations was higher in patients initially admitted to the Department of Endocrinology before surgery than that in patients directly admitted to the Department of Urology (90.0 vs 73.1%, P = 0.033). The same phenomenon was observed in the PPV for the BMAH code (100.0 vs 60.9%, P = 0.012). Misinterpreted or confusing situations caused by coders (68.1%) and by the omission or denormalized documentation of symptomatic diagnosis by clinicians (26.1%) accounted for the main source of coding errors.
CONCLUSIONS: Hospital DAD is an effective data source for evaluating the etiology of CS but not ACA and BMAH. Improving surgeons' documentation, especially in the delineation of symptomatic and locative diagnoses in discharge abstracts; department- or disease-specific training for coders and more multidisciplinary collaboration are ways to enhance the applicability of administrative data for CS etiologies.

Introduction

Endogenous Cushing’s syndrome (CS) comprises signs and symptoms caused by pathological hypercortisolism, which seriously affects the metabolic and endocrine systems. Endogenous CS is divided between adrenocorticotropic hormone (ACTH)-dependent and ACTH-independent etiologies. The former includes Cushing’s disease (CD) and ectopic ACTH syndrome (EAS), and the latter includes adrenocortical adenoma (ACA), adrenocortical carcinoma (ACC), primary bilateral macronodular adrenal hyperplasia (BMAH) and primary pigmented nodular adrenocortical disease (PPNAD) (1). Since CS has a low incidence of 0.2–5.0 per million patients per year (2, 3, 4), enriching the epidemiological data through a massive population-based screening project is extremely difficult. Fortunately, epidemiological profile of CS were obtained from several data sources, including the national patient registry database from Netherlands and Japan (5, 6), the European Registry on Cushing’s syndrome (ERCUSYN) (7) and the questionnaire survey from Italy (8); of these data sources, hospital administrative data, such as hospital discharge abstract data (DAD), including demographic data, discharge diagnoses, surgeries or procedures and corresponding ICD codes, are increasingly applied in various kinds of clinical epidemiological studies and can be used to effectively detect patients with different types of diseases (9, 10, 11). Endocrinologists with substantial expertise and considerable clinical experience are needed to provide accurate judgment and interpretation of CS, and the particularity of both the symptomatic (qualitative) and locative diagnosis of CS may lead to the complexity, misclassification and omission of administrative ICD-10 codes. In addition, several studies have demonstrated that there are always some potential sources of error interposed between the clinician’s diagnosis and the nosological diagnosis code assigned by the clinical coder (12, 13). To date, studies on the validity of discharge ICD-10 codes from DAD in identifying the etiology of CS patients have not yet been reported. Therefore, we evaluated the validity of discharge ICD-10 codes or code combinations to identify the etiology of endogenous CS by comparing the hospital DAD with the established diagnoses in the medical records in a hospital setting.

Materials and methods

This was a retrospective study conducted at Peking Union Medical College Hospital (PUMCH), which is a large general hospital proficient in diagnosing and treating endocrine disorders (14, 15, 16, 17). Our work was approved by the Ethics Committee of PUMCH at the Chinese Academy of Medical Sciences & Peking Union Medical College and was consistent with the 1964 Declaration of Helsinki and its later amendments or comparable ethical standards. Written or verbal informed consent was obtained from all individual participants included in the study.

Data sources

Since the diagnosis of CS is established on a combination of clinical presentations, each of which are not pathognomonic, and biochemical tests that are not infallible, we added surgical intervention for primary lesions performed during the study period as an additional eligibility requirement to reduce diagnostic misclassification. Thus, we retrospectively retrieved data from the hospital DAD for patients who were admitted to the hospital for the first time and endured either transnasal transsphenoidal surgery (TSS) in the Department of Neurosurgery, adrenal surgery in the Department of Urology or surgical resection of abnormal tumors or tissues from other sites in the Department of General Surgery or the Department of Thoracic Surgery from January 1, 2015, to December 31, 2017. The medical record numbers, demographic data, admission departments, discharge departments, 20 discharge diagnoses with ICD-10 codes, 10 surgeries or procedures and pathology results for these patients’ hospitalizations were extracted directly from the hospital DAD to establish dataset A. For CS patients with multiple operations, we included only the DAD related to the first admission. The corresponding admission department related to the first hospitalization according to the inclusion criteria was collected for further analysis.

Coding mode

During the study period, the Department of Medical Records of PUMCH employed six coders who were responsible for all inpatient stays at our hospital. All coders were not disease- or department-specific coders, and none of them had advanced targeted training or particular expertise in CS. Coding at our institution is performed and completed after a patient is discharged but before the 20th of the next month, which are the data uploading deadlines for the Beijing Municipal Commission of Health and Family Planning Information Center. At the time of the study, all coding was performed manually. Coders were able to access the entire inpatient medical record. All coders received the qualification certificate of coding released by the Chinese Medical Record Association.

Diagnostic strategies for CS patients

In accordance with 2008 Endocrine Society Clinical Practice Guidelines (18), qualitative diagnosis (symptomatic diagnosis) of each endogenous CS patient at our institution was made based on the presence of clinical characteristics combined with at least one altered biochemical test, such as 24-h urinary free cortisol level, serum cortisol diurnal variation and the 1 mg overnight low-dose dexamethasone suppression test (LDDST). The locative diagnosis of CS was determined based on the serum ACTH level test, the high-dose dexamethasone suppression test (HDDST) and imaging examinations, including routine T1-weighted gadolinium-enhanced magnetic resonance imaging (MRI), dynamic gadolinium-enhanced MRI of the pituitary glands and plain and contrasted enhanced computed tomography (CT) of the adrenal, thoracoabdominal and pelvic areas. Since bilateral inferior petrosal sinus sampling (BIPSS) with desmopressin (DDAVP) has been demonstrated to have similar sensitivity and specificity to BIPSS with corticotropin-releasing hormone (CRH) (15, 19, 20), which is not available in China, we employed the DDAVP test in situations requiring BIPSS to differentially diagnose ACTH-dependent CS. The BIPSS procedures were conducted as described by Doppman et al. (21). Pathological results were also referenced as evidence for establishing locative diagnoses. The details of the procedure and the interpretation of the results of the biomedical tests and BIPSS are described in a previous study conducted at our hospital (15).

Re-adjudication of cases based on medical record review

Electronic and paper medical records were used for review by authors, who included neurosurgeons, endocrinologists and a urologist, to review and re-classify the etiology of CS. Evidence within the medical record of each patient, such as clinical manifestation from the admission record, laboratory data, results of BIPSS with DDVAP recorded in the progress note, physician or surgeon assessment, imaging reports, operative procedure note and the results of pathological examinations, was reviewed in detail to substantiate the established diagnosis according to the CS diagnostic strategies. In cases of uncertainty or in complex cases requiring additional expertise, another experienced endocrinologist adjudicated.

Identification of true and false-positive-coded CS patients with certain etiologies

We retrospectively retrieved the medical records for CS patients with a certain etiology out of dataset A using CS-related ICD-10 codes or the code combinations listed in Table 1, in accordance with the international recognized etiological classification of CS (1). The retrieved medical records were reviewed by the authors. Within the coded CS group, we adjudicated cases with correct ICD-10 codes or code combinations as determined by comparing established diagnoses from medical records into the true-positive group of the corresponding CS etiology-related ICD-10 codes group. Patients assigned CS etiology-related ICD-10 codes or code combinations who did not have clinically confirmed CS or did not have the congruent etiology matched with the assigned codes according to the established diagnosis were adjudicated into the false-positive group of the corresponding CS etiology-related ICD-10 codes group (Fig. 1).

Table 1

Etiological diagnoses and corresponding ICD-10 code or code combinations for endogenous CS.

Etiological diagnosis	ICD-10 code or code combinations	ICD-10 modification used in the hospital DAD
ACTH-dependent CS
CD- or ACTH-secreting pituitary adenoma or pituitary dependent CS	E24.0/(D35.2 + E24.9)	E24.001/(D35.2 + E24.901)
EAS	E24.3	E24.301
ACTH-independent CS
ACA	D35.0 + (E24.8/E24.9)	D35.001 + (E24.801/E24.901)
ACC	(C74.0/C74.9) + (E24.8/E24.9)	(C74.001/C74.901) + (E24.801/E24.901)
BMAH	E24.8	E24.802
PPNAD	E24.8	E24.803

Figure 1

The flow chart of validation process of CS patients with six etiologies. FN, false negative; FP, false positive; TN, true negative; TP, true positive.

Identification of true-negative patients (non-coded patients without confirmed CS) and false-negative patients (non-coded patients with CS)

When judging the true-negative or false-negative CS patients of the corresponding CS etiology-related ICD-10 codes group, different data subsets for patients without ICD-10 codes indicating either one of the six etiologies were extracted from dataset A, which are shown in Fig. 1. To effectively identify the missing etiologically confirmed CS cases without proper ICD-10 codes, we further matched the six subsets separately with the data of patients who had biochemical tests of either a 24-h urinary-free cortisol, serum cortisol or serum ACTH during the study period by a unique medical record number from the hospital laboratory database. Thus, unmatched patients who did not have any CS-related endocrine hormone tests during the study period were considered to be part of the true-negative non-coded group of the corresponding CS etiology-related ICD-10 codes group; the biochemical tests from the matched patients’ medical records were thoroughly reviewed by the author board. Based on the medical records, patients diagnosed with one of the six etiological groups of CS according to the diagnostic strategies were moved to the false-negative non-coded CS group of the corresponding CS etiology related ICD-10 codes group. The remainder of patients made up the true-negative group of non-coded patients for the corresponding etiology-related ICD-10 codes group. The true-negative group together with the false-positive group comprised the validation of the non-CS patients for each etiology. Similarly, the true-positive group and false-negative group comprised the validation of the positive CS patients for each etiology (Fig. 1).

Statistical analysis

SAS 9.0 software (version 9.0, SAS Institute) was used to analyze the data. Using the established diagnoses from the medical records of CS patients as the gold standard, the sensitivity, specificity, positive predictive values (PPV), negative predictive values (NPV) and corresponding 95% CIs of the ICD-10 codes or code combinations listed in Table 1 were calculated. The etiological spectra between the two identification strategies were compared by the chi-squared test, and differences in code sensitivity and PPV were compared by the same test between different admission departments for CD, ACA and BMAH. P < 0.05 indicated a significant difference.

Results

Sensitivity, specificity, PPV and NPV of CS etiology-related ICD-10 code and code combinations

The numbers of patients undergoing TSS, adrenal surgery or surgical resection of abnormal tumors or tissues from other sites in either the Department of General Surgery or the Department of Thoracic Surgery were 2455, 1872 and 23,079, respectively. Of these patients, 319, 25, 93, 1, 35 and 10 patients were coded as CD, EAS, ACA, ACC, BMAH and PPNAD, respectively; the codes of 312, 25, 93, 1, 26 and 10 patients were corrected by comparing administrative data with the established diagnoses for the corresponding etiologies. In negatively coded patients matched with hormone tests for cortisol or ACTH, the number of clinically confirmed CS patients was 22, 1, 25, 0, 5 and 0 for the etiological groups CD, EAS, ACA, ACC, BMAH and PPNAD, respectively (Fig. 1). The sensitivity, specificity, PPV and NPV for the ICD-10 code or code combinations reflecting the six etiologies of CS are listed in Table 2. Coding for patients with ACA or BMAH demonstrated a relatively low sensitivity, with 78.8% (95% CI: 70.1–85.6%) and 83.9% (95% CI: 65.5–93.9%) in DAD, respectively. BMAH had the lowest PPV of 74.3% (95% CI: 56.4–86.9%) among the six etiologies.

Table 2

The sensitivity, specificity, PPV and NPV of ICD-10 codes or code combinations for different etiologies of CS.

Etiology	TP	FN	FP	TN	Sensitivity (95% CI)	Specificity (95% CI)	PPV (95% CI)	NPV (95% CI)
CD	312	22	7	1446	93.4 (90.1–95.7)	99.5 (99.0–99.8)	97.8 (95.3–99.0)	98.5 (97.7–99.0)
EAS	25	1	0	233	96.2 (78.4–99.8)	100.0 (98.0–100.0)	100.0 (83.4–100.0)	99.6 (97.3–99.9)
ACA	93	25	0	631	78.8 (70.1–85.6)	100.0 (99.3–100.0)	100.0 (95.1–100.0)	96.2 (94.4–97.5)
ACC	1	0	0	748	100.0 (5.5–100.0)	100.0 (99.4–100.0)	100.0 (5.5–100.0)	100.0 (99.4–100.0)
BMAH	26	5	9	709	83.9 (65.5–93.9)	98.7 (97.5–99.4)	74.3 (56.4–86.9)	99.3 (98.3–99.7)
PPNAD	10	0	0	739	100.0 (65.5–100.0)	100.0 (99.4–100.0)	100.0 (65.5–100.0)	100.0 (99.4–100.0)

FN, false negative; FP, false positive; TN, true negative; TP, true positive.

Comparisons of etiological spectrum between two identification strategies

A total of 483 and 520 CS patients with certain causes were identified by the hospital DAD and established diagnoses from medical records, respectively. The distributions of CS etiologies resulting from these two identification strategies are detailed in Fig. 2. The respective proportions derived from hospital DAD and established diagnoses were as follows: 66.1 vs 64.2% for CD, 5.2 vs 5.0% for EAS, 19.3 vs 22.7% for ACA, 7.3 vs 6.0% for BMAH, 2.1 vs 1.9% for PPNAD and 0.2 vs 0.2% for ACC. No significant differences in etiological proportions were observed between the two strategies (P > 0.05).

Figure 2

Comparisons of the etiological spectrum of CS between hospital DAD versus established diagnoses from medical records. The horizontal axis represents the six etiological groups of CS patients and the vertical axis represents number of patients. The column color of dark blue refers to the CS patients identified by clinical objective data and the light blue refers to the CS patients detected by hospital discharge abstract data.

The influence of the admission department on the sensitivity and PPV of coding for CS with certain causes

The admission departments were different for patients with clinically confirmed CD, ACA or BMAH, while all patients with confirmed EAS or PPNAD were initially admitted to the same Department of Endocrinology before being transferred to the Surgical Department. The sensitivity and PPV of ICD-10 codes or code combinations for detecting the cause of CS in patients with different admission departments are illustrated in Table 3. In clinically confirmed ACA patients, the sensitivity was significantly higher in patients admitted to the Department of Endocrinology before being transferred to the Surgical Department than in those directly admitted to the Department of Urology (90.0 vs 73.1%, P = 0.033). In patients coded for BMAH, the PPV for the corresponding ICD-10 code was also significantly higher in patients admitted to the Department of Endocrinology than in those admitted to the Department of Urology (100.0 vs 60.9%, P = 0.012). The PPV of CD codes or code combinations in patients admitted to the Department of Urology was significantly lower than that in patients admitted to the Department of Neurosurgery or the Department of Endocrinology (20.0 vs 100.0%, 20.0 vs 98.9%, P < 0.001).

Table 3

The influence of the admission department on the sensitivity and PPV of coding for CS with certain causes.

Etiology	Admission department	Comparison of Sensitivity				Comparison of PPV
		Cases confirmed by established diagnoses (n)	TP (n)	Sensitivity (95% CI) (%)	P	Coded cases	TP (n)	PPV (95% CI) (%)	P
CD	Neurosurgery	41	39	95.1 (82.2–99.2)	0.151	39	39	100.0 (88.8–100.0)	<0.001
	Endocrinology	292	272	93.2 (89.5–95.7)		275	272	98.9 (96.6–99.7)
	Urology	1	1	100.0 (5.5–100.0)		5	1	20.0 (1.1–70.1)
EAS	Endocrinology	26	25	96.1 (78.4–99.8)	–	25	25	100.0 (83.4–100.0)	–
ACA	Endocrinology	40	36	90.0 (75.4–96.7)	0.033	36	36	100.0 (88.0–100.0)	–
	Urology	78	57	73.1 (61.6–82.2)		57	57	100.0 (91.1–100.0)
ACC	Urology	1	1	100.0 (5.5–100.0)	–	1	1	100.0 (5.5–100.0)	–
BMAH	Endocrinology	14	12	85.7 (56.2–97.5)	1.000	12	12	100.0 (69.9–100.0)	0.012
	Urology	17	14	82.4 (55.8–95.3)		23	14	60.9 (38.8–79.5)
PPNAD	Endocrinology	10	10	100.0 (65.5–100.0)	–	10	10	100.0 (65.5–100.0)	–

Categories of coding errors in identifying patients with CS by ICD-10 codes

A review of all 69 identified coding inaccuracies yielded three categories of errors (Table 4). The first category was errors resulting from misinterpreted or confused situations caused by the coder’s little understanding of the etiologies of CS, accounting for 68.1% (47/69) of all errors. The second category of coding error was due to delay of the pathology report, which caused a discrepancy between the discharge diagnosis determined according to observations during surgery and the ultimate pathological results, accounting for 5.8% (4/69) of all errors. For instance, two patients were diagnosed with BMAH, but this diagnosis was inconsistent with the final pathological results confirming adrenal cortical adenomas, leading to both false-positive ICD-10 codes for BMAH and false-negative codes for ACA. The third category is false-negative errors caused by clinicians’ omission or denormalized documentation of symptomatic diagnosis of CS, accounting for 26.1% (18/69) of all errors.

Table 4

Description of coding errors in identifying the cause of CS for two types of inconsistencies.

Type of inconsistency	Errors categories	CD	EAS	ACA	BMAH	Total
False-positive	Misinterpretation or confusion made by Coders	7	0	0	7	14
	Discrepancy between pathological diagnosis and clinical diagnosis	0	0	0	2	2
	Total	7	0	0	9	16
False-negative	Misinterpretation or confusion made by Coders	22	0	7	4	33
	Discrepancy between pathological diagnosis and clinical diagnosis	0	0	2	0	2
	Omission or denormalized documentation of symptomatic diagnosis	0	1	16	1	18
	Total	22	1	25	5	53

Discussion

Some previous studies have focused on the validity of ICD codes to identify patients with certain kinds of diseases in hospital administrative data (22, 23). However, to date, no study has reported the validity of ICD-10 codes from hospital DAD in reflecting the etiology of endogenous CS. Our study is the first to investigate the validity of ICD-10 codes for detecting the etiology of surgically treated endogenous CS patients; specifically, we evaluated the feasibility of using hospital DAD to obtain the CS etiology spectrum during a specific period. In this study, based on the established diagnoses from medical records, not only PPV for each ICD-10 code or code combinations were calculated, but also other indicators of validity, including sensitivity, specificity and NPV were obtained in our study. Furthermore, a detailed summary and analysis of the categories of errors were conducted in our study.

The results revealed no significant differences between the two identification strategies for obtaining the etiological spectrum of CS. Thus, the hospital DAD were demonstrated to be an effective data source for evaluating the etiology of CS in a single medical center. However, a gap in data collection occurred when we focused on a certain etiological group of CS. The sensitivity of coding for ACA and BMAH were disappointingly low (78.8 and 83.9%, respectively), implying that approximately one-fifth of patients with ACA or BMAH were not reported according to the hospital DAD. There are two main reasons accounting for the lower sensitivity of ACA ICD-10 code combinations. The first reason is that physicians fail to document the precise etiological diagnosis, that is ‘Cushing’s syndrome’ or ‘ACTH-independent Cushing’s syndrome’. Such omission accounted for most coding errors (64.0%) among ACA patients. The second reason is due to the clinical coders’ misinterpretation and confusion from the lack of knowledge and poor understanding of the classifications of CS etiology. In this study, seven unidentified ACA patients were unexpectedly miscoded as CD because the coders had confused the terminology of ‘Cushing’s syndrome’ with that of ‘Cushing’s disease’. Similarly, eleven cases were wrongly coded for BMAH because coders confused terminology of ‘ACTH-independent bilateral adrenal macronodular hyperplasia’ with the nonspecific description of ‘nodular hyperplasia of adrenal cortex’ in the pathology report, thus leading to the low sensitivity and PPV of the BMAH ICD-10 code.

Another finding of our study was the lower sensitivity for ACA code combinations (73.1 vs 90.0%) and the lower PPV for the BMAH code (60.9 vs 100.0%) observed in patients who were directly admitted to the Department of Urology than those observed in patients who were initially admitted to the Department of Endocrinology before being transferred to the Department of Urology. In cases where patients were directly admitted to the Department of Urology, surgeons tended to pay much more attention to the surgical intervention rather than to the endocrine function of the tumors. However, when patients were directly admitted to the Department of Endocrinology before surgery, endocrinologists usually document both the symptomatic and locative diagnoses in a standardized and comprehensive manner, which provides a useful reference for surgeons and is conducive to the coding process. For patients coded with CD, the reason for the difference in the PPV between different departments was related solely to coder’s ignorance of complex specific terminology, such as ‘Cushing’s disease’ and ‘Cushing’s syndrome’.

Potential influencing factors for coding errors also stem from the ambiguity and complexity inherent in the ICD-10 system. When identifying ACA, two codes are required simultaneously for both the tumor and its endocrine function. However, the ICD-10 system does not yet provide a comprehensive code to directly identify the etiology of ACA and cannot offer better diagnostic specificity than the terminology used in the clinical practice, which should be highly emphasized among both coders and doctors. However, these problems will be solved by the newly released ICD-11 (https://icd.who.int/browse11/l-m/en), in which the definition and etiological classification of CS are clearly provided and several index terms are available to cover clinical diagnostic terminologies.

Consequently, both coder education and surgeon behavioral modification should be implemented to improve the sensitivity of coding for ACA and BMAH and the PPV of coding for BMAH. For coders, advanced specific training and department-specific coding modes would improve coders’ familiarity with medical terminology and deepen the understanding of CS etiological classification. For surgeons, especially urologists, their knowledge base should include a basic understanding of the structure of the ICD-10 and the rules for what constitutes accurate coding to ensure high-quality documentation in the discharge abstract of CS patients. There are several shortcomings in our study. First, this is a single-center retrospective study, and the results cannot represent the coding modes and the validity of coding for CS in a whole. Second, CD patients were identified with a compound coding strategy; thus, the validity of E24.0 alone was not obtained, and the evaluation of the impact of the coding sequence on etiology identification was not included. Third, because the etiology of CS is complicated and the diagnosis is also challenging, all the subjects enrolled in this study were CS patients who had undergone surgery for primary lesions to obtain more information from pathological results to assist in etiological diagnosis. Thus, the validity of the coding results in this study should not be extrapolated to all endogenous CS patients. Fourth, the recruited number of ACC patients was very limited due to the selection criteria; thus, additional cases are needed to narrow the 95% CI for both sensitivity and PPV to evaluate the validity of ICD-10 codes for ACC.

Conclusion

Hospital DAD is a reliable and effective data source for evaluating the etiology of CS, with the exceptions of ACA and BMAH. Researchers interpreting results from studies using hospital DAD to identify ACA, ACC or BMAH patients should be cautious. Coding errors mainly stem from both coders and surgeons, especially urologists. To depict a full picture of CS patients, the diagnosis process requires clear delineation of both symptomatic and locative diagnoses. Multidisciplinary collaboration should be fortified in the areas of advanced targeted training, department-specific coding modes for coders, basic understanding of the ICD-10 and high-quality documentation for surgeons to improve the validity of CS etiology-related ICD-10 coding. Once the problems identified in this study have been corrected, a prospective study should be conducted.

Declaration of interest

The authors declare that there is no conflict of interest that could be perceived as prejudicing the impartiality of the research reported.

Funding

This study was supported by a grant from the Fundamental Research Funds for the Central University, No. 3332018043, and a grant from the Medical and Health Science and Technology Innovation Project of Chinese Academy of Medical Sciences, No. 2017-I2M-3-014.

Author contribution statement

J Z, B X and Y W were responsible for the conception and design of the study. J Z, M Z, L L, X G and L G contributed to acquisition, analysis and interpretation of the data. The manuscript was critically revised by B X, Y W, L L, W Y and M Z. J Z and H P performed statistical analysis. All the authors participated in drafting the manuscript and approved the final version.