Development and validation of a risk prediction model for severe hypoglycemia in adult patients with type 2 diabetes: a nationwide population-based cohort study.

PURPOSE: There is a scarcity of long-term prediction models for severe hypoglycemia (SH) in subjects with type 2 diabetes mellitus (T2DM). In this study, a model was developed and validated to predict the risk of SH in adult patients with T2DM. PATIENTS AND METHODS: Baseline and follow-up data from patients with T2DM who received health evaluations from January 1, 2009, to December 31, 2010 (n=1,676,885) were analyzed as development (n=1,173,820) and validation (n=503,065) cohorts using the National Health Insurance Database (DB) in Korea. New SH episodes were identified using ICD-10 codes. A Cox proportional hazards regression model and Cox model coefficients were used to derive a risk scoring system, and 14 predictive variables were selected. A risk score nomogram based on the risk prediction model was created to estimate the 1-year risk of SH.
RESULTS: In the development cohort, a total of 5,325 (0.45%) patients experienced SH episodes during the follow-up period. After multivariable adjustment, older age, female sex, current smoker, drinking, low body mass index, lack of exercise, previous SH events, insulin or multiple oral hypoglycemic agent use, presence of hypertension or chronic kidney disease, longer duration of diabetes, low or high glucose level, and high Charlson Comorbidity Index score were found to be significant risk factors for the development of SH and were incorporated into the risk model. The concordance indices were 0.871 (95% confidence interval, 0.863-0.881) in development cohort and 0.866 (95% CI, 0.856-0.879) in the validation cohort. The calibration plot showed a nearly 45° line, which indicates that this model predicts well an absolute SH event.
CONCLUSION: This 14-variable prediction model for SH events may be a useful tool to identify high-risk patients and guide prevention of SH in adult patients with T2DM.

Introduction

Many large epidemiological studies have emphasized that glycemic control with the HbA1c target <7% is the best strategy for the prevention of diabetes-related vascular complications.1 International practice guidelines recommend that various classes of antidiabetic medications, along with lifestyle modifications, should be initiated to achieve glycemic goals.2,3 In addition, a stepwise combination and intensification of medications are recommended for proper glycemic control in the updated guidelines. However, intensive glycemic control with antidiabetic medications or insulin injections can increase the risk of macrovascular events and mortality in patients with comorbidities and can be accompanied by the risk of hypoglycemia in patients with type 2 diabetes mellitus (T2DM).4,5

Hypoglycemia associated with antidiabetic drugs is a major barrier in diabetes treatment.6 It is categorized as mild if it is self-limiting and severe if it requires external assistance for recovery.7 Only a small fraction of patients with T2DM with glycemic control experience severe hypoglycemia (SH) episodes. One US study suggested that the age- and sex-standardized rate of SH was 1.3 events per 100 person-years in 2013, and a UK retrospective study showed that admission for hypoglycemia was 72.3 per 100,000 total hospital admissions in 2013–2014.8,9 However, the outcome of an SH event can be serious. Hypoglycemia, regardless of severity, is clinically important not only due to its association with increased cardiovascular disease (CVD) and cardiovascular (CV) mortality but also first and foremost due to the risk of loss of consciousness, falls, seizures, motor vehicle accidents, other injuries, and possible coma and death.10 Meta-analyses and large clinical studies have demonstrated that SH is significantly associated with a higher risk of CVD and an increased risk of all-cause and CV mortality in patients with T2DM.11–13 Although the causal relationship between SH and CVD or CV death has not been clearly elucidated, SH has the potential to trigger fatal arrhythmic events by more than one mechanism in patients with a high CV risk.14,15 Therefore, methods to identify high-risk groups and interventions to prevent SH events are urgently needed. An accurate prediction model for SH would be very helpful for this purpose. Despite the clinical importance of a prediction model, there is a scarcity of currently available long-term prediction models for SH in T2DM.

In Korea, the national health care program operated by the Ministry of Health and Welfare, called the National Health Insurance Service (NHIS), covers the entire Korean population through a social insurance benefits system.16 The computerized database (DB) includes all of the national health examination DBs, claims DBs, and information on drug prescriptions using the ICD-10 coding system.16,17 From this DB, we can sort out claims and national health examination data for patients with T2DM within a certain time period and obtain subsequent specific clinical outcome results during follow-up.

The aim of this study was to create development and validation cohorts and to develop an SH risk prediction model in adult patients with T2DM using the NHIS DB in Korea. To the best of our knowledge, the present study is the first long-term risk prediction tool for identifying individuals at high-risk for SH in a large number of subjects with T2DM.

Patients and methods

DB source

In this retrospective cohort study, we used the NHIS DB maintained by the Korean NHIS, a government-affiliated agency under the Korean Ministry of Health and Welfare that supervises all medical services in Korea. All Korean citizens are encouraged to receive regular biannual or pre-employment health evaluations provided by the NHIS. This regular health evaluation includes assessments of anthropometric measurements, blood pressure, social history, physical activity levels, and laboratory tests after overnight fasting, including serum glucose, total cholesterol, creatinine, liver function, and urinalysis. Quality control procedures for laboratory tests are performed in accordance with the Korean Association of Laboratory Quality Control.17 Medical history and alcohol, smoking, and exercise habits are collected by standardized self-reporting questionnaires. Additionally, the NHIS has information on patients’ demographics, medication use, transaction information, healthcare evaluations, and the claims DB.16,17

This study was based on data from NHIS which were fully anonymized and de-identified for the analysis. This study was approved by the institutional review board of the Catholic University of Korea (VC17RESI-0160). The study was conducted in compliance with the Declaration of Helsinki.

Eligibility criteria and definition of T2DM

Among all NHI beneficiaries, the source population for this study consisted of patients who visited clinics or hospitals from January 1, 2009 to December 31, 2010. Screening rates of national health examination between 2009 and 2010 was 54.2% of total population aged over 30 years. Patients were classified as having T2DM 1) when they had at least one service claim with a diagnosis of T2DM based on ICD-10 (E11–14) in the outpatient or inpatient setting and were prescribed at least one antidiabetic drug (metformin, sulfonylureas, alpha-glucosidase inhibitors, meglitinides, dipeptidyl peptidase-4 (DPP-4) inhibitors, thiazolidinediones, and any type of insulin) at any time over 1 year to exclude prediabetic or non-diabetic individuals or 2) when they showed fasting plasma glucose ≥7.0 mmol/L during health examinations (newly diagnosed diabetes).16–18 We excluded duplicate individuals who underwent multiple health examinations in consecutive years.

The inclusion criteria for patients with T2DM were as follows: 1) ≥30 years old; 2) received health evaluations between January 1, 2009, and December 31, 2010. Patients with T1DM, with gestational diabetes, or with missing data were excluded. The number of excluded participants due to missing data for each variable was as follows: current smoker, 6,615 (0.4% of total subjects); alcohol consumption, 20,791 (1.2%); exercise, 7,684 (0.4%); hypertension, 2,486 (0.1%); total cholesterol, 2,279 (0.1%); body mass index (BMI), 2,101 (0.1%), and estimated glomerular filtration rate (GFR), 1,859 (0.1%), and missing data of the other variables were all included in excluded data (Figure S1). Among the participants who had received a national health examination between January 1, 2009 and December 31, 2010 (n=1,934,024), 1,676,885 patients with T2DM were enrolled in this study (Figure S1), and whether they experienced SH episodes within the 3 years before the date of the current health examination (January 1, 2006 to December 31, 2008) was identified (Figure S2). The claims DBs were followed until December 31, 2011, to evaluate the occurrence of SH among the included participants.

Cohort development

The development and validation data sets were formed by splitting the original cohort DB into two data sets. For the development cohort, 70% of the eligible subjects were sampled from the DB from January 1, 2009 to December 31, 2010. A total of 1,173,820 individuals with T2DM were finally selected. For the internal validation cohort, 503,065 subjects with T2DM (the remaining 30% of the total eligible subjects in the DB) were extracted from the same DB according to the Harrell’s bootstrap resampling method (Figure S1).

SH event assessment as an outcome

Generally, SH was defined as any hypoglycemic event requiring the assistance of another person to actively administer carbohydrates or other corrective actions, hospitalization, or medical care.17 It also includes hypoglycemia associated with severe cognitive impairment requiring external assistance for recovery. Due to the limitation of the claims DB, from which glucose concentrations during SH events could not be confirmed, we used the ICD-10 codes for SH (E16.x, E11.63, E13.63, E14.63) in the enrolled patients with T2DM from the claims DB records of hospitalizations, clinic visits, or emergency room for 1 year from the index date.19,20 Only the first SH episode within the observation period was included. We searched for participants who had experienced at least one SH episode according to ICD-10 codes within the 3 years prior to the date of the current health examination.

Predictor variables

All the available values were extracted from the NHIS DB. Demographic information and personal clinical information were obtained, including age, sex, BMI, and socioeconomic status as quartile income status. Age range was divided into three groups (<40, 40–64, and ≥65 years) according to the logistic regression results to simplify the risk model. We also classified the subjects by BMI into five groups according to the BMI category of WHO recommendations for Asians (<18.5, 18.5–22.9, 23.0–24.9, 25.0–29.9, and ≥30 kg/m2) and fasting glucose levels into six groups (<5.6, 5.6–6.9, 7.0–8.2, 8.3–9.9, 10.0–11.6, and ≥11.7 mmol/L). Heavy drinking was defined as alcohol consumption ≥30 g/day, and physical activity was defined as 30 minutes of moderate exercise ≥5 days/week or 20 minutes of vigorous exercise 3 days/week. Current smoking status and lifestyle habits were determined by self-reported questionnaires. Hypertension was defined by the ICD-10 codes of I10-I13 and I15 and treatment with anti-hypertensive agents or systolic or diastolic blood pressure ≥140 and ≥90 mmHg, respectively. Dyslipidemia was defined by the ICD-10 code of E78 and treatment with lipid-lowering agents or total cholesterol ≥6.2 mmol/L.15 Duration of diabetes was categorized as <5 years and ≥5 years using the date of the first claim in the pair as the index date. Laboratory parameters were measured after an overnight fast. Estimated GFR (eGFR) was calculated from serum creatinine using the Modification of Diet in Renal Disease Study Group equation.21 Chronic kidney disease (CKD) was defined as an eGFR <60 mL/min/1.73 m2.22

We identified potential risk factors for SH based on the literature10,20 and selected variables that were most likely to have good predictive abilities.23 We finally considered the following 14 candidate predictors: age, sex, current smoking, BMI, exercise, alcohol consumption, insulin or multiple oral hypoglycemic agent (OHA) use, Charlson Comorbidity Index score, hypertension, CKD, previous SH history within the past 3 years, duration of diabetes, and fasting glucose level.

The primary outcome was newly diagnosed SH after the health examination, including diagnosis (ICD-10 code of SH) during hospitalization, in the emergency room, or in the outpatient clinics or until censored by death.24

Statistical analysis

Statistical analyses were performed using SAS version 9.4 (SAS Institute Inc., Cary, NC, USA). Descriptive statistics are presented as means ± SD or 95% CIs. Data are also expressed as numbers and frequencies as percentages (%). Chi-squared test was used to determine the differences in the proportion of categorical variables, and independent Student’s t-tests were conducted to evaluate the differences between the means of two continuous variables. Incidence rates are expressed as events per 1,000 patient-years.

A Cox proportional hazards regression model was used to estimate the hazard ratio (HR) and corresponding 95% CI to predict the study outcome (SH). Variable selection to build the risk prediction model was conducted using the multivariate model. We identified potential risk factors for SH based on the literature and also selected variables that were most likely to have good predictive ability in our model. The coefficients from the fitted Cox model were used to develop an interactive web-based tool that calculates the predictions automatically (available from http://md.koobian.com/sh/index.html). The proportional hazards assumption was evaluated by Schoenfeld residuals with the logarithm of the cumulative hazards function based on Kaplan–Meier curves. We assigned risk scores based on the HR for each risk factor in the final Cox hazard regression model. Each of the 14 variables, including age, sex, current smoking, drinking, BMI, exercise status, insulin use, presence of hypertension or CKD, previous SH events within the past 3 years, duration of diabetes, number of OHA classes taken, fasting glucose level category, and Charlson Comorbidity Index score was applied with scores of 0–100. Each variable corresponded to a specific point by drawing a line straight up the score axis.

To assess discrimination, receiver operating characteristic (ROC) curves and Harrell C-statistics were compared with the risk prediction model. The C-statistic is a concordance measure analogous to the area under the ROC curve.25 A C-statistic value of 0.5 means a random chance and the usual C-statistic range for a prediction model is 0.60–0.85; a range of 0.80–0.90 is considered excellent.25 Observed and predicted risks of SH at 1 year in the development and validation cohorts were compared by ranking subjects into deciles of predicted risk. Calibration was also assessed using the quintiles of the predicted SH event at 1 year and calculation of the corresponding observed Kaplan–Meier estimates.

The χ2 statistic was calculated by first dividing the data into deciles based on the predicted probabilities produced by the model in ascending order. Then, the average predicted probabilities were compared with the actual event rate estimated by the Kaplan–Meier approach in each decile. We did not need to consider missing data during the follow-up because all the claims details from the medical institutions and death certificates (censoring by death) for the participants were monitored using the DBs.

The risk prediction model for SH was translated into a risk score nomogram.26 A two-sided P-value <0.05 was considered statistically significant, and all analyses were performed using complete data only.

Results

Clinical characteristics of the study population in the development cohort

From January 1, 2009, to December 31, 2010, 1,676,885 participants with T2DM were included in this study after exclusion. At baseline, the mean age was 57.9±12.1 years, and 39.9% of the patients were female in the development cohort. During the follow-up period of 1.0 years, 5,325 individuals (0.45% of the total patients with T2DM) had a newly identified SH event. Compared with the patients without SH events, patients with SH events were older, had a lower BMI (23.6±3.5 vs 25.0±3.3 kg/m2), and were more likely to be female than male (51.7% vs 39.7%), have hypertension (77.8% vs 57.7%), be on insulin treatment (41.6% vs 8.6%), be taking multiple classes of OHAs (72.3% vs 43.9%), have had diabetes for a duration of ≥5 years (67.8% vs 33.8%), have higher Charlson Comorbidity Index score (score over 6 points: 36.2% vs score 0 or 1 point: 8.1%), and have a previous history of an SH event (4.4% vs 0.1%) (Table 1). The age- and sex-adjusted HRs and HRs for all 14 variables for the model are presented (Table 2). The age-and sex-adjusted HR for SH in older patients (≥65 years) according to age category was 7.92 (95% CI 7.35–8.53). This significant result persisted after adjusting for age category, sex, exercise level, BMI, smoking and drinking habits, presence of hypertension and CKD, insulin or multiple OHA use, glucose level, recent SH history, a Charlson Comorbidity Index score, and duration of diabetes. The fasting glucose levels, which were categorized into six levels, showed a U-shaped relationship. A BMI <18.5 kg/m2 revealed a higher HR than those of the normal or obese populations. Additionally, female sex (HR 1.07, 95% CI 1.01–1.14), presence of hypertension (HR 1.29, 95% CI 1.21–1.38) or CKD (HR 1.92, 95% CI 1.81–2.04), use of insulin (HR 2.76, 2.60–2.93), use of ≥2 classes of OHAs (HR 1.68, 95% CI 1.52–1.85), ≥2 SH events within the previous 3 years (HR 8.02, 95% CI 7.01–9.19), and duration of diabetes >5 years (HR 1.43, 95% CI 1.35–1.53) were also significant predictive factors for the development of SH after adjustment for all 14 variables (Table 2). A risk score nomogram based on the risk prediction model was created to estimate the 1-year risk of SH (Figure 1, Table S1).

Table 1

Baseline characteristics of the study participants according to SH events in the development and validation cohorts

	Development cohort (n=1,173,820)			Validation cohort (n=503,065)
	SH event		P-value	SH event		P-value
	No	Yes		No	Yes
N	1,168,495	5,325		500,821	2,244
Age (years)			<0.0001			<0.0001
<40	615,314 (52.7)	948 (17.8)		264,250 (52.8)	429 (19.1)
40–64	331,484 (28.4)	1,568 (29.5)		141,996 (28.4)	671 (29.9)
≥65	221,697 (19.0)	2,809 (52.8)		94,575 (18.9)	1,144 (51.0)
Sex (male)	704,703 (60.3)	2,574 (48.3)	<0.0001	301,274 (60.2)	1,134 (50.5)	<0.0001
Smoking (current)	292,127 (25.0)	916 (17.2)	<0.0001	125,667 (25.1)	390 (17.4)	<0.0001
Drinking (heavy)	99,000 (8.5)	280 (5.3)	<0.0001	42,340 (8.5)	122 (5.4)	<0.0001
Exercise (yes)	567,338 (48.6)	1,712 (32.2)	<0.0001	243,160 (48.6)	756 (33.7)	<0.0001
SH within recent 3 years (times)			<0.0001			<0.0001
0	1,161,444 (99.4)	4,575 (85.9)		497,794 (99.4)	1,943 (86.6)
1	6,010 (0.5)	515(9.7)		2,569 (0.5)	191 (8.5)
≥2	1,041 (0.1)	235 (4.4)		458 (0.1)	110 (4.9)
Chronic kidney disease (yes)	144,597 (12.4)	2,122 (39.9)	<0.0001	62,145 (12.4)	880 (39.2)	<0.0001
Hypertension (yes)	674,000 (57.7)	4,140 (77.8)	<0.0001	288,787 (57.7)	1,732 (77.2)	<0.0001
Dyslipidemia (yes)	491,420 (42.1)	2,472 (46.4)	<0.0001	210,640 (42.1)	1,006 (44.8)	<0.0001
Insulin (yes)	100,450 (8.6)	2,213 (41.6)	<0.0001	42,682 (8.5)	920 (41.0)	<0.0001
Number of OHA classes			<0.0001			<0.0001
0	426,528 (36.5)	565 (10.6)		182,958 (36.5)	224 (1.0)
1	228,968 (19.6)	912 (17.1)		98,571 (19.7)	376 (16.8)
≥2	512,999 (43.9)	3,848 (72.3)		219,292 (43.8)	1,644 (73.3)
Glucose level (mmol/L)			<0.0001			<0.0001
<5.6	117,642 (10.1)	1,705 (32.0)		50,493 (10.1)	742 (33.1)
5.6–6.9	233,052 (19.9)	1,299 (24.4)		99,633 (19.9)	568 (25.3)
7.0–8.2	450,440 (38.6)	852 (16.0)		193,162 (38.6)	303 (13.5)
8.3–9.9	189,245 (16.2)	591 (11.1)		80,999 (16.2)	248 (11.1)
10.0–11.6	80,524 (6.9)	316 (5.9)		34,504 (6.9)	136 (6.1)
≥11.7	97,592 (8.4)	562 (10.6)		42,030 (8.4)	247 (11.0)
Charlson Comorbidity Index score			<0.0001			<0.0001
0, 1	467,947 (40.1)	376 (7.1)		200,839 (40.1)	185 (8.2)
2, 3	385,839 (33.0)	1,283 (24.1)		165,064 (33.0)	536 (23.9)
4, 5	219,689 (18.8)	1,736 (32.6)		94,028 (18.8)	759 (33.8)
≥6	95,020 (8.1)	1,930 (36.2)		40,890 (8.2)	764 (34.1)
Duration (≥5 years)	395,092 (33.8)	3,612 (67.8)	<0.0001	168,906 (33.7)	1,504 (67.0)	<0.0001
Age (years)	57.9±12.1	68.1±10.5	<0.0001	57.87±12.1	67.8±10.5	<0.0001
Weight (kg)	66.2±11.6	58.8±10.3	<0.0001	66.12±11.56	58.9±10.4	<0.0001
BMI (kg/m2)	25.0±3.3	23.6±3.5	<0.0001	25.02±3.3	23.5±3.3	<0.0001
Waist circumference (cm)	85.5±8.4	83.6±8.9	<0.0001	85.44±8.44	83.7±8.7	<0.0001
Glucose (mmol/L)	8.0±2.5	7.4±3.2	<0.0001	8.0±2.5	7.4±3.3	<0.0001
Systolic BP (mmHg)	129.1±15.7	130.0±18.0	<0.0001	129.09±15.69	129.2±17.9	<0.0001
Diastolic BP (mmHg)	79.0±10.1	77.5±10.8	<0.0001	78.96±10.12	76.7±10.5	<0.0001
Total cholesterol (mmol/L)	5.1±1.1	4.8±1.1	<0.0001	5.1±1.1	4.7±1.1	<0.0001
HDL-cholesterol (mmol/L)	1.3±0.3	1.3±0.4	<0.0001	1.3±0.3	1.3±0.4	<0.0001
LDL-cholesterol (mmol/L)	2.9±1.0	2.7±1.0	<0.0001	2.9±1.0	2.7±1.0	<0.0001
eGFR (mL/min/1.73 m2)	82.0±21.8	67.8±26.1	<0.0001	82.02±21.83	67.1±24.9	<0.0001
Triglyceride (mmol/L)*	1.6 (1.1–2.4)	1.4 (1.0–2.1)	<0.0001	1.7 (1.5–1.9)	1.5 (1.4–1.5)	<0.0001

Note: Data are number (%) or mean ± SD.

Triglyceride, median (25th–75th percentile).

Abbreviations: BP, blood pressure; eGFR, estimated glomerular filtration rate; OHA, oral hypoglycemic agents; SH, severe hypoglycemia.

Table 2

Hazards ratios (95% CIs) for SH events

	N	Event	Duration	IR (per 1,000 person-year)	Model 1	Model 2
Age (years)
<40	616,262	948	615,362.09	1.54	1 (ref)	1 (ref)
40–64	333,052	1,568	331,778.36	4.73	3.00 (2.77–3.25)	1.53 (1.41–1.67)
≥65	224,506	2,809	222,291.94	12.64	7.92 (7.35–8.53)	2.79 (2.57–3.03)
Sex
Male	707,277	2,574	704,790.41	3.65	1 (ref)	1 (ref)
Female	466,543	2,751	464,641.99	5.92	1.11 (1.05–1.17)	1.07 (1.01–1.14)
Smoking status
No	880,777	4,409	877,290.72	5.02	1 (ref)	1 (ref)
Current	293,043	916	292,141.68	3.13	1.20 (1.11–1.30)	1.17 (1.08–1.26)
Alcohol consumption
No	1,074,540	5,045	1,070,411.19	4.71	1 (ref)	1 (ref)
Heavy	99,280	280	99,021.21	2.82	1.04 (0.92–1.18)	1.31 (1.16–1.49)
Exercise
No	604,770	3,613	601,772.39	6.00	1 (ref)	1 (ref)
Yes	569,050	1,712	567,660.01	3.01	0.69 (0.65–0.73)	0.77 (0.73–0.82)
SH within recent 3 years
0	1,166,019	4,575	1,162,110.99	3.93	1 (ref)	1 (ref)
1	6,525	515	6,196.35	83.11	11.83 (10.79–12.98)	4.64 (4.22–5.09)
≥2	1,276	235	1,125.05	208.87	26.83 (23.50–30.64)	8.02 (7.01–9.19)
Chronic kidney disease
No	1,027,101	3,203	1,024,223.88	3.12	1 (ref)	1 (ref)
Yes	146,719	2,122	145,208.52	14.61	2.77 (2.62–2.94)	1.92 (1.81–2.04)
BMI (kg/m2)
<18.5	18,277	314	17,953.95	17.48	1.92 (1.70–2.16)	1.83 (1.62–2.06)
18.5–22.9	297,961	2,078	296,160.15	7.01	1 (ref)	1 (ref)
23.0–24.9	299,286	1,233	298,301.72	4.13	0.64 (0.56–0.69)	0.68 (0.63–0.73)
25.0–29.9	475,675	1,464	474,555.27	3.08	0.52 (0.48–0.55)	0.56 (0.52–0.60)
≥30	82,621	236	82,461.32	2.86	0.56 (0.49–0.64)	0.55 (0.48–0.63)
Hypertension
No	495,680	1,185	494,489.55	2.39	1 (ref)	1 (ref)
Yes	678,140	4,140	674,942.85	6.13	1.59 (1.49–1.70)	1.29 (1.21–1.38)
Dyslipidemia
No	679,928	2,853	677,345.76	4.21	1 (ref)	1 (ref)
Yes	493,892	2,472	492,086.64	5.02	1.13 (1.07–1.19)	0.90 (0.85–0.95)
Insulin
No	1,071,157	3,112	1,068,442.22	2.91	1 (ref)	1 (ref)
Yes	102,663	2,213	100,990.18	21.9	6.08 (5.76–6.42)	2.75 (2.58–2.92)
Number of OHA classes
0	427,093	565	426,229.22	1.32	1 (ref)	1 (ref)
1	229,880	912	229,113.86	3.98	1.84 (1.65–2.04)	1.01 (0.91–1.13)
≥2	516,847	3,848	514,089.32	7.48	3.68 (3.37–4.02)	1.68 (1.52–1.85)
Glucose level (mmol/L)
<5.6	119,347	1,705	118,166.85	14.42	5.07 (4.67–5.51)	2.59 (2.37–2.82)
5.6–6.9	78,197	483	77,818.74	6.20	2.11 (1.94–2.30)	1.38 (1.26–1.50)
7.0–8.2	156,154	816	155,545.42	5.24	1 (ref)	1 (ref)
8.3–9.9	451,292	852	450,352.21	1.89	1.60 (1.44–1.77)	1.10 (0.99–1.22)
10.0–11.6	146,202	430	145,794.2	2.94	2.11 (1.86–2.40)	1.21 (1.06–1.38)
≥11.7	222,628	1,039	221,754.97	4.68	3.50 (3.151–3.89)	1.59 (1.42–1.77)
Charlson Comorbidity Index score
0, 1	468,323	376	467,873.96	0.80	1 (ref)	1 (ref)
2, 3	387,122	1,283	386,111.69	3.32	2.92 (2.60–3.28)	1.95 (1.73–2.20)
4, 5	221,425	1,736	220,105.78	7.89	6.05 (5.41–6.78)	3.04 (2.70–3.43)
≥6	96,950	1,930	95,340.97	20.24	13.50 (12.06–15.11)	4.81 (4.26–5.44)
Diabetic duration (years)
<5	775,116	1,713	773,284.44	2.21	1 (ref)	1 (ref)
≥5	398,704	3,612	396,147.96	9.11	2.87 (2.71–3.04)	1.43 (1.35–1.53)

Notes: Data are HRs (95% CI). Model 1: Age category, sex, Model 2: Age category, sex, smoking status, drinking, exercise level, income, CKD, BMI level, hypertension, insulin use, glucose level, diabetic duration, numbers of OHA classes, recent SH events within 3 years.

Abbreviations: CKD, chronic kidney disease; IR, incidence rate; OHA, oral hypoglycemic agents; SH, severe hypoglycemia.

Figure 1

A nomogram for the prediction of the 1-year probability of severe hypoglycemia (SH).

Notes: Each of the 14 variables (age, sex, current smoking, drinking, BMI, exercise status, insulin use, presence of hypertension or chronic kidney disease, previous SH events within the past 3 years, diabetes duration, number of oral hypoglycemic agents, fasting glucose level category, and Charlson Comorbidity Index score) was applied with scores of 0–100. Each variable corresponds to a specific point by drawing a line straight up to the score axis. The total score, which is the sum of the scores for each of the 14 variables at the bottom of the nomogram, ranges from 0 to 495.

The concordance index for SH prediction was 0.871 (95% CI 0.863–0.881) in the development cohort. The sensitivity and specificity of the model prediction in the development cohort were 0.799 and 0.806, respectively.

Validation cohort of the risk model

The mean age and follow-up period of the validation cohort (n=503,065) were 57.9±12.1 and 1.0 years, respectively. The percentage of female subjects was 39.8%. Among them, 2,244 patients (4.46%) developed SH during the observation period. The clinical characteristics of the validation and development cohorts were similar, including age, BMI, sex ratio, and rate of SH events (Table 1). The concordance index for SH prediction was 0.866 (95% CI 0.856–0.879) in the validation cohort. The sensitivity and specificity of the model prediction in the validation cohort was 0.802 and 0.797. We believe that this prediction model provides accurate risk prediction for SH in patients with T2DM.

Development of scores for SH prediction

The total score, which was the sum of the scores for all 14 variables, ranged from 0 to 495. For example, a woman (3 points) with T2DM aged 60 years (20 points), with no smoking or alcohol habits (0 points each), with a BMI of 22.0 kg/m2 (29 points), with moderate-intensity exercise habits (0 points), with CKD (31 points) and hypertension (12 points), and Charlson Comorbidity Index score of 3 points (31 points), with a duration of diabetes ≥5 years (17 points), with a previous history of one SH event within the past 3 years (73 points) and a fasting glucose level of 122 mg/dL (15 points), and who was treated only with insulin (48 points) would have a total risk score of 279, and her 1-year incidence probability for SH events is estimated to be 5.7% (Figure 2). Interactive web-based platform that calculates the predictions automatically is available (http://md.koobian.com/sh/index.html, Figure S3A and B). If the total score is >400 points, the 1-year SH risk is estimated to be >50%.

Figure 2

The 1-year incidence probability of severe hypoglycemia (SH) according to the total score.

The range of cutoff values according to deciles in the development and validation cohorts is demonstrated in Figure 3. Based on this prediction model, the subjects in the highest decile (total score 192–550) showed an SH incidence rate of 29.38 per 1,000 person-years. To determine how well the tool predicts an absolute outcome, we checked the relationship between the model-derived and actual SH outcome within calibration plots (Figure S4). The dashed line represents the performance of an ideal nomogram, where the predicted outcome would correspond perfectly with the actual outcome. The performance of the created model is plotted as the solid line. The calibration plot showed a nearly 45° line, which indicates that this model predicts an absolute SH event well.

Figure 3

Predicted incidence rate (per 1,000 person-years) based on the development and validation cohorts.

Notes: The x-axis represents the range of scores from the nomogram according to each decile. The y-axis reveals the predicted incidence rate of severe hypoglycemia (SH).

Among the 14 variables, we found that previous SH events, older age, lower BMI (<18.5 kg/m2), and insulin use mainly contributed significantly to future SH risk. We also created simple prediction model for SH using these four variables (Figure S5; Table S2). The concordance indices of this simple model were 0.827 (95% CI 0.816–0.838) in the development cohort and 0.819 (95% CI 0.810–0.832) in the validation cohort.

Discussion

Using an NHI claims DB in Korea, we developed and validated a simple and useful risk prediction model for SH that can be applied to adult patients with T2DM. To the best of our knowledge, the present study is the first long-term risk prediction tool for identifying high-risk individuals for SH in a large number of patients with T2DM. Our model has good calibration and discrimination, with a C-statistic of 0.871 (95% CI 0.863–0.881). We found that older age, female sex, longer duration of diabetes, low BMI, use of insulin or multiple classes of glucose-lowering medications, current smoker, drinking, lack of exercise, presence of hypertension or CKD, previous SH history, multiple comorbidities, and low or high glucose levels were independently predicted an increased risk of SH in T2DM.

Intensive glycemic control is necessarily accompanied by an increased frequency and severity of hypoglycemic episodes in patients with T2DM. According to the Action in Diabetes and Vascular Disease: Preterax and Diamicron Modified Release Controlled Evaluation (ADVANCE) Collaborative group study, 231 of 11,140 patients with T2DM (2.1%) had at least one SH event during the study period. Moreover, SH was significantly associated with an increased risk of macrovascular complications, adverse clinical outcomes, and all-cause mortality.10,11,27 Therefore, the risks and clinical implications of SH should be considered in the management of T2DM, and accurate prediction and management of the risk factors prior to an SH event are essential for patients with T2DM.

Several risk factors for SH are known from previous studies, such as hypoglycemia unawareness, long duration of diabetes, strict glycemic control, older age, multiple comorbidities, impaired renal function, poor cognitive function, and certain biomarkers (C-peptide and 1,5-anhydro-glucitol).20,24,26,28 CKD is also one of the well-known risk factors for SH due to its role in gluconeogenesis and drug clearance.29 Using these meaningful clinical and laboratory variables, a risk prediction model for SH can be built from qualified clinical studies.30

Literature regarding risk prediction models for SH specific to T2DM is extremely rare. Most studies have been performed in a small group of patients with T1DM short-term by self-monitored blood glucose (SMBG) recordings or continuous glucose monitoring system (CGMS).31–33 Prediction of SH using the SMBG recordings in T1DM and T2DM for 4–6 months revealed that a specific blood glucose fluctuation pattern can partially predict imminent SH.31 Some studies have also suggested that long-term prediction of SH in T1DM patients is not possible based on well-known risk factors for SH assessed over 12 years.34 In the Kaiser Permanente Colorado cohort, a 16-variable model to predict the 6-month risk of SH was developed for individuals treated with hypoglycemic agents (4.3% of T1DM patients).23 The Diabetes Control and Complication Trial demonstrated that only 6%–12% of future (within several months) SH episodes could be explained from known variables.35 A short-term (within 24 hours) SH prediction model was suggested using SMBG readings to detect a characteristic pattern of glucose fluctuations in T2DM, which was able to predict ~60% of episodes.31 However, for practical application, frequent SMBG has the problem of long-term adherence, and CGMS is not routinely available due to cost issues.

We developed a long-term risk prediction model for SH in T2DM patients using a national DB. Generally, large, representative, current data sets reflecting the characteristics of the target population are ideal for creating models; therefore, our NHIS DB is valuable.25,36 In addition to anthropometric and clinical information, our model included a single measure of fasting serum glucose and creatinine level at baseline, which helped increase the predictive power of our model compared with clinical information alone.

Our prediction model consisted of 14 variables, each with a score ranging from 0 to 100, with the total score ranging from 0 to 495, and presented 1-year SH risk. A risk scoring system should be simple, contain clinically important variables influencing hazard outcomes, and be convenient to use for physicians or healthcare providers. Based on this 14-item assessment, healthcare professionals are able to easily screen high-risk patients for SH and immediately translate medical advice into actions, such as dosage adjustments of insulin or OHA, patient education, or evaluation of hypoglycemia unawareness.

There were some limitations to this analysis. First, subjects with disease codes E11–14 who were not on anti-diabetic medication were not included. In addition, various comorbidities related to higher SH risk, including retinopa-thy, CVD, depression, dementia, and heart failure, were not included in this model. We could not obtain clinical information on HbA1c levels, exact duration of diabetes, other medications that can affect the glycemic status of the subjects including beta-blocker or corticosteroid, and glucose levels during the SH event. Patients with SH episodes without claims data who did not present to emergency departments or clinics were not detected in this study. External validation and impact studies are essential in the future. The GLP-1RA was introduced in Korea by the end of 2008. However, health insurance coverage was available in November 2010 with strict conditions. Therefore, the GLP-1RA was not included in this analysis. We defined CKD from only one test of eGFR value. Due to the nature and limitation of our NHIS DB, we could not obtain eGFR value from two separate tests more than 3 months apart. Finally, risk scores may not be applicable to other ethnic groups. According to the Atherosclerosis Risk in Communities study, African ancestry was significantly associated with an increased risk of SH.20 However, other studies have demonstrated similar rates among African Americans and Caucasians.8,37,38

Despite these limitations, the major strength of this study was that the data were based on a nationwide Korean population covering nearly 100% of Korean patients with T2DM. The high prediction accuracy obtained through cross-validation of the model suggests strong internal validity of these results. This risk model also included anthropometric and lifestyle factors. Models based on clinical information in electronic medical records or face-to-face acquisition during visits can be more easily implemented in the clinical setting than complicated computer-based algorithms. Therefore, the web-based calculator derived from this model is an easily accessible format for clinical use. We expect that our SH risk calculator will make this model more user-friendly for clinical use through a mobile application or web-based program. Healthcare professionals should consider prompt intervention in patients with high scores determined by this calculator, including setting a less stringent A1C target, adjusting antidiabetic treatment, obtaining frequent SMBG readings, applying lifestyle modifications with a regular meal schedule, and re-education.2,39

Conclusion

In summary, we presented a multivariable risk model that predicts the development of SH in T2DM patients. For this specific population, an individualized glycemic target, changes to safer drugs, recommendations of behavioral interventions, and intensive reeducation about SH are clearly needed. Our SH screening score may serve as a practical tool for individuals with T2DM who are unaware of their risk and provide clinicians a way to prevent future SH events. Future studies will focus on the external validation of this risk model in a larger cohort with T2DM to strengthen the predictive power. At the same time, tailored intervention programs for T2DM patients in specific score ranges should be developed.

Data sharing statement

The complete data set and statistical code that support the findings of this study are available from NHIS in Korea but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available. Data are, however, available from the corresponding author upon reasonable request and with the permission of NHIS in Korea.

Study design summary.

Study design summarizing sample recruitment and follow-up.

(A) Web-based application for 1-year severe hypoglycemia (SH) risk model (http://md.koobian.com/sh/index.html). (B) This calculator shows the 1-year SH risk prediction automatically.

Calibration plots depicting the correlation between predicted and actual 1-year severe hypoglycemia event.

A nomogram for the four-variable simple prediction model of the 1-year probability of SH.

Abbreviations: BMI, body mass index; SH, severe hypoglycemia.

Table S1

Scores for each risk factor category

No.	Variable	Categories and score
1	Age (years)	<40	40–64	≥65
		0	20	49
2	Sex	Male	Female
		0	3
3	Current smoker	No	Current
		0	7
4	Heavy drinker	No	Yes
		0	13
5	BMI (kg/m2)	<18.5	18.5–22.9	23.0–24.9	25.0–29.9	≥30.0
		59	29	11	1	0
6	Exercise	Yes	No
		0	12
7	Insulin use	No	Yes
		0	48
8	Hypertension	No	Yes
		0	12
9	Chronic kidney disease	No	Yes
		0	31
10	Previous history of SH	0	1	2
		0	73	100
11	Diabetes duration (years)	<5	≥5
		0	17
12	Number of OHAs	0, 1	2
		0	24
13	Glucose level (mmol/L)	<5.6	5.6–6.9	7.0–8.2	8.3–9.9	10.0–11.6	≥11.7
		46	15	0	5	9	22
14	Charlson Comorbidity Index score	0, 1	2, 3	4, 5	≥6
		0	31	52	74

Abbreviations: OHA, oral hypoglycemic agents; SH, severe hypoglycemia.

Table S2

Four-variable simple model for predicting SH and the score for each variable

	HR (95% CI)	P-value	Score
Age (years)		<0.001
<65	Reference		0
≥65	3.91 (3.69–4.16)		49
BMI (kg/m2)		<0.001
<18.5	Reference		32
≥18.5	2.41 (2.15–2.70)		0
Insulin use		<0.001
No	Reference		0
Yes	5.17 (4.88–5.47)		59
Previous SH (times)		<0.001
0	Reference		0
1	8.07 (7.34–8.86)		76
≥2	15.83 (13.84–18.10)		100

Abbreviation: SH, severe hypoglycemia.