Development and validation of a diabetes mellitus and prediabetes risk prediction function for case finding in primary care in Hong Kong: a cross-sectional study and a prospective study protocol paper.

INTRODUCTION: Diabetes mellitus (DM) is a major non-communicable disease with an increasing prevalence. Undiagnosed DM is not uncommon and can lead to severe complications and mortality. Identifying high-risk individuals at an earlier disease stage, that is, pre-diabetes (pre-DM), is crucial in delaying progression. Existing risk models mainly rely on non-modifiable factors to predict only the DM risk, and few apply to Chinese people. This study aims to develop and validate a risk prediction function that incorporates modifiable lifestyle factors to detect DM and pre-DM in Chinese adults in primary care. METHODS AND ANALYSIS: A cross-sectional study to develop DM/Pre-DM risk prediction functions using data from the Hong Kong's Population Health Survey (PHS) 2014/2015 and a 12-month prospective study to validate the functions in case finding of individuals with DM/pre-DM. Data of 1857 Chinese adults without self-reported DM/Pre-DM will be extracted from the PHS 2014/2015 to develop DM/Pre-DM risk models using logistic regression and machine learning methods. 1014 Chinese adults without a known history of DM/Pre-DM will be recruited from public and private primary care clinics in Hong Kong. They will complete a questionnaire on relevant risk factors and blood tests on Oral Glucose Tolerance Test (OGTT) and haemoglobin A1C (HbA1c) on recruitment and, if the first blood test is negative, at 12 months. A positive case is DM/pre-DM defined by OGTT or HbA1c in any blood test. Area under receiver operating characteristic curve, sensitivity, specificity, positive predictive value and negative predictive value of the models in detecting DM/pre-DM will be calculated. ETHICS AND DISSEMINATION: Ethics approval has been received from The University of Hong Kong/Hong Kong Hospital Authority Hong Kong West Cluster (UW19-831) and Hong Kong Hospital Authority Kowloon Central/Kowloon East Cluster (REC(KC/KE)-21-0042/ER-3). The study results will be submitted for publication in a peer-reviewed journal. TRIAL REGISTRATION NUMBER: US ClinicalTrial.gov: NCT04881383; HKU clinical trials registry: HKUCTR-2808; Pre-results. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

The risk prediction function will be developed from a Chinese population-representative sample dataset to enhance its validity.

Different methods, including machine learning, will be used to increase the reliability and predictive power of the final prediction function.

The functions will be validated using an external prospective sample for validity and generalisability.

The risk prediction function incorporates lifestyle factors to improve validity and clinical application.

The lack of data on family history of diabetes mellitus in the Population Health Survey 2014/2025 may affect the accuracy of the risk prediction models.

Introduction

Diabetes mellitus (DM) is the second most common chronic non-communicable disease (NCD) and a significant public health problem. In 2017, it was estimated that 451 million adults worldwide had DM, a number that is anticipated to rise to 693 million by 2045.1 In China, the prevalence of DM has increased rapidly in the past two decades, with approximately 109.6 million Chinese adults (25.8% of all cases worldwide) currently living with the condition.2 Among the Chinese population, Hong Kong has one of the highest prevalence of DM.3 Hong Kong’s Population Health Survey (PHS) in 2014/2015 found a prevalence of 8.4% of DM among persons aged 15–84 years, more than half (4.5%) of which were previously unknown.4 Unpublished data from the PHS 2014/2015 showed a further 9.5% of persons aged 15–84 years had pre-diabetes (pre-DM) but were unaware of the problem before the survey.4

DM can result in severe complications, which lead to disabling morbidity and premature mortality. Several randomised controlled trials have found that lifestyle interventions (eg, diet, exercise) and pharmacological treatments are effective in preventing DM and its complications.5 6 However, it has been reported that 224 million adults (49.7% of all cases) worldwide are unaware that they have the condition,1 which is consistent with the finding of the Hong Kong PHS 2014/2015. DM can be present for 9–12 years before a diagnosis and is often only detected when patients present with complications.7 Hence, there is a need for earlier detection of DM so that appropriate interventions can be provided to prevent and/or delay progression to complications. It would be even more effective if individuals could be identified at the prediabetes stage when there may still be an opportunity to revert to normoglycaemia by lifestyle modifications.8 While DM satisfies all Wilson and Jungner’s criteria of screening,9 studies have shown that general population screening is not effective10 and the current recommendation is case finding targeting at high-risk individuals (persons aged≥45 years old or having DM risk factors).11 Indeed, a cost-effectiveness analysis reported that screening for DM and prediabetes was cost-saving among patients identified as being at high risk (eg, body mass index (BMI)>35 kg/m2, systolic blood pressure≥130 mm Hg or >55 years of age) when compared with no screening.12

To identify high-risk individuals more accurately, multivariate risk prediction models have been developed and incorporated into DM prevention programmes.13 Such models usually include sociodemographic factors (eg, age, sex), clinical factors (eg, family history of DM, gestational DM) or biomarkers (eg, BMI, blood pressure). However, most of these models were developed primarily in Caucasian populations and have not performed well among Chinese people.14–16 For example, the Prospective Cardiovascular Münster, Cambridge, San Antonia and Framingham models were found to have inferior discrimination in a cohort of Chinese people (area under the curve (AUC): 0.630, 0.580, 0.662 and 0.675, respectively).14 This can be due to ethnic differences as well as lifestyle and socioeconomic factors, calling for the need for population-specific risk prediction models.

Since 2009, several DM risk prediction models and scoring algorithms have been developed specifically for Chinese populations.3 17–22 The majority of these models and algorithms were developed and validated in Mainland China,17–21 with only three models developed for the Hong Kong Chinese population.3 17 22 The first Hong Kong model used self-reported factors and laboratory measurements to create a scoring algorithm.3 The cut-off score of ≥16/30 performed well in the development sample (AUC: 0.73) and two validation samples (AUC: 0.681 and 0.772).3 However, the model’s applicability to primary care patients may be limited as 70% of the subjects of the development and validation samples had known risk factors for DM, and laboratory tests are required. The second Hong Kong model was developed with data from 3357 asymptomatic non-diabetic professional drivers.17 Non-laboratory risk factors included age, BMI, family history of DM, regular physical activity (PA) and high blood pressure. Triglyceride was added to the laboratory-based scoring algorithm. The AUC for the non-laboratory-based and laboratory-based algorithms was 0.709 and 0.711, respectively. At the optimal cut-off score of ≥18, the sensitivity and specificity were 57.9% and 68.9% for the non-laboratory-based algorithm and 66.2% and 60.2% for the laboratory-based algorithm. The application of this risk prediction model is limited because the sample was predominately male (92.7%), and the accuracy was modest. The third Hong Kong model, the Non-invasive Diabetes Score (NDS), used only three non-invasive factors (ie, age, BMI and diagnosis of hypertension) to develop a risk score system.22 At a cut-off score of ≥28/50, the model showed good discrimination (AUC: 0.720) with an external validation sample, which consisted of mainland Chinese. Also, sensitivity and specificity were reported as 60.8% and 69.7%, respectively. Although the model did not include any lifestyle factors, it is worth noting that it was developed based on data from 1995 and validated using mainland Chinese data from 2007. Lifestyle behaviours factors, that is, dietary pattern and PA, could have changed over the years, which might affect the validity and applicability of the NDS.

Despite pre-DM being a crucial time to prevent DM progression, there are only a few models that include pre-DM and DM as a positive case. Often, such models lack prospective external validation and have modest predictive performance (AUC: 0.646).20 23 It is also noted that most factors included in existing models are non-modifiable (eg, family history of DM, gestational DM, age). There is a need for research to incorporate more lifestyle factors to improve risk prediction models’ predictive validity and impact.13 Lifestyle factors that may be associated with DM and pre-DM include PA level,17 dietary factors (eg, fibre, sugar or fat intake), alcohol consumption, smoking and sleep.18 19

This study aims to develop a new DM and pre-DM risk prediction function that incorporates traditional and modifiable lifestyle factors for the Hong Kong general Chinese population. We will apply the novel method of machine learning (ML) and the conventional logistic regression in model development to improve validity, reliability and predictive power. Further, a prospective study on individuals attending primary care clinics will be carried out to validate the models externally. Therefore, we hope the results of this study will enable opportunistic case finding of asymptomatic DM and pre-DM in primary care effectively so that early diagnosis and interventions can be given to prevent diabetic complications and mortality and morbidity from this common but silent NCD.

The study has three specific objectives: (1) to develop a risk prediction function using non-laboratory parameters to predict DM and pre-DM from the data of the Hong Kong’s PHS 2014/2015, (2) to develop a risk-scoring algorithm and determine the cut-off score for predicting DM and pre-DM and (3) to validate the risk prediction function and determine its sensitivity and specificity in predicting DM and pre-DM in Chinese adults in primary care. Our hypotheses are as follows: (1) the DM and pre-DM risk prediction function developed from the PHS 2014/2015 data has good discriminatory power with an area under the receiver operating characteristic (AUROC) curve of >0.7. (2) The DM and pre-DM risk prediction models developed by ML are more discriminatory and accurate than those developed by logistic regression. (3) The DM and pre-DM risk algorithm with the optimal cut-off score has a sensitivity of ≥75% in identifying incident cases of DM or pre-DM over 12 months.

Methods and analysis

Study design

This study consists of two parts. The first is a cross-sectional study to develop a risk prediction function for DM and pre-DM using data of 1857 subjects from the general population in Hong Kong collected by the PHS 2014/2015.4 24 The second part is a 12-month prospective study of 1014 Chinese adults (aged 18–84 years) attending public and private primary care clinics, to test the validity, sensitivity and specificity of the risk prediction function in case finding of people with DM and pre-DM.

Study population

Development study

We will include subjects who had participated in the PHS 2014/2015 and completed the health examination, including physical measurements (body height, weight, BMI, waist and hip circumference) and blood tests (fasting plasma glucose, haemoglobin A1C (HbA1c) and lipid profile). A population-representative sample of 12 022 people completed the PHS 2014/2015, and 2347 randomly selected persons aged 15–84 years (19.5%) participated in the health examination. Of the 2347 persons, we have identified 1857 subjects (male: 885; female: 972) aged 18–84 (mean age: 41.37 years) without self-reported doctor-diagnosed DM, hypertension, cardiovascular disease (coronary heart disease, stroke), cancer, renal disease or anaemia eligible for inclusion in the sample for the development of a risk prediction model of DM and pre-DM. Among the 1857 subjects, the prevalence of previously unknown but blood test confirmed DM was 3.77% (70 subjects) and pre-DM was 11.31% (210 subjects). The total prevalence of newly detected DM and pre-DM was therefore 15.08%.

Validation study

Patients attending private and public primary healthcare clinics will be recruited by doctors, research assistants and self-referral. Printed posters and leaflets will be placed in waiting areas and consultation rooms of participating clinics for distribution to attending patients who are also encouraged to refer their friends and family. We will purposefully sample subjects to ensure representation from both gender and people over and below 40 years. Consecutive sampling will be deployed to invite every eligible participant who meets all the inclusion criteria and none of any exclusion criteria to participate until we reach the required sample size. The inclusion and exclusion criteria are listed below.

Inclusion criteria:

Aged 18–84 years.

Chinese.

Can communicate in Chinese.

Consent to participate in the study.

Exclusion criteria:

Any history of doctor-diagnosed DM, high blood glucose, cardiovascular disease (coronary heart disease, stroke), cancer, chronic kidney disease or anaemia.

Inability to complete the survey or blood test because of sickness or cognitive impairment.

Do not give consent to the study.

Outcome measures

A positive case is DM or pre-DM defined by Oral Glucose Tolerance Test (OGTT) or HbA1c criteria in any one blood test. Case definitions of DM and pre-DM are based on the WHO, and the Hong Kong Reference Framework for Diabetes Care for Adults in Primary Care Settings.11 25 The DM case definition is as follows: (1) in OGTT, fasting glucose≥7 mmol/L or 2 hours post 75 g glucose≥11.1 mmol/L or (2) HbA1c≥6.5%. The pre-DM case definition is: (1) in OGTT, fasting glucose between 6.1 and 6.9 mmol/L or 2 hours post 75 g glucose between 7.8 and 11 mmol/L, or (2) HbA1c between 5.7% and 6.4%.

Primary outcome

The sensitivity of the risk prediction function in detecting DM and pre-DM in primary care.

Secondary outcomes

AUC, specificity, positive predictive value (PPV) and negative predictive value (NPV) of the risk prediction scoring algorithm in predicting DM and pre-DM in primary care.

Sample size calculation

A minimum sample size of 995 will be required from the PHS 2014/2015 data for the model development by multivariate logistic regression, applying the rule of thumb of at least 15 events per predictor, assuming 10 predictors to be included in the model and the prevalence of undiagnosed DM and pre-DM as 15.08%. Considering the data splitting at a two-to-one ratio for model development and internal validation, a total sample size of at least 1493 is therefore necessary.

The primary outcome is the sensitivity of the risk prediction function in detecting new cases of DM and pre-DM. The sample size calculation is based on a point prevalence of 15.08% of undiagnosed DM and pre-DM found in the PHS 2014/2015, and the utility of the new risk prediction function is expected to have a sensitivity of 75%. A sample of 710 subjects (107 with pre-DM or DM and 603 with normal glycaemic status) will be required to achieve the lower limit of the 95% CI greater than 0.6.26 We plan to recruit 1014 subjects, 50% from public and 50% from private primary care clinics, to allow for 30% attrition.

Data collection

Data on the relevant risk factors and results of the fasting blood glucose and HbA1c of the eligible subjects will be extracted from the database of the PHS 2014/2015 for model development. Risk factors (independent variables) of DM and pre-DM that have been reported in the literature and are readily available in primary care without the need of blood tests, including patient’s sociodemographic (age, gender, education, occupation), clinical parameters (BMI, systolic blood pressure, diastolic blood pressure, waist circumference, waist-to-hip ratio), and lifestyle (smoking, alcohol consumption, PA level, daily portions of fruit, vegetable and sugar-sweetened beverages, frequency of eating-out, sleep duration and quality) will be included. The Development study started in August 2020 and ended in February 2021.

Trained research assistants will screen and invite eligible subjects referred from public and private primary care clinics to participate in the study from April to December 2021. Subjects who agree will be asked to sign a written consent form (online supplemental appendix A) and complete a questionnaire on sociodemographics, personal and family history of medical conditions and lifestyle (including PA level, dietary factors (eg, fibre, sugar or fat intake), alcohol consumption, smoking and sleep). An investigation form will be given to each subject to attend a designated private laboratory, which is accredited by the National Association of Testing Authorities, Australia and Royal College of Pathologists of Australasia for compliance with the International Organization for Standardization 15189, for measurements (blood pressure, weight, height, waist and hip circumferences), and a blood test on OGTT, HbA1c, complete blood count and lipid profile within 3 months. The quality standards of methodology in the questionnaire survey, anthropometric measurements and laboratory investigations of the PHS 2014/2015 will be applied to subjects in the validation study. The OGTT and HbA1c results will be screened to identify cases of DM and pre-DM. Since anaemia may affect the validity of HbA1c, subjects with haemoglobin<10 g/dL will be excluded from the validation study. Subjects with abnormal results will be contacted for counselling or referral for further assessment or management as indicated. Subjects who have normal OGTT and HbA1c will be sent another investigation order form for a repeat blood test on OGTT and HbA1c 12 months from the recruitment date, and the results will be screened and followed as explained above. Data collection for the validation study is expected to end by December 2022. The prospective validation study flow chart is shown in figure 1.

Figure 1

Study flow diagram. CBC, complete blood count; DM, diabetes mellitus; FU, follow-up; HbA1c, haemoglobin A1C; OGTT, Oral Glucose Tolerance Test; Pre-DM, pre-diabetes mellitus.

Data analysis

Descriptive statistics will be used to calculate the incidence of DM and pre-DM, in total and respectively. The distribution of risk factors will be cross-tabulated by groups of DM, pre-DM and normal glycaemia for the development and validation samples, respectively. Unadjusted associations between the risk factors and glycaemic group categories will be assessed by analysis of variance (ANOVA) for continuous variables or χ2 test for categorical variables.

We will randomly split the 1857 eligible subjects of the PHS 2014/2015 into 2/3 as the derivation sample and 1/3 as the internal validation sample. We will use two methods to develop the prediction models using the data from the derivation sample. The first is the traditional multivariable logistic regressions of all relevant independent factors with a stepwise method to develop a risk prediction model for DM and pre-DM. If the main term of a risk factor is selected in the model, the quadratic term of the risk factor and its interaction term with age will be evaluated. The natural logarithm of the OR of each selected risk factor in the final model will be used as a coefficient (weight) in the prediction function. The risk equations for DM and pre-DM will be established by combining these weights with the logistic function. The second method is Extreme Gradient Boosting (XGB),27 a tree-based ensemble ML algorithm. XGB has been widely used in disease risk prediction tasks, showing better performance than other commonly used ML methods, such as neural networks and random forest.28–30 The loss function of the XGB model will be a cross-entropy error, which is defined as follows:

where is the observed event and is the predicted risk.

The hyperparameters of XGB, including booster parameters and tree structure parameters, will be determined using grid search based on fivefold cross-validation. To avoid overfitting, once the training loss is observed to be increasing for five iterations, the training process will be stopped. The Shapley Addictive Explanations method, which is based on Shapley value, will be used to evaluate the risk factors’ importance and to interpret the ML model. The most important risk factors will be selected using the wrapper feature selection method31 to build the final XGB model.

For both the logistic regression and ML models, ROC curves of predicted risk against observed events (DM and pre-DM) will be used to determine the cut-off value of the optimal trade-off between sensitivity and specificity by the Youden’s index.32

To validate the risk prediction models, each logistic regression and ML model will be applied to the data collected from subjects recruited prospectively from primary care clinics. An ROC curve of predicted risk against observed events (DM and pre-DM) will be used to calculate the AUROC curve. An AUROC of less than 0.7 indicates limited discriminating power, 0.7–0.8 is acceptable, and >0.8 suggests strong discrimination. Applying the risk threshold score to the validation samples, the sensitivity, specificity, PPV and NPV and likelihood ratios of the observed events of DM and pre-DM will be calculated. The area under the precision–recall curve and F score will be further used to measure the models’ discrimination as they are more reliable and recommended for imbalanced datasets.33

Calibration of the model’s ability to correctly estimate the absolute risks will be examined by Hosmer-Lemeshow test and calibration plots. The Hosmer-Lemeshow test measures the statistical difference between the distributions of the predicted probability and the observed event rate, where a p value higher than 0.05 indicates good model calibration. A calibration plot with scatters (observed incidence of an event by decile of predicted risk) along the 45° line indicates perfect calibration. All the validation will be carried out in the whole validation sample and different age/sex subgroups to strengthen the validity of the results.

STATA software V.13 (STATA Corp) and Python 3.5.4 will be used for data analyses and model development. Overall, 5% is used as the level of significance in all statistical tests.

Patient and public involvement

This research will not include patient involvement. Patients will not be invited to comment on the study design and will not be consulted to develop patient-relevant outcomes or interpret the results. Patients will also not be asked to contribute to the writing or editing of the future manuscript for readability or accuracy.

Discussion

Given the COVID-19 global pandemic, careful considerations were given to adjust the procedures of the prospective validation study so that it can be carried out smoothly while ensuring the safety of the study team and participating subjects. As the pandemic poses significant impacts on conducting clinical research, the safety of the involved is of paramount importance.34 For instance, instead of taking the non-laboratory measurements (ie, blood pressure and anthropometric measures) onsite at the recruiting clinic as we had initially planned, subjects will have all of their measurements taken when they attend the private laboratory for the blood test. This precautionary measure minimises the physical contact required and reduces the contact time between our research staff and subjects. In addition, we will send each subject a copy of the report with their physical measurements and laboratory tests electronically, and also by mail if a doctor referral letter is required. Additional measures (ie, one-to-one identity verification and password-encrypted reports) are incorporated into the procedures to ensure the safety of personal data. We hope such measures can enable more subjects to participate and alleviate the challenges in conducting clinical research during the pandemic.35

There are several important strengths of this study. First, the local validity of the prediction models should be high as it is developed using data from a population-representative sample (Hong Kong PHS 2014/2015). Second, the use of an external prospective sample for validation of the prediction models enhances the validity and generalisability. Third, using different methods, including ML, increases the validity, reliability and power of the final prediction function. The results of the study could enable opportunistic case finding of asymptomatic DM and pre-DM patients in primary care so that early diagnosis and interventions can be given to prevent diabetic complications and hospitalisation. There is also the potential to develop a Chinese DM risk prediction mobile application. Such an application could empower the public to monitor their DM and pre-DM risk, raise awareness, motivate healthy lifestyle and encourage more appropriate medical consultations.

In terms of limitations, data on family history of DM and history of gestational DM were not collected in the PHS 2014/2015 and, therefore, cannot be included in the development of the prediction models. There could be recall bias on the lifestyle factors collected via a questionnaire in the prospective validation study. Finally, the findings from Chinese people in Hong Kong may not apply to those in other parts of the world.

Data management and monitoring

Members of the research team from The University of Hong Kong will take full responsibility for the conduct of research staff and study participants to ensure protocol compliance, proper study management and timely completion of study procedures.

An external data monitoring committee is not deemed necessary for this study. Data will be monitored by the research team that includes several clinicians (ETYT, EYTY, WYC, WWKK, DVKC, KCBT and CLKL), a statistician (CKHW) and an artificial intelligence algorithm engineer (WD). Models will be monitored by the model developer (WD). The development study is a retrospective cohort study with no obvious risks. The validation study is considered low risk as subjects will be referred to receive medical care if abnormal results are found during any of the blood tests if clinically indicated. Data are available on reasonable request.

Collection and assessment of reported adverse events and other unintended effects of the study, or study conduct, will be performed continuously if such events arise. Queries identified will be resolved promptly by the research team. All unintended effects and adverse events will be reported every 6 months to the Institutional Review Board (IRB) of the University of Hong Kong and Queen Mary Hospital. Interim analyses will be reported to the IRB and funding body every 12 months. The principal investigator (CLKL) will oversee the interim analyses and any decisions to stop the study.

Ethics and dissemination

Ethics approval has been received from the IRB of The University of Hong Kong/Hong Kong Hospital Authority Hong Kong West Cluster (IRB reference number: UW19-831) and Research Ethics Committee of the Hong Kong Hospital Authority Kowloon Central/Kowloon East Cluster (IRB reference number: REC (KC/KE)-21-0042/ER-3) who have reviewed and approved the study procedures, ethics, subject information and consent, and subject safety. The trial results will be submitted for publication in a peer-reviewed journal.