Novel observational study protocol to develop a prediction model that identifies patients with Graves' ophthalmopathy insensitive to intravenous glucocorticoids pulse therapy.

INTRODUCTION: Intravenous glucocorticoids pulse therapy is the first-line treatment for moderate-to-severe and active Graves' ophthalmopathy, with a large proportion of patients having poor efficacy and exposed to the risk of glucocorticoids adverse effects. We introduce a novel protocol to develop a prediction model designed to identify patients with Graves' ophthalmopathy who are not likely to benefit from intravenous glucocorticoids pulse therapy before administration, so that these patients can advance the time to receive appropriate treatment. Existing prediction models for prognosis of Graves' ophthalmopathy have usually focused on traditional clinical indicators without adequate consideration of orbital soft tissue changes. Our protocol for model development will address this limitation by using artificial intelligence models to quantify facial morphological changes. METHODS AND ANALYSIS: This study is a single-centre, prospective and observational study. A sample size of 278 patients with moderate-to-severe and active Graves' ophthalmopathy will be prospectively recruited at ophthalmology clinic of Shanghai Ninth People's Hospital to collect clinical and artificial intelligence model's baseline data as potential variables to develop the prediction model. They will receive 12-week intravenous glucocorticoids pulse therapy according to the 2021 European Group on Graves' Orbitopathy treatment guideline. After standard medication course and following 12-week observation, patients will be evaluated for the effectiveness of treatment in our ophthalmology clinic and divided into glucocorticoids-sensitive and glucocorticoids-insensitive groups. The model will be developed by means of multivariate logistic regression to select the best variables for the prediction of glucocorticoids treatment efficacy before administration. The result of the study will provide evidence for the use of a prediction model to personalise treatment options for patients with moderate-to-severe and active Graves' ophthalmopathy. ETHICS AND DISSEMINATION: The study received approval from the Ethics Committee of Shanghai Ninth People's Hospital (ethical approval number: SH9H-2020-T211-1. Findings will be disseminated via peer-reviewed publications and conference presentations. TRIAL REGISTRATION NUMBER: ChiCTR2000036584 (Pre-results). © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This study is prospectively conducted to collect various clinical indicators as comprehensively as possible, which provides a more accurate picture of patient status.

We propose to use artificial intelligence (AI) algorithms to give quantitative representation of morphological changes, which allows rapid, objective and repeatable evaluation of orbital soft tissue.

We aim to incorporate the positive prediction values from four settled AI models as ‘examination results’, along with other traditional clinical indicators, into the prediction model development, providing a new way of integrating AI models with clinical research.

This study is single centre study conducted with a relatively homogeneous population (race/ethnicity), which may limit the application of the developed prediction model to other populations.

Structured recording of medical history information and standardised collection of multiple indicators may add to the complexity and repetitive work of the study.

Introduction

Graves’ ophthalmopathy (GO), thyroid eye disease or thyroid associated orbitopathy (TAO) is an autoimmune inflammation of the orbital tissues and the most common extra-thyroid symptom of Graves’ disease.1 It is the most prevalent orbital disease, causing malformation and blindness,2 3 with approximately 10 million patients throughout China. GO has a complex aetiology, diverse manifestations and many therapeutic options. Glucocorticoids (GCs), immunosuppressants, radiotherapy, monoclonal antibodies and surgery are all available for the treatment or improvement of GO symptoms.4 Current clinical researches on orbital disease conducted internationally focus on functional MRI imaging of GO,5 diagnosis of optic neuropathy6 and computer navigation for orbital surgery,7 which has contributed to improving the accuracy of diagnosis and efficiency of treatment.8 The proportion of patients with moderate-to-severe and active GO is large, whose conditions are fast progressing and most valuable for treatment. International TAO treatment guidelines8 (the Guidelines of the European Thyroid Association and the 2021 European Group on Graves’ Orbitopathy (EUGOGO 2021)) stipulate that high-dose systemic GCs are the first-line treatment for moderate-to-severe and active GO.8–10 Intravenously administered GCs are more effective and better tolerated than oral GCs.11–15 Therefore, GCs are preferentially administered intravenously to treat moderate-to-severe and active GO. However, intravenous GCs effectiveness is greatly limited by individual variability.16 Only about 60% of patients are sensitive to GC therapy, with the rest 40% having poor efficacy, which delays the best treatment time and brings about GCs side effects, severely limiting the overall treatment level of the disease.17 Adverse events related to intravenous GCs treatment are not rare and may be severe even at the recommended dose.10 18 If GC-insensitive patients can be screened before administration of GCs and receive second-line treatment timely, it can greatly improve patients’ prognosis and save economic and social costs. The establishment of a prognostic model for GO is, therefore, of great significance.

Patients with GO often present with ‘proptosis’ and ‘change in appearance’. Their orbital soft tissue anatomy is significantly altered, resulting in unique facial features.17 19 20 However, traditional clinical indicators cannot adequately depict imaging, appearance and facial structural changes. Soft tissue morphological factors are not sufficiently considered, resulting in a limited dimension of patient portraits. Therefore, the current statistical models based on clinical or biological indicators are unsatisfactory for screening GC-insensitive patients before treatment. There is a lack of effective model for predicting the prognosis of GO.

Research aim

We aim to develop a multivariate prediction model that identifies patients with GO who cannot benefit from intravenous GCs pulse therapy. We propose to integrate the positive predictive values (PPVs) output by the four artificial intelligence (AI) models built in the team’s previous work as ‘examination results’, along with other traditional clinical indicators, into a novel prediction model, enriching the imaging details, patient appearance and facial morphological changes depiction that are usually ignored in the traditional clinical data. Then, we can set up a multivariate model to predict the efficacy of GCs therapy in patients with GO before administration and assist doctors in better treatment decisions.

Methods

Feasibility

Ophthalmology Department of Shanghai Ninth People’s Hospital afflicted to Shanghai Jiaotong University School of Medicine has a reliable source of patients with GO. In the preliminary research, our team has achieved ocular structure recognition, parameter labelling and automatic measurement by using multiple networks, laying the technical foundation for accurate automatic labelling of 2D images of the eye. Various aspects of automated processing and recognition of orbital image data have been explored, including intelligent recognition and extraction of orbital regions in CT, and automatic correction of orbital CT. In these studies, we have systematically optimised the orbital image data, effectively front loading and unifying the data cleaning process that used to take a lot of time to complete. Therefore, we can control the noise that may be encountered during the study. Besides, the developed series of tools are not confined to specific disease research purposes; they are universal for disease species instead and have low needs for image orientation and angle. And the tools can run automatically without manual operation. Thus, all types of orbital imaging studies can be directly conducted on this basis without the need for extensive pre-calibration of coordinate systems and error calculations, raising research efficiency.21

In the previous study, our team retrospectively collected data from 335 patients with moderate-to-severe and active GO. Using facial key points scanning technique and AI algorithms, we set up 4 AI models, all of which achieved an area under curve (AUC) of 0.8 or higher. Two articles on these AI models have been published by our team.5 22 The settled AI models are a powerful complement to the imaging details, patient appearance and depiction of facial morphological changes. Combining them with traditional clinical data can enrich the patient portrait dimension and is expected to predict GCs efficacy before treatment, improving doctors’ decision-making level and patients’ prognosis. This prospective study was approved by the Ethics Committee of Shanghai Ninth People’s Hospital and registered at Chinese Clinical Trial Register website. We plan to enrol eligible patients from 15 June 2021 to 15 February 2022.

Study design

This project is a single-centre, prospective and observational study without additional diagnostic and therapeutic interventions (see figure 1). No randomisation is involved, with only statistician and evaluator blinded. Patients who meet the inclusion criteria will be prospectively enrolled to receive intravenous GCs pulse therapy according to the EUGOGO 2021 Guideline8 at Shanghai Ninth People’s Hospital, Ophthalmology Department. Before treatment, PPV output by the four built AI models, along with traditional clinical data will be recorded as potential variables. At the end of study, the patients will be differentiated into GCs-sensitive and GCs-insensitive groups based on GCs efficacy assessment. A GCs efficacy prediction model will be developed by using multivariate logistic regression analysis to achieve pretreatment screening of GC-insensitive patients.

Figure 1

Study design. AUC, area under curve; EUGOGO 2021, the 2021 European Group on Graves’ Orbitopathy; GCs, glucocorticoids; PPV, positive predictive value.

Patient and public involvement statement

It was not appropriate to involve patients or the public in the design, or conduct, or reporting, or dissemination plans of our research. As the study is observational without additional therapeutic interventions. All participants are diagnosed and receive their treatment according to the internationally recognised guideline. The statistical analysis and outcome production are done with multivariable logistic regression.

Patients recruitment and eligibility

Inclusion criteria

Patients between 18 years and 60 years of age who are diagnosed as moderate-to-severe and active GO (Clinical Activity Score (CAS) ≥3) according to the EUGOGO 2021 Guideline8 and have not been treated are eligible to participate. All the patients are to receive intravenous GCs pulse therapy. Patients who meet all enrolment criteria and provide informed consent will be admitted into the study.

Exclusion criteria

Patients with severe cardiac, hepatic and renal insufficiency, diabetes mellitus, uncontrolled hypertension, psychiatric diseases, autoimmune diseases such as systemic lupus erythematosus and rheumatoid arthritis, tuberculosis, HIV infection or AIDS, and GC allergy will be excluded. Pregnant women, those who are breast feeding or those who have received hepatitis vaccination within 1 month before enrolment, will be excluded. More exclusion criteria are: having received systemic immunotherapy (oral or intravenous GCs, various types of immunosuppressants and GO-related targeted drugs), having received orbital radiotherapy or eye surgery before enrolment and having received dysthyroid optic neuropathy (DON). This process will exclude those patients who are not recommended to receive intravenous GCs treatment.

Baseline data collection

All the data collection will be done at the beginning of the study before treatment by a specific experienced researcher to minimise possible systemic errors. We will use the MedCohort system of the Shanghai Ninth People’s Hospital for data entry, so we can ensure the standardisation and safety of clinical data and improve research data utilisation efficiency.

Traditional clinical data are collected before treatment. The specific items of every indicator are listed in table 1. The PPV output by the four settled AI models (table 2) is collected along with traditional clinical data before treatment. The four AI models focus on different sites and avoid functional duplications in principle, depicting patient portraits from different dimensions with good complementary effects.

Table 1

Traditional clinical information collected before intravenous GC treatment

Demographic	History	Laboratory indicators
Gender	History of thyroid diseaseDuration of GO	Liver function
Age	Smoking index	Kidney function
Height	Medication history for thyroid disease	Thyroid function*
Weight	Medication history for ophthalmology
Eye examination†	Imaging	Systemic condition
Exophthalmos‡	Optic nerve of orbital apex§	Chest X-ray¶
Eye muscle duction**		ECG††
Eyelid aperture‡‡
Corrected visual acuity§§
Intraocular pressure¶¶
CAS***

*Thyroid function examination items include free triiodothyronine, free thyroxine, thyroid stimulating hormone, thyroglobulin, thyroid receptor autoantibodies, thyroid peroxidase autoantibodies and thyroglobulin autoantibodies.

†For patients whose both eyes meet the inclusion criteria, the eye examination will be carried out on both eyes to assess the treatment efficacy according to EUGOGO 20218 guideline.

‡The measurement will be repeated three times with the Hertel tonometer by one specialised researcher. Exophthalmos of the research eye and orbital distance will be recorded as millimetres.

§Determination of whether the orbital apical optic nerve is compressed by the extraocular muscles.

¶Preliminary determination of the presence of active tuberculosis.

**The values of eye muscle ductions will be measured in degrees in the four main orthogonal gaze directions (upgaze, abduction, downgaze and adduction) with a Förster perimeter arc by one specialised researcher repeatedly for three times.32

††Preliminary determination of the presence of serious cardiac diseases such as ventricular premature, atrioventricular block of degree II or above.

‡‡Eyelid aperture is measured as the distance from the upper lid margin to the lower lid margin through the middle of pupil when the patient naturally gazes forward. Eyelid aperture will be measured by one specialised researcher repeatedly for three times and recorded as millimetres.

§§The best corrected visual acuity will be measured using the international standard visual acuity chart.

¶¶The intraocular pressure will be measured by non-contact tonometer.

***There are seven items in CAS scores, with one point for each: eyelid congestion, moderate or severe eyelid oedema, moderate or severe conjunctival congestion, conjunctival oedema, spontaneous retrobulbar pain, pain on gaze or eye movement and swelling of the lacrimal caruncle.33 The CAS will be evaluated by one specialised researcher according to the standard in EUGOGO 20218 guideline.

CAS, Clinical Activity Score; EUGOGO, European Group on Graves’ Orbitopathy; GC, glucocorticoid; GO, Graves’ Ophthalmopathy.

Table 2

Model data collected before intravenous GC treatment

Model A	Model B
A convolutional neural network model for GO diagnosis based on CT, with AUC=0.91922	A convolutional neural network model for GO facial discrimination based on facial photographs, with AUC=0.924
Model C	Model D
A convolutional neural network model for GO activity discrimination based on MRI, with AUC=0.9615	A machine learning model for eyelid surgery protocol decision based on facial key point features, with AUC=0.83334 35

AUC, area under curve; GO, Graves’ Ophthalmopathy.

Medication use

All participants in this study will be treated with first-line therapy as prescribed by the GO treatment guidelines (EUGOGO 2021)8: intravenous GCs pulse regimen: 500 mg methylprednisolone weekly for 6 weeks and then changed to 250 mg weekly for 6 weeks (12 weeks in total with cumulative dose of 4.5 g). To prevent GCs side effects during the treatment, 40 mg omeprazole will be given intravenously weekly. Calcium, vitamin D and gastric mucosal protective drugs will be routinely applied in conjunction with GCs.

Follow-up and effectiveness assessment

The overall follow-up time is 24 weeks composed of 12-week treatment and another 12-week observation due to the lag in GCs treatment efficacy. After patient enrolment, the patients will be first followed-up before treatment to collect all the necessary baseline data as potential variables for the development of the prediction model. The second follow-up time is 12 weeks after completing GCs therapy. Patients will be evaluated on treatment effectiveness criteria of EUGOGO 20218 guideline. It is composed of entirely objective measures: ≥2 mm reduction of lid aperture, ≥1 point reduction in 5-item CAS (excluding subjective, patient-reported spontaneous or gaze-evoked pain), ≥2 mm reduction in exophthalmos and ≥8° increase of eye muscle duction. Improvement in ≥2 features in one eye without deterioration in the other eye might be considered a positive response to treatment.8 Throughout the follow-up, researchers will closely monitor GCs side effects and take timely symptomatic treatment if necessary.

Prediction model development

The PPVs output by the four settled original AI models (reflecting imaging, cosmetic and facial structural changes, respectively) before GCs treatment will be included as variables of the prediction model, along with the traditional clinical indicators (demographic indicators, history, laboratory indicators, eye examination, CAS, imaging and general condition). At the end of the study, the changes in exophthalmos, CAS, diplopia score, oculomotor score and compressive optic neuropathy judgement will be recorded, based on which the treatment effectiveness will be discriminated. Those who are insensitive to GCs treatment will be used as study group matched with control group of GCs-sensitive patients. The correlation between every potential variable and the prognosis of patients will be analysed by multivariate logistic regression analysis. Akaike information criterion, R2 and C-index will be used to screen the variables to obtain the best prediction model. The receiver operator characteristic curve will be established to select the best cut-off value of the model. The Hosmer-Lemeshow goodness of fit test will be used to evaluate the calibration of the prediction model. The results of the prediction model will be graphically presented using column line plots.

Sample size

According to the logistic regression modelling using the events per variable (EPV) method, the usual EPV value of 10 will be used, which means that the number of events with fewer occurrences in the dependent variable is 10 times the number of independent variables. Combining the above-mentioned AI model PPVs and the traditional clinical indicators, a total of 10 independent variables are expected to be retained in this study. It is also a conservative approach to calculate the sample size with 10 independent variables in the logistic regression model. Therefore, based on the EPV method to estimate the sample size, 100 (10×10) patients need to be recruited in the GCs-insensitive group. According to the previous literature, the proportion of GCs-insensitive patients is about 40%, so the total sample size will be 250 (100/0.4) cases. Considering the 10% loss of follow-up rate, the number of patients to be recruited is 278.

Outcome

The primary outcome of the study will be the AUC of the developed prediction model. The rest indicators of the model, including sensitivity, specificity, PPV, negative predictive value, positive likelihood ratio, negative likelihood ratio and Jorden index, will be the secondary outcome of the study.

Data analysis plan

Statistical analyses will be performed using R software V.4.0.2. Efficacy tests will all be performed using a two-sided test, and p values ≤0.05 will be considered statistically significant for the differences tested.

All paper versions of the original materials will be photographed and saved in an encrypted database. All electronic data will be stored in the electronic medical records of the Shanghai Ninth People’s Hospital.

Discussion

Current studies on GO treatment outcome prediction cover clinical examination and genomics data, but they generally have limitations such as small sample size, few variables of interest and interactive factors, which limits the utility of prognostic judgement models and the effective development of personalised treatment based on them.23–25 In addition, genomics research has the limitations of being invasive and expensive. And its increasingly large data volume poses challenges of manual management and results interpretation, reducing its scalability and reproducibility in clinical practice.26 Current hot spots of interest in the study of GO prognosis-related factors include internal features, that is, alterations in genomic, proteomic, blood tests and tear-related features; imaging features, that is, specific measurements of CT and MRI of the orbit; and external features, that is, observation of congestion and oedema of the eyelids, conjunctiva and cornea,27–29 with inadequate consideration of soft tissue morphological factors. The research value of midface morphological data (especially soft tissues) is underestimated and there is a lack of respective methods for measurement and quantitative representation. Traditionally, the determination of morphological changes like swelling is based on the subjective determination of doctors with clinical experience, which is unstable. This may be a main obstacle in the development of GO treatment outcome prediction models.17 19 With the announcement of the Large Scale Facial Model (LSFM) algorithm, face recognition technology is becoming increasingly mature. With up to 2000 eye vertices in the LSFM, the soft tissue morphology of the eye can be represented without omission by three-dimensional coordinates of up to 2000 variables and the positional relationships between them. This allows rapid, objective, repeatable and quantitative evaluation of eye tissue, which makes it possible for quantitative representation of non-quantitative descriptions such as morphological changes, swelling and swelling subsidence. The quantitative description of soft tissue morphology makes a more precise observation possible, but it is not equivalent to the endpoint indicator of clinical studies, nor can it be used to directly describe the disease status of GO. In order to establish a clear relationship between the quantitative observation and the endpoint indicators, traditional statistical methods such as clustering methods need to be introduced to analyse the characteristics of the data itself, so as to objectively quality control the labelling process of key soft tissue information.30 Therefore, in the study of intelligent models, our team proposed to use AI algorithms to participate in the modelling process by extracting deep features and outputting a certain quantitative expectation value. In this study, we aim to incorporate the PPVs from the four settled AI models into the prediction model as ‘examination results’ along with other traditional clinical indicators, as described above, to provide a more accurate and comprehensive picture of patient status.

Currently, there is a lack of efficient models for predicting the effectiveness of GCs therapy in patients with GO. In the pre-study, we developed a logistic regression prognostic judgement model based on traditional clinical indicators in 335 patients. Exophthalmos, imaging judgement of whether the extraocular muscles compress the orbital apical optic nerve, eyelid swelling, conjunctival congestion, smoking index and gender were the six indicators retained in the prediction model. But the AUC was only 0.63, which made it difficult to be applied in clinical practice and addressed the need to develop a novel modified prediction model. Therefore, we plan to prospectively recruit patients with moderate-to-severe and active GO who are to receive intravenous GCs pulse therapy so that we can record various clinical indicators as comprehensively as possible. The established model will be expected to predict the efficacy of intravenous GCs pulse therapy in patients with moderate-to-severe and active GO before treatment, to screen GCs-insensitive individuals and to assist in the decision of GCs pulse regimen to avoid ineffective treatment. Based on this, the development of ‘new first-line therapy’ for GCs-insensitive patients is possible, advancing the personalised treatment of GO.

The innovation of this study is that when multiple AI models are used to conduct clinical research, each AI model is considered as a new objective test. The PPV for a specific subject is included in the clinical prediction model as a ‘test result’, and its application value is determined by the statistical logic of the prediction model itself rather than artificially predetermined. This provides a new way of integrating AI models with clinical research, by allowing the value of AI models to be determined objectively. Medical researchers without AI background knowledge can directly use the model sufficiently. As for limits, this study is a single-centre study conducted with a relatively homogeneous population (race/ethnicity), which may limit the application of the developed prediction model to other populations since the presentations of GO are different in Asian and Western populations due to anatomical, immunological, genetic, socioeconomic and environmental variability.31 In the baseline data collection process, structured recording of medical history information and standardised collection of multiple indicators lay a solid foundation for the following statistics processing and analysis, but also add to the complexity and repetitive work of the study. In addition, after establishing the prediction model, new patients need to be recruited to conduct diagnostic tests with 5% error and 95% confidence level to verify the reported sensitivity and specificity of the model, so as to assess the clinical application value of it, which poses high requirements for the treatment management and data management work of the project.

Ethics and dissemination

The study received approval from the Ethics Committee of Shanghai Ninth People’s Hospital (ethical approval number: SH9H-2020-T211-1). Findings will be disseminated via peer-reviewed publications and conference presentations.