Development and Validation of the Cluster Headache Screening Questionnaire.

BACKGROUND AND
PURPOSE: Cluster headache (CH) is frequently either not diagnosed or the diagnosis is delayed. We addressed this issue by developing the self-administered Cluster Headache Screening Questionnaire (CHSQ).
METHODS: Experts selected items from the diagnostic criteria of CH and the characteristics of migraine. The questionnaire was administered to first-visit headache patients at nine headache clinics. The finally developed CHSQ included items based on the differences in responses between CH and non-CH patients, and the accuracy and reliability of the scoring model were assessed.
RESULTS: Forty-two patients with CH, 207 migraineurs, 73 with tension-type headache, and 18 with primary stabbing headache were enrolled. The CHSQ item were scored as follows: 3 points for ipsilateral eye symptoms, agitation, and duration; 2 points for clustering patterns; and 1 point for the male sex, unilateral pain, disability, ipsilateral nasal symptoms, and frequency. The total score of the CHSQ ranged from 0 to 16. The mean score was higher in patients with CH than in non-CH patients (12.9 vs. 3.4, p<0.001). At a cutoff score of >8 points, the CHSQ had a sensitivity, specificity, positive predictive value, and negative predictive value of 95.2%, 96%, 76.9%, and 99.3%, respectively.
CONCLUSIONS: The CHSQ is a reliable screening tool for the rapid identification of CH.

INTRODUCTION

Cluster headache (CH) is characterized by severe recurrent unilateral head pain, ipsilateral cranial autonomic symptoms, and an impaired quality of life.123 Although little is known about the population-based epidemiology of CH, a meta-analysis found wide ranges for the lifetime and 1-year prevalence rates, of 56–381 per 100,000 and 3–150 per 100,000, respectively.4 Men are reportedly 4.3–7 times more likely to be affected than women, with a peak age range at onset of 20–31 years.56 Furthermore, CH has a major socioeconomic impact in general due to both the direct healthcare costs and the indirect costs caused by loss of working capacity during the active working period.7

Despite CH being a very severe type of headache with distinct clinical features, the condition is underdiagnosed and undertreated. Several studies have found that diagnosis delays longer than 3 years are common in patients with CH,8910 which may be due to this severe unilateral type of headache being misdiagnosed as migraine. The relatively low prevalence, low public awareness, and small number of medical personnel specialized in primary headache disorders further contribute to the diagnosis delay of CH. However, CH is not uncommon in headache clinics, and the established treatments for CH such as triptans, oxygen therapy, and occipital steroid injection demand early diagnosis and rapid therapeutic approaches.1112

Several self-administered screening questionnaires for identifying CH have been developed with the aim of expediting the diagnosis of CH.131415 Although the previously reported CH screening tools exhibited good validity and reliability, they have some limitations. Previous studies have validated these self-administered questionnaires in mixed cohorts of first-visit or previously diagnosed patients with CH or self-reported CH patients.1314 The inclusion of patients diagnosed with CH or self-reported CH may result in bias in the questionnaire responses; namely, patients who are aware of CH may provide more-positive responses to questions related to the characteristic symptoms of CH. Previous studies calculated the positive predictive value (PPV) and negative predictive value (NPV) based on comparisons of small numbers of patients with other headache disorders. Considering the relatively low prevalence of CH compared with migraine, screening tools specific for CH need to be developed and validated in a cohort of sufficient size with common headache disorders, for application in real-world situations such as a headache clinic or population studies.16

The objective of this study was to develop a simple and reliable screening tool to identify patients experiencing CH from among first-visit patients with a headache.

METHODS

Participants

The study participants were recruited from March to August 2017 at neurology clinics across nine hospitals in Korea. Subjects were eligible if they were first-visit headache patients, before completing the questionnaire, and were aged 1–65 years. Headache specialists examined the patients after they had answered the self-administered questionnaire. Each of the participating investigators made diagnoses according to the criteria of the International Classification of Headache Disorder, Third Edition, beta version (ICHD-3β).17 We included patients diagnosed with CH and probable CH. We additionally included patients who met the criteria of a definite diagnosis of migraine, tension-type headache (TTH), or primary stabbing headache (PSH), since the purpose of this study was to develop a simple screening tool for identifying CH from among first-visit primary headache patients. Patients with cognitive or psychologic impairment, secondary headache disorder, and other types of headache disorder were excluded. The goal was to recruit 300 patients, based on the minimum number of participants required for tool validation test statistics.18 Considering the low prevalence of CH, we planned to recruit at least 30 patients with CH so that they comprised ≥10% of the study population. The final study population comprised 340 patients; 42 patients with CH and 298 patients with non-CH. All of the study procedures were approved by the institutional review boards at each participating hospital, and the institutional review board allowed the requirement for informed consent to be waived considering that the questionnaire was administered prior to the diagnosis and there was no risk of infringing personal information (2016-396-I, KBSMC 2016-10-031).

Questionnaire development

To develop a tool for screening CH at the outpatient clinic, we performed an extensive literature review and convened several expert meetings. The expert group consisted of 12 neurologists with ≥10 years of clinical experience in treating patients with CH. The literature review and expert-group discussion resulted in seven items for screening CH being agreed upon, all of which were from the ICHD-3β criteria.17 The chosen items comprised two items for the duration of headache attack and unilateral pain, three items for the associated symptoms during a headache attack, and two items for the frequency of headache attacks and the duration of headache bout. We additionally included three items that are major characteristics of migraine (disability, nausea, and photophobia) to determine whether these items may be helpful in distinguishing between migraine and CH.19 The item regarding severe pain from the ICHD-3β criteria was initially included, but it was subsequently discarded since disability was considered a better expression. Respondents were instructed to answer the items assessing duration, unilateral pain, and associated symptoms of the headache on the following 4-point Likert scale: 1=never, 2=rarely, 3=sometimes, and 4=frequently. A dichotomous scale (yes=1 and no=0) was used for items for the frequency of headache attacks, clustering features of headache attacks, and characteristics of migraine. The Likert scale was later transformed into a dichotomous scale for the analysis (never, rarely, or sometimes vs. frequently).

The questionnaire was then pilot tested with 10 patients at the headache clinics at 2 Korean hospitals (Dongtan Sacred Heart Hospital and Kangbuk Samsung Hospital). Patients were asked to complete the survey and participated in a brief interview thereafter, in which we asked whether the instructions and wording of the questionnaire were clear, and whether the questionnaire included the symptoms they were experiencing. This revealed that all of the questionnaire items were well understood, and no specific issues were raised. The resulting questionnaire comprising 10 items was administered to the study participants, who were first-visit patients presenting with headache.

In addition to administering the questionnaire, we obtained demographic and clinical information including age, sex, clinically diagnosed type of headache (gold standard), and duration of headache from hospital medical records. The gold-standard diagnosis used to test the validity of the questionnaire was the diagnosis made by a headache specialist based on the ICHD-3β after the patient had completed the questionnaire.

Questionnaire validation and reliability testing

We calculated Cronbach's alpha to check the internal consistency and reliability of the questionnaire. We used Cronbach's alpha ≥0.70 as the standard for defining acceptable instrument reliability.

Considering the overlap between the characteristics of CH and other headaches (Supplementary Table 1 in the online-only Data Supplement), we decided to apply different weights to the scores for the various items. Items were scored 3 points if the difference in the prevalence of symptoms between CH and non-CH patients was >60%, 2 points if the difference was 50–60%, and 1 point (i.e., no weighting) if the difference was 30–49.9%.

The sensitivity and specificity values and the Youden Index (sensitivity+specificity−100)20 for various cutoff scores for the total scores of the final version of the questionnaire were then calculated. The corresponding Cronbach's alpha and the area under the curve (AUC) using the receiver operating characteristics test were also calculated.

The criterion validity was assessed by calculating the sensitivity and specificity for CH. Furthermore, the positive likelihood ratio, negative likelihood ratio, PPV, and NPV were calculated.21 Likelihood ratios can range from 0 to infinity, where a value of 1 indicates no diagnostic value, values greater than 1 indicate a higher probability of disease (positive likelihood ratios), and values below 1 indicate a lower probability of disease (negative likelihood ratios).22

Considering that the questionnaire applies to people who have a headache, the PPV and NPV were additionally calculated using the estimated prevalence of 0.2% in the general population and 1% in headache clinics.42324

Statistical analysis

Descriptive statistics were used for reporting the characteristics of participants and mean±SD values of each item. The independent-samples t-test was applied to continuous variables, and the chi-square test was used for categorical variables. All statistical analyses were performed using the STATA software package (version 14, STATA Corporation, College Station, TX, USA), and two-sided probability values of p<0.05 were considered significant.

RESULTS

The participants were aged 42.9±11.5 years, and 60.6% were female. Among the 304 participants, 12.4% were diagnosed with CH (episodic in 41 and chronic in 1), 60.9% were diagnosed with migraine (episodic in 141 and chronic in 66), 21.5% were diagnosed with TTH (episodic in 53 and chronic in 20), and 5.3% were diagnosed with PSH. The mean ages of the patients with CH, migraine, and other headaches (TTH or PSH) were 36.8, 41.9, and 48.1 years, respectively. Compared with non-CH patients, CH patients were more likely to be male (p<0.01) (Table 1).

Table 1

Demographics and headache diagnoses of the patients (n=340)

	CH (n=42)	Migraine (n=207)	TTH (n=73)	PSH (n=18)	p
Age, years	36.8±9.0	41.9±11.3	48.0±11.2	48.2±11.0	<0.001
Sex, male	34 (81.0)	47 (22.7)	40 (54.8)	13 (72.2)	<0.001

Data are n (%) or mean±SD values.

CH: cluster headache, PSH: primary stabbing headache, TTH: tension-type headache.

The differences in the responses to the questionnaire between CH and non-CH patients were >60% for item 1 (headache improves within 3 hours), item 2 (headache is accompanied by conjunctival injection and/or tearing on the headache side), and item 4 (headache is accompanied by a sense of restlessness or agitation). These three items were scored 3 points if the patient responded positively. The difference in the prevalence was 50% to 60% for item 9 (headache is repeated intensively for over a week) and was scored 2 points. Other items with differences of 30–49.9% were scored 1 point, and included items 3, 5, 6, and 10, and the male sex. Responses to item 7 (headache is accompanied by nausea or stomach sickness) and item 8 (photophobia) did not differ between patients with CH and migraine, and so these two items were discarded in the final scoring (Table 2). Consequently, the total score of the Cluster Headache Screening Questionnaire (CHSQ) ranged from 0 to 16 (Table 3). The CHSQ score was higher in CH patients than non-CH patients (12.9±3.0 vs. 3.4±2.5, p<0.001) (Fig. 1). Cronbach's alpha was 0.74, and the high accuracy of the scoring model was indicated by an AUC of 0.98 (Fig. 2).

Table 2

Distribution of positive responses to each item of the CHSQ according to the finally diagnosed headache types

Item	CH (n=42)	Migraine (n=207)	TTH or PSH (n=91)	p* (CH vs. Migraine)
1. Headache improves within 3 hours	35 (83.3)	21 (10.1)	18 (19.8)	<0.001
2. Headache is accompanied by conjunctival injection and/or tearing on the headache side	31 (73.8)	16 (7.7)	2 (2.2)	<0.001
3. Headache is accompanied by nasal congestion and/or rhinorrhea on the headache side	16 (38.1)	7 (3.4)	0	<0.001
4. Headache is accompanied by sense of restlessness or agitation	35 (83.3)	43 (20.8)	3 (3.3)	<0.001
5. Headache is unilateral on the right or left side	38 (90.5)	104 (50.2)	26 (28.6)	<0.001
6. Headache limits you from doing what you need to do	41 (97.6)	140 (67.6)	16 (17.6)	<0.001
7. Headache is accompanied by nausea and/or stomach sickness	31 (73.8)	167 (80.7)	28 (30.8)	0.315
8. Headache worsens when around light	26 (61.9)	118 (57.0)	14 (15.4)	0.558
9. Headache is repeated intensively for over a week	37 (88.1)	59 (28.5)	28 (30.8)	<0.001
10. Headache recurs more than three times weekly	36 (85.7)	96 (46.4)	49 (53.9)	<0.001

Data are n (%) values. Items answered on the Likert scale (items 1 to 5) were transformed into a dichotomous scale for the analysis (never, rarely, or sometimes vs. frequently). Detailed responses are presented in Supplementary Table 1 in the online-only Data Supplement.

*p values of all items among three groups and between CH and TTH or PSH were <0.001.

CH: cluster headache, CHSQ: Cluster Headache Screening Questionnaire, PSH: primary stabbing headache, TTH: tension-type headache.

Table 3

Final 9-item Cluster Headache Screening Questionnaire scoring sheet

	Never	Rarely	Sometimes	Frequently
Headache improves within 3 hours	0	0	0	3
Headache is accompanied by conjunctival injection and/or tearing on the headache side	0	0	0	3
Headache is accompanied by a sense of restlessness or agitation	0	0	0	3
Headache is unilateral on the right or left side	0	0	0	1
Headache is accompanied by nasal congestion and/or rhinorrhea on the headache side	0	0	0	1

Sometimes represents less than half the time; frequently represents half the time or more.

Fig. 1

Distribution of the CHSQ scores between CH and non-CH patients. CH: cluster headache, CHSQ: Cluster Headache Screening Questionnaire.

Fig. 2

ROC curve for the diagnostic accuracy of the Cluster Headache Screening Questionnaire scoring system. Area under ROC curve=0.98. ROC: receiver operating characteristic.

The most-appropriate cutoff score as calculated based on the Youden Index was 8 points; 76.9% of the participants with a score of >8 points were found to have CH. While 95.2% of CH patients scored >8 points, only 4% of non-CH patients achieved this. At the cutoff of >8 points, the sensitivity and specificity for CH were 95.2% and 96%, respectively, while the PPV and NPV were 76.9% and 99.3%, respectively. At a cutoff of >10 points, the risk score achieved sensitivity, specificity, PPV, and NPV values of 83.3%, 99.3%, 94.6%, and 97.7%, respectively (Table 4).

Table 4

Predictive parameters of the CHSQ for detecting CH

Predictive parameter	CHSQ score >8	CHSQ score >10
Sensitivity, %	95.2	83.3
Specificity, %	96.0	99.3
PPV, %	76.9	94.6
NPV, %	99.3	97.7
PLR	23.8	119.0
NLR	0.1	0.2
PPV*, %	4.6	32.6
NPV*, %	100.0	100.0

*Predicted PPV and NPV in the general population (estimated prevalence of CH in the general population of 0.2%).

CH: cluster headache, CHSQ: Cluster Headache Screening Questionnaire, NLR: negative likelihood ratio, NPV: negative predictive value, PLR: positive likelihood ratio, PPV: positive predictive value.

We estimated the PPV of the CHSQ tool when applying it to the prevalence of CH in the general population and headache clinics. Considering a presumed 1% prevalence of CH in headache clinics and 0.2% in the general population, the CHSQ achieved PPV and NPV values of 55.6% and 99.8% in headache clinics, and 32.6% PPV and 100% NPV in the general-population sample at a cutoff score of 10 points.

DISCUSSION

In this multicenter study, we developed a self-administered CHQS to rapidly identify individuals with CH from among various primary headache patients, based on the differences in the prevalence of symptoms between CH and non-CH patients. We found that the weighted scoring tool with nine questions for the male sex, attack duration, cranial autonomic symptoms, irritability, unilateral pain, disability, frequency, and clustering pattern is a valid and accurate tool for identifying CH among first-visit headache outpatients. When applied to first-visit headache patients, the sensitivity and specificity of the CHQS were 83.3% and 99.3%, respectively, at a cutoff score of 10 points, and 95.2% and 96% at a cutoff score of 8 points, relative to a CH diagnosis performed using the ICHD-3β.

CH is a severe primary headache disorder with characteristic clinical symptoms. Despite its well-defined clinical features and diagnostic criteria, the mean time to CH diagnosis is approximately 5 years; however, in one-third of cases this interval was found to be 12 years.8910 Even when patients were referred to a headache center, ≥1 year could elapse before a correct diagnosis.25 This indicates the need for a simple self-administered questionnaire tool for identifying CH. Several groups have previously developed different screening questionnaires for diagnosing CH. A three-item questionnaire comprising questions on unilaterality, attack duration, and ipsilateral conjunctival injection/lacrimation achieved a sensitivity of 78% and a specificity of 100% in 96 patients with primary headache disorders, including 37 with CH.14 Although this three-item questionnaire was reported to be very simple, some migraine patients with unilateral autonomic symptoms can be misclassified as CH with a yes/no question about autonomic symptoms.2627 A 16-item self-administered questionnaire was validated in 71 patients with primary headache disorders, including 30 patients with CH, which suggested an 8-question cluster was the best discriminatory tool, while applying 16 items would be too time-consuming.15 Furthermore, the Web-based Leiden University Cluster Headache Analysis program questionnaire was designed to diagnose CH among self-reported CH patients who fulfilled the inclusion screening questionnaire.13 Patients' awareness of CH symptoms may affect their responses to specific questionnaire items, and so we excluded previously diagnosed and self-reported CH patients from the present study.

While trigeminal autonomic symptoms are major features of CH, mild infrequent cranial autonomic symptoms may be present during migraine attacks.28 A reliable screening tool should therefore have a high power for discriminating between CH and migraine. The present study showed that 32.7% of migraine patients reported lacrimation and/or injection during attacks (combining rare, occasional, and frequent responses). This result is similar to prevalence rates of 26.9–45% for autonomic symptoms in previous epidemiologic and clinic-based studies.272829 However, regarding the frequency, only 7.6% of migraine patients reported frequent autonomic symptoms, compared with 73.8% of CH patients who reported frequent symptoms in the relevant item of the present questionnaire. Therefore, although autonomic symptoms are relatively prevalent in migraine patients, they could represent a powerful discriminating factor when using a Likert scale instead of a binary (yes/no) response option.

The responses to the questionnaire regarding migrainelike features also needed to be addressed, because disability, nausea, stomach sickness, and photophobia are frequently reported by patients with CH.930 Actually, disability was more common in patients with CH than in migraineurs (97.6% vs. 67.6%), and other features (nausea and photophobia) did not differ between the two groups. Therefore, as expected, the present study demonstrated that these migraine-like features might not be useful for differentiating CH from migraine.

CH is more prevalent in men than women. Therefore, male patients were scored 1 point in the CHSQ scoring tool. A previous report also suggested that the male sex might be valid screening option for CH.13 Although a decreasing trend in the male/female ratio over time was suggested, the male-to-female ratio is still high, especially in Asian studies.624 Furthermore, considering the female predominance of migraine and other primary headache disorders, scoring for the male sex may contribute to the ability to discriminate between CH and migraine patients.

Epidemiologic studies have found clear discrepancies between CH diagnoses made by questionnaires and physicians. Among a total of 182 subjects diagnosed with suspected CH by the screening questionnaire, 4 cases (2.2%) were confirmed as CH and the remaining were finally diagnosed as migraine with trigeminal autonomic symptoms in a German population study.16 Other population-based studies have also found low PPVs (3.1–12.6%) for screening questionnaires in diagnosing CH.3132 A screening method with higher validity is therefore required in epidemiologic studies of CH. When applied to the prevalence of CH in the general population (0.2%), the estimated PPV and NPV were 32.6% and 100%, respectively, for a cutoff of 10 points. These values suggest that it might be useful to apply the CHSQ in epidemiologic studies.

The merit of the CHSQ tool is that the cutoff score for diagnosing CH is flexible and can be adjusted depending on the characteristics of the population (e.g., specialized headache clinic, primary care clinic, or general population-based study). Although we speculated an optimal cutoff score of 8 points for diagnosing CH in headache clinics, the PPV of the tool can be improved by using a cutoff score of 10 points. Conversely, researchers using this tool in population-based studies might choose a lower cutoff score in order to increase the sensitivity for screening purposes. Another merit of the CHSQ is its short length. The CHSQ is composed of eight items (plus a score for the male sex) and takes <5 min to complete, and so it is expected to be easily applicable in general-population studies and achieve high response rates. Unlike previous studies, this study included patients with PSH, which are frequently encountered in headache clinics.33

The present study was subject to several limitations. First, although this screening questionnaire was tested in first-visit headache patients before a diagnosis was performed, all participants were enrolled in the neurology department of a secondary or tertiary hospital, which may have resulted in selection bias of the enrolled patients. Therefore, the CHSQ should be further validated in a primary-care setting with a larger population. Second, we enrolled first-visit headache patients aged 18–65 years among the Korean population and only one patient with chronic CH was included. This tool might therefore not be generalizable to screening chronic CH and so should be validated in other language. Finally, the gold standard of this study was a clinical diagnosis based upon the criteria in the previous ICHD-3β version. The main changes in the current ICHD-3 criteria were deleting recently added associated symptoms (fullness in the ear, forehead and facial flushing) and extending the maximum remission period of chronic CH up to 3 months. Although the presence of ear fullness and facial flushing have been reported to not influence the diagnosis of CH, the CHSQ should still be validated with the current ICHD-3 criteria.34

In conclusion, we have developed a highly reliable and simple eight-item questionnaire plus a score for the male sex that can reliably identify patients with CH among first-visit headache patients. The application of the CHSQ tool will facilitate the early identification of CH in clinics. Further validation is warranted in other clinical settings and different populations.