ESSENCE-Q - a first clinical validation study of a new screening questionnaire for young children with suspected neurodevelopmental problems in south Japan.

BACKGROUND: Early identification of autism spectrum disorder, intellectual developmental disorder, attention-deficit/hyperactivity disorder, and other neurodevelopmental disorders/problems is crucial, yet diagnosis is often delayed for years under the often misguided "wait-and-see" paradigm. The early symptomatic syndromes eliciting neurodevelopmental clinical examinations-questionnaire (ESSENCE-Q) is a brief (12-item) screening questionnaire developed specifically for the purpose of speeding up the identification process of a wide variety of neurodevelopmental problems. The aims were to 1) estimate the reliability of the ESSENCE-Q, 2) evaluate the clinical cutoff levels suggested by the author of the ESSENCE-Q, and 3) propose optimal cutoff levels based on receiver operating characteristic analysis.
METHODS: The ESSENCE-Q was used for 1 year by a psychiatrist in Kochi, Japan, assessing children under the age of 6 years referred for developmental problems. The children were also clinically assessed with regard to whether or not they met criteria for a developmental disorder (diagnosis positive and diagnosis negative groups). We contrasted the results of the ESSENCE-Q and those of clinical diagnostic assessments in 130 cases.
RESULTS: Cronbach's alpha was 0.82, sensitivity was 0.94 (95% confidence interval [CI]: [0.88, 0.98]), and specificity 0.53 (95% CI: [0.28, 0.77]), which are reasonable psychometrics for a first-step screening tool. Based on receiver operating characteristic analysis, we recommended an optimal cutoff level of yes ≥2 or maybe/a little ≥3 on the ESSENCE-Q (0.87 (95% CI: [0.79, 0.92]) sensitivity and 0.77 (95% CI: [0.50, 0.93]) specificity). CONCLUSION AND IMPLICATION: The ESSENCE-Q can be a good instrument for use as a screening tool for aiding in the process of early identification of neurodevelopmental disorders in clinical settings. To establish the broader validity and reliability of the ESSENCE-Q, case-control studies and general population studies of children in different age groups are needed.

Introduction

ESSENCE is an acronym for early symptomatic syndromes eliciting neurodevelopmental clinical examinations. The concept was launched by Gillberg in 2010.1 ESSENCE is not a diagnostic term, but a concept that covers different kinds of early onset of developmental problems and disorders in children. Some of the most common disorders subsumed under ESSENCE are autism spectrum disorder (ASD), attention-deficit/hyperactivity disorder (ADHD), oppositional defiant disorder (ODD), tic disorder, developmental coordination disorder (DCD), speech and language disorder (SLD), and intellectual disability/intellectual developmental disorder (IDD). ESSENCE also covers difficulties/problems not meeting criteria for a certain disorder, including borderline intellectual functioning and other nonspecific learning difficulties. The concept further implies the almost universal coexistence of such disorders and problems, and that these coexistences are the rule rather than the exception.2 This clinical reality was already mentioned in the 1970s and early 1980s, in papers discussing results suggesting that deficits in attention, motor control and perception, currently equivalent to ADHD with DCD, were generally associated with social, language, and behavioral impairments; that is, consistent with ASD as conceptualized today.3,4

The realization that neurodevelopmental disorders share symptoms across disorders has come to be largely ignored because of a preference for categorical “diagnostic boxes” with mutually exclusive diagnostic criteria. ESSENCE was launched with a view to alert clinicians and researchers to the complexity and overlap of neurodevelopmental problems with early onset, that is, before age 3 (−5) years. If a child before this age has one or more (usually several) major problems, lasting more than several months, in the following 12 fields: general development, communication/language, social interrelatedness, perception, motor coordination, attention/“listening”, activity, “behavior”, mood, feeding, sleep, and/or episodes/absences in the first years of childhood, this should be seen as a red flag for possible ASD, ADHD, ODD, DCD, tic disorder, SLD, and/or IDD. If unrecognized and not intervened, ESSENCE may predispose to chronic or life-long neurodevelopmental disorders, other mental and psychiatric problems, drug abuse, physical disorders, antisocial behaviors, and premature death. With regard to prevalence, ESSENCE can be estimated to affect at least 10% of children under the age of 18 years. About half of the whole group is currently probably discovered already by age 6 years. It can also be expected that half or more of all “chronic” adult psychiatric patients suffer from disorders that are linked to ESSENCE.

ESSENCE is “the essence in child psychiatry and child public health”.5 In clinical practice today, there is a trend to ascertain “precise diagnosis” – such as ASD or ADHD – and that this should be followed by “specific interventions” targeting “the” diagnosis. Early detection of the child’s problems and starting intervention is important, because it helps to improve the abilities of the child and the quality of family life. On the other hand, the current trend of “very early diagnosis and very early intervention” often focuses on one diagnosis such as “ASD only” or “ADHD only” and tends to continue underestimating or ignoring the complexity of the real-life situation.6 If a child suffers from ASD, it is very likely that he/she may also suffer from ADHD7 and/or learning disabilities including IDD8,9 or SLD.10 If a child has ADHD, it is also likely that he/she may suffer from other developmental disorders, such as obsessive compulsive disorder, DCD, and ODD.11 All this means that children with ESSENCE need to get a holistic approach. In the public health system, children with developmental problems need to be identified as early as possible even in cases where we might not immediately be able to pinpoint “a precise diagnosis”.12

The above-mentioned clinical aspects of ESSENCE also apply to Japan.

For early detection of developmental delay or problems, using validated screening tools is indispensible.13,14 To capture problems within the ESSENCE group, the ESSENCE-Questionnaire (ESSENCE-Q) was developed in 2012.15 ESSENCE-Q is a brief, simple “quick and easy” questionnaire consisting of 12 items covering the 12 areas referred to above. Responses of Yes (Y), Maybe/A Little (M/AL), or No (N) should be checked for each item. It is currently believed that all children scoring one or more Y or three or more M/AL should be regarded as being at risk for neurodevelopmental disorders and should be assessed by a developmental specialist or screened for possible further problems by a health visitor/nurse.15 However there is no published study of the psychometrics of the ESSENCE-Q yet. We translated this questionnaire into Japanese with the aim to use it as a parent questionnaire for a pilot study of the psychometrics of the ESSENCE-Q.

The aims of the study were to 1) estimate the reliability (internal consistency) of the ESSENCE-Q, 2) evaluate the clinical cutoff level suggested by the author of the ESSENCE-Q (Y ≥1 or M/AL ≥3) as a parent questionnaire in a clinical setting, and 3) propose an optimal cutoff level based on receiver operating characteristic (ROC) analysis.

Method

This is a clinic-based investigation using the Japanese version of the ESSENCE-Q.

ESSENCE-Q

The ESSENCE-Q (Figure S1) is intended for use both in clinical practice and population research to help clinicians and researchers identify children with neurodevelopmental disorders. It is proposed to be useful as a brief interview by a specialist or as a questionnaire to be completed by parents and/or caregivers. It is not a diagnostic instrument but a primary screening tool.15 It is freely available at www.gnc.gu.se. For this study, we used the first version of the ESSENCE-Q, consisting of eleven items.

The translation procedure

Even though the words used in the ESSENCE-Q are simple, the cultural difference between European countries and Japan required a very careful translation process, that is, the Brislin’s translation model.16 After getting the author’s permission, a child psychiatrist in Japan (psychiatrist A, see below = first author of this paper), who has been working in the field of neurodevelopmental disorders for more than 25 years and understands the concept of ESSENCE, translated the ESSENCE-Q into Japanese. A native English speaker, who has been living in Japan for the previous 14 years and has been working as a lecturer of Japanese literature in a university for 5 years and with enough knowledge of the Japanese language, conducted the back-translation into English. The author reviewed and compared the back-translation to the original instruments. After repeating this process, we established the Japanese version of the ESSENCE-Q.

Setting

The research period was 1 year (from May 21, 2012 to May 8, 2013). The place was a developmental clinic in the Kochi Prefectural Medical Welfare Centre. The center is located in Kochi City, the capital of Kochi Prefecture on the island of Shikoku in southern Japan (population of about 760,000, about 5,300 new births per year). The center has been in existence for about 20 years (the predecessor was the prefectural institution of orthopedically impaired children established in 1956), and a network and referral system have been built up over time so that the vast majority of young children with suspected developmental problems in the prefecture are referred to the center. Most cases are referred by pediatricians, child psychiatrists, speech therapists, preschool teachers, or public health nurses, but some come directly because of parental concerns regarding the child’s development. School children had also been referred to the center during the research period and a few adults had been assessed and diagnosed with a neurodevelopmental disorder, such as ASD and/or ADHD. During the research period, the number of new patients assessed at the center was 721. Of these, 539 were under the age of 6 years (75%).

Two psychiatrists (A and B), one pediatric neurologist and one orthopedist were responsible for the neurodevelopmental/neuropsychiatric/medical assessments at the center. Psychiatrist A mainly assessed the preschool children and psychiatrist B (Figure 1) mainly the school children referred for different types of neurodevelopmental/neuropsychiatric problems. The pediatrician mainly saw the youngest children (mostly under the age of 3 years) referred because of neurodevelopmental concerns. The referral reason for all children assessed by the orthopedist was concern regarding major motor developmental problems, especially cerebral palsy.

Figure 1

The flowchart shows all new patients under the age of 6 years referred to the Neurodevelopmental Center during the research period.

Note: The children assessed with the ESSENCE-Q are in the bold rectangle at the bottom of the chart.

Abbreviation: ESSENCE-Q, ESSENCE-Questionnaire.

Subjects

The ESSENCE-Q (Figure S1) was used by the research psychiatrist (psychiatrist A, see above) at the center. Of the 182 patients seen by him during the research period, 145 were under the age of 6 years. A nurse forgot to provide the ESSENCE-Q in five children. Ten children stopped coming to the center before a final decision with regard to diagnosis had been made. The remaining 130 children (mean age 3.5 years, boys:girls =5.2:1), assessed with the ESSENCE-Q and given a full clinical evaluation, and with a final decision regarding diagnosis/no diagnosis, were the study subjects (Figure 1).

Clinical ESSENCE examination procedure

The ESSENCE-Q was completed by a parent (or a caretaker) of the referred child before the first clinical examination. The independent clinical assessment aimed at examining all kinds of developmental areas and disorders included under the ESSENCE umbrella (overall development, motor and perceptual performance, social communication and related behaviors, social interest, joint attention, imitation, play, reciprocal affective behavior, and insistence on sameness/stereotypes).17–19 The following diagnostic categories were covered in the assessment; ASD, ADHD, IDD, DCD, SLD, ODD, borderline intellectual functioning, and learning difficulties. It also included unstructured clinical observations at the clinic and unstructured reports/interviews from parents and preschool teachers, combined with interviews using the attention deficit hyperactivity disorder Rating Scale-IV.20 Cognitive assessments were performed mainly by using the Kyoto Scale of Psychological Development 2001,21 which has excellent psychometric properties, and with results that are closely correlated with those obtained at testing of young children with the Bayley Scales of Infant Development.22 The Tanaka–Binet Scale of Intelligence23 was also used in several cases. The Tanaka–Binet test is a modified version for Japanese of the original Binet–Simon test, being a method by which mental age is evaluated and intelligence quotient is calculated from the ratio of mental age to chronological age. A cognitive assessment was performed in all cases of children displaying symptoms or who otherwise raised suspicion of general cognitive delay. For a diagnosis of IDD, both the developmental/intellectual quotient and adaptive behavior levels were taken into account.24 For a diagnosis of DCD, occupational therapists together with the psychiatrist examined children while standing, walking, throwing and catching a ball, and performing other fine motor skills.25–29 The evaluations were repeatedly done over at least two different sessions separated by a minimum of a few weeks.30 The International Statistical Classification of Diseases-10/Diagnostic and Statistical Manual of Mental Disorders-IV criteria for diagnoses of child psychiatric/neurodevelopmental disorders were used throughout.

Children who had been diagnosed with at least one ESSENCE diagnosis were classified as diagnosis positive, and children with no such diagnosis/diagnoses after clinical assessment were classified as diagnosis negative.

Analysis

To estimate the reliability of the ESSENCE-Q, we used Cronbach’s alpha for internal consistency.

ROC curves were established for all possible sum scores of Y (1–11) and M/AL (1–11) separately. Youden index was also calculated. Youden index is ([sensitivity +specificity] −1) and ranges from 0 to 1.31

All analyses and drawings of ROC curves were performed using the R software (R version 3.2.2; The R Foundation for Statistical Computing, Vienna, Austria), and sensitivities, specificities, Youden indices, and their respective 95% confidence intervals (CIs) were calculated using the package epiR (The R Foundation for Statistical Computing).

Ethics

The study was approved by the Ethics Review Board of Kochi Prefectural Medical Welfare Center. Verbal informed consent was obtained from all parents of the participating children.

Results

A total of 130 children, 109 boys (84%) and 21 girls (16%), were evaluated using the ESSENCE-Q and a full clinical assessment. The informants included 124 mothers (95%), five fathers (4%), and one other caregiver (1%). The age at first visit varied from 21 to 72 months (mean age 41.6, standard deviation [SD] 13.7). Of the 130 children, 113 (96 boys and 17 girls) (87%) were given ESSENCE diagnoses, and 17 (13 boys and four girls) (13%) were not.

In the diagnosis positive group, the age at the first visit varied from 22 to 72 months (mean 41.7, SD 13.7). In the diagnosis negative group, the first visit had occurred between 21 and 64 months (mean 40.5, SD 13.5). Table 1 shows the diagnoses and other demographic information.

Table 1

Diagnosis and other information of the 130 children

Diagnosis	Boys	Girls	Total	Mean age (months) (SD) at first visit and ESSENCE-Q screening	Mean age (months) (SD) at diagnosis	Mean period between first visit and diagnosis (months) (SD)
ASD	32	10	42	37.6 (12.7)	41.5 (13.4)	3.9 (6.9)
ADHD	14	0	14	49.1 (15.5)	60.7 (13.5)	11.6 (13.0)
IDD	11	1	12	43.9 (13.3)	46.6 (16.6)	2.7 (8.6)
BIF	1	1	2	46.5	46.5	0
LD	2	1	3	49.7 (11.4)	62.3 (11.4)	12.7 (11.4)
ASD + IDD	17	3	20	35.4 (12.5)	36.4 (12.5)	1.0 (2.5)
ASD + ADHD	8	1	9	48.6 (9.6)	57.3 (9.3)	8.8 (9.7)
ASD + DCD	3	0	3	61.3 (5.9)	64.3 (3.2)	3.0 (3.0)
ADHD + IDD	4	0	4	42.5 (13.5)	49.8 (13.0)	7.3 (10.4)
ADHD + DCD	1	0	1	46	70	24
LD + DCD	1	0	1	47	67	20
ASD + ADHD + DCD	2	0	2	44	69	25
Diagnosis (+)	96	17	113	41.7 (13.7)	47.3 (16.0)	5.5 (9.1)
Diagnosis (−)	13	4	17	40.5 (13.5)	54.8 (12.1)	13.5 (12.0)
All	109	21	130	41.6 (13.7)	48.1 (157)	6.6 (9.8)

Abbreviations: ADHD, attention-deficit/hyperactivity disorder; ASD, autism spectrum disorder; BIF, borderline intellectual functioning; DCD, developmental coordination disorder; ESSENCE-Q, ESSENCE-Questionnaire; IDD, intellectual development disorder; LD, learning difficulties; SD, standard deviation.

The Cronbach’s alpha across all eleven items was α=0.815.

Table 2 shows sensitivities and specificities of all possible cutoff levels. At the clinical cutoff levels by the author, the sensitivity was 0.94 (95% CI: [0.88, 0.98]), and the specificity was 0.53 (95% CI: [0.28, 0.77]).

Table 2

Sensitivity and specificity in all possible cutoff levels

Number of “Yes” responses	Number of “Maybe/A Little” responses
	1	2	3	4	5	6	7	8	9	10	11
1	(0.99, 0.18)	(0.97, 0.35)	(0.94, 0.53)*	(0.89, 0.59)	(0.89, 0.65)	(0.87, 0.65)	(0.87, 0.65)	(0.86, 0.65)	(0.86, 0.65)	(0.86, 0.65)	(0.86, 0.65)
2	(0.98, 0.18)	(0.94, 0.47)	(0.87, 0.77)**	(0.79, 0.82)	(0.76, 0.88)	(0.73, 0.88)	(0.71, 0.88)	(0.69, 0.88)	(0.69, 0.88)	(0.69, 0.88)	(0.69, 0.88)
3	(0.97, 0.18)	(0.89, 0.47)	(0.79, 0.77)**	(0.70, 0.82)	(0.63, 0.88)	(0.55, 0.88)	(0.52, 0.88)	(0.49, 0.88)	(0.49, 0.88)	(0.49, 0.88)	(0.49, 0.88)
4	(0.96, 0.18)	(0.87, 0.53)	(0.76, 0.82)	(0.66, 0.88)	(0.55, 0.94)	(0.46, 0.94)	(0.43, 0.94)	(0.40, 0.94)	(0.39, 0.94)	(0.39, 0.94)	(0.39, 0.94)
5	(0.93, 0.18)	(0.84, 0.53)	(0.71, 0.82)	(0.59, 0.88)	(0.47, 1.00)	(0.37, 1.00)	(0.34, 1.00)	(0.30, 1.00)	(0.29, 1.00)	(0.29, 1.00)	(0.29, 1.00)
6	(0.91, 0.18)	(0.81, 0.53)	(0.65, 0.82)	(0.49, 0.88)	(0.35, 1.00)	(0.25, 1.00)	(0.21, 1.00)	(0.18, 1.00)	(0.17, 1.00)	(0.17, 1.00)	(0.17, 1.00)
7	(0.89, 0.18)	(0.79, 0.53)	(0.61, 0.82)	(0.42, 0.88)	(0.27, 1.00)	(0.16, 1.00)	(0.12, 1.00)	(0.09, 1.00)	(0.08, 1.00)	(0.08, 1.00)	(0.08, 1.00)
8	(0.87, 0.18)	(0.77, 0.53)	(0.58, 0.82)	(0.38, 0.88)	(0.24, 1.00)	(0.12, 1.00)	(0.09, 1.00)	(0.05, 1.00)	(0.04, 1.00)	(0.04, 1.00)	(0.04, 1.00)
9	(0.87, 0.18)	(0.77, 0.53)	(0.58, 0.82)	(0.37, 0.88)	(0.23, 1.00)	(0.12, 1.00)	(0.08, 1.00)	(0.04, 1.00)	(0.04, 1.00)	(0.04, 1.00)	(0.04, 1.00)
10	(0.87, 0.18)	(0.77, 0.53)	(0.56, 0.82)	(0.35, 0.88)	(0.21, 1.00)	(0.10, 1.00)	(0.06, 1.00)	(0.03, 1.00)	(0.02, 1.00)	(0.02, 1.00)	(0.02, 1.00)
11	(0.87, 0.18)	(0.75, 0.53)	(0.54, 0.82)	(0.34, 0.88)	(0.20, 1.00)	(0.08, 1.00)	(0.04, 1.00)	(0.01, 1.00)	(0.00, 1.00)	(0.00, 1.00)	(0.00, 1.00)

Notes:

Cutoff level by the author of the ESSENCE-Q.

Cutoff levels that have the sensitivity and the specificity both in 0.7–0.8 ranges, and also sensitivity > specificity.

Abbreviation: ESSENCE-Q, ESSENCE-Questionnaire.

Table 3 shows cutoff levels, sensitivities, and specificities with the three highest Youden indices.

Table 3

Combinations of cutoff levels with the three highest Youden indices

Cutoff levels	Sensitivity 95% CI	Specificity 95% CI	Youden index 95% CI
Y ≥2 or M/AL ≥5	0.76 (0.67, 0.84)	0.88 (0.64, 0.99)	0.64 (0.31, 0.82)
Y ≥2 or M/AL ≥3	0.87 (0.79, 0.92)	0.77 (0.50, 0.93)	0.63 (0.29, 0.86)
Y ≥2 or M/AL ≥4	0.79 (0.70, 0.86)	0.82 (0.57, 0.96)	0.61 (0.27, 0.82)

Notes: Youden index =sensitivity +specificity −1. Bold combination best fit with rule of thumb sensitivity > specificity, sensitivity and specificity both in 0.70–0.80 ranges, and Youden index high.

Abbreviations: CI, confidence interval; M/AL, Maybe/A Little; Y, Yes.

Discussion

The reliability (internal consistency) of the ESSENCE-Q was found to be acceptable as good. The sensitivity and the specificity of the ESSENCE-Q using the suggested clinical cutoff level were 0.94 and 0.53, respectively.

The ESSENCE-Q is not a tool for diagnosis but a tool for primary screening of ESSENCE. In screening, it is important not to miss children being at risk for neurodevelopmental disorders who should be assessed by a developmental specialist or screened for possible further problems by a health visitor/nurse.15 Therefore, the emphasis of the ESSENCE-Q is on the sensitivity and not on the specificity. It is generally agreed that the ideal developmental screening test should err in the direction of false positive.32 Therefore, the cutoff scores suggested by the author are reasonable, even though specificity is only moderate.

On the other hand, a screening tool with high false positive rate will be costly.33 For a developmental screening test, it has been suggested that the best sensitivity and specificity balance is one where both are around 0.70–0.80.13,14 High sensitivity is crucial not to miss the children with developmental problems, whereas, to minimize over referrals, specificity should be closer to 80%.13 From the results in Table 2, the cutoff levels that have higher sensitivity than specificity and that have both sensitivity and specificity in 0.70–0.80 were Y ≥2 or M/AL ≥3, and Y ≥3 or M/AL ≥3. Among cutoff levels that had relatively high Youden indices on the basis of the ROC analysis (Table 3), we would recommend Y ≥2 or M/AL ≥3 as the optimal cutoff level in a clinical neurodevelopmental center. When we used Y ≥2 or M/AL ≥3 as a cutoff level, the sensitivity was 0.87 (95% CI: [0.79, 0.92]), and the specificity was 0.77 (95% CI: [0.50, 0.93]).

In this study, we explored cutoff levels of the ESSENCE-Q to improve the usability. We cannot decide one fixed cutoff level only from this pilot study, but if we could find a proper cutoff level (cutoff levels depend on the usage), it seems likely that the ESSENCE-Q can be a good instrument for use by nurses or general practitioners at child health care centers as a screening tool of neurodevelopmental disorders. It can also be administered by pediatricians or child psychiatrists as a parent screening questionnaire in clinical settings to cover important developmental areas.

Screening tools should be simple, easy to be done, acceptable, and cheap,33 also inexpensive in terms of training and brief in its administration.32 ESSENCE-Q is fitting all these conditions. In the ESSENCE-Q, the questions are about “concern” for each of the ESSENCE domains, not about specific symptoms or behaviors. “Yes”, “No”, and “Maybe/A Little” are the response alternatives. It may appear to be somewhat unspecified and vague, but this is actually the unique strength of the ESSENCE-Q. Behaviors of young children are very varied, and if we focus on a specific behavior too much, we tend to miss the behaviors that do raise some concern but that we have difficulty pinpointing exactly. Using the ESSENCE-Q provides an opportunity to catch nonspecified but critical information of the child’s development. ESSENCE-Q is not geared toward a specific neurodevelopmental diagnosis, and this could be the strength for screening across a whole range of ESSENCE problems. Thus, the ESSENCE-Q was developed to be used as a broad neurodevelopmental screening instrument in young children and differs from many other assessment tools targeting for example speech and language problems.34 The questionnaire could also be useful in a public health situation as a parent questionnaire and/or as an interview questionnaire and/or as an observation record. The cutoff scores might have to change according to the usage, and further studies are needed for this.

Two major limitations of the present study are that 1) the analyses were based on a clinical sample, and accordingly we do not yet know how well (or not well) the instrument works in a general population sample of children and 2) that the number of cases in the diagnosis negative group was very small. To establish the validity and the reliability of ESSENCE-Q, case–control studies in large numbers and population studies are needed. Additionally, we are planning other studies of the ESSENCE-Q, including some propositions for a decision-making classification on preclinical and clinical stages.

ESSENCE-Q.