Validation of two parent-reported autism spectrum disorders screening tools M-CHAT-R and SCQ in Bamako, Mali.

BACKGROUND: Early screening is crucial for early autism spectrum disorders (ASD) diagnosis and intervention. ASD screening tools have mostly been constructed based on the Western cultural context. We hypothesized that their use in Mali may require a prior validation.
OBJECTIVE: To validate the modified checklist for autism in toddlers-Revised (M-CHAT-R) and the social communication questionnaire (SCQ) in the Malian sociocultural context for ASD screening. STUDY
DESIGN: We administered M-CHAT-R and SCQ in 947 toddlers aged 16-30 months old at the district and community health centers in Bamako and 120 patients (60 autistic and 60 age and sex matched controls) aged ≥4 years old at the psychiatry department in Bamako. Toddlers at moderate to high risk of ASD underwent M-CHAT-R/F and clinical evaluation by an ASD multidisciplinary team. M-CHAT-R and SCQ were evaluated for cultural appropriateness by Malian anthropologists. The sensitivity, specificity, PPV, NPV were determined for both M-CHAT-R and SCQ. Health professionals have been trained during ASD seminary on how to use M-CHAT-R and SCQ for ASD screening in Bamako.
RESULTS: We found for the M-CHAT-R a sensitivity of 50%, a specificity of 100%, a PPV of 100% and a NPV of 87%. The SCQ had a sensitivity of 71%, a specificity of 72%, a PPV of 73% and a NPV of 70%. We have found four out of 20 items on the M-CHAT-R that were culturally inappropriate in the Malian context. DISCUSSION: M-CHAT-R and SCQ can be used for early autism screening in Mali. In the future, we plan to train a descent number of Malian physicians in chief and pediatricians at the district hospitals across the country to integrate the early ASD screening into the national health system.
CONCLUSION: M-CHAT-R has a perfect specificity and SCQ a fair diagnostic accuracy for ASD in Mali.

Introduction

The modified checklist for autism in toddlers-Revised (M-CHAT-R) and the social communication questionnaire (SCQ) are parent report-based autism spectrum disorder (ASD) screening tools developed and widely used in the Western countries. The M-CHAT-R is used to screen toddlers aged 16–30 months old, whereas the SCQ is for autistic individuals aged ≥4 years old. The use of such ASD screening tools in Africa requires prior validation in the local sociocultural context.

In Mali, ASD is not only stigmatized, but also not well recognized by most people, including health professionals. ASD is more common in Mali than previously expected. Our preliminary data showed 1 in 27 neuropsychiatric outpatients (4.5% or 105/2343) at the psychiatry department of the University hospital Point G was autistic. Parents and families more often don't seek diagnosis and care for their autistic children. Autistic children with epilepsy as comorbidity aggressive behavior are more likely to be visited by a physician or a health professional in late childhood (unpublished data). The grandmothers of toddlers at risk of ASD are more likely to grow suspicious about early ASD symptoms in their families. Most health professionals including pediatricians were unaware of ASD before our ASD awareness activities in Bamako, the capital city of Mali. ASD was not taught at the medical until 2014 when our ASD research first started at the faculty of medicine and odonto-stomatology (FMOS) in Bamako. Training health professionals at the community health centers in the use of M-CHAT-R and SCQ in rural and urban Mali is crucial in raising ASD awareness and facilitating the recruitment of ASD families into our ASD research protocol. Especially in the context of how few child psychiatrists or neurologists there are in all of Mali and Africa for that matter. M-CHAT-R and SCQ would fill the immense void in early ASD screening in Mali.

We hypothesized that M-CHAT-R and SCQ could be successfully validated in the Malian population. In this study, we aimed to validate M-CHAT-R and SCQ lifetime in a general population based survey at district health centers (CSRef) and community health centers (CSCOM) in Bamako.

Materials and methods

Description of the health system in Mali

The Malian health system is organized into three tiers in Bamako from the bottom/local to the top in terms of intensity: the community health centers, CSCom (n = 57); the district health centers, CSRef (n = 6), one in each of the six administrative communes, and the University hospital centers, CHU (n = 6). Private health structures exist at each level. The referrals are normally done from the bottom (CSCOM) to the top (CHU). Public and private health structures are under the direct authority of the regional direction of health (DRS) and the National direction of health (DNS). The ministry of health and public hygiene is the highest health authority in the country.

Generation of preliminary data

In 2014, we reviewed 12,000 medical charts of children aged 14 years old and younger from a decade of medical practice (outpatient visits) at the psychiatry department of the University hospital Point G and at the private clinic Kaidara in Bamako, Mali. The diagnostic of ASD was based on the Diagnostic and Statistical Manual of Mental Disorders (DSM-V) and the International Statistical Classification of Diseases and Related Health Problems-10 (ICD-10).

M-CHAT-R/F administration and scoring

M-CHAT-R administration

Our ASD research protocol, consent and assent forms, and questionnaires have been approved by the IRB at the faculty of medicine and odonto-stomatology (FMOS)/Faculty of Pharmacy (FAPH) at the University of Sciences, Techniques and Technologies of Bamako (USTTB). Fourteen surveyers (12 health professionals and 2 non-health professionals, graduates from the school of public administration in Bamako) were trained during a 1-day seminary on ASD, M-CHAT-R administration and the study design. After the approval of the regional direction of health (DRS) in Bamako, we paired up the health professional surveyers and each pair was assigned one of six district health centers (ex-CSRef) and its two most frequented community health centers (CSCOM). The two non-health professionals worked with each pair of surveyors to administer in parallel the study questionnaire to the same toddlers (100 in total). The 2-page study questionnaire consisted of the French version of the M-CHAT-R and another questionnaire to collect sociodemographic and risk factors data for the toddlers aged 16–30 months old and personal information of his/her parents. At the CSCOM level, community consents were obtained with the traditional leaders and mayors; local guides were used for community engagement. Each study participant received compensation. The survey lasted five weeks (three at the CSRef level and two at the CSCOM level).

M-CHAT-R scoring

Each item in the M-CHAT-R requires a dichotomous “yes”/“no” response, and each scored item receives 1 point for presence of the abnormal behavior and 0 points for its absence or the presence normal behavior. Yes means abnormal behavior in all the items except Items 2, 5 and 12 for which No means abnormal behavior. The ASD risk was estimated as followed: score = 0–2 (low risk), score = 3–7 (moderate risk) and score = 8–20 (high risk).

M-CHAT-R/F administration

The parents or guardians of toddlers at moderate to high ASD risk were contacted and invited for the M-CHAT-R/F and clinical evaluation by our multidisciplinary ASD research team at the University hospital Point G. The discussion during the clinical evaluation was based on the presence or absence of early ASD red flag signs and symptoms by age one year old. Each study participant was compensated for the participation and reimbursed for transportation.

SCQ administration and scoring

A medical student, as part of his MD thesis, and a junior psychiatrist, both used the SCQ lifetime version to screen 60 patients aged 4–20 years old diagnosed with ASD per the DSMV criteria during the weekly ASD clinic at the psychiatry department of the University hospital Point G and 60 age and sex matched controls (26 epilepsy cases and 34 other neuropsychiatric disorders including mental retardation, acute psychosis, cerebral palsy, etc.…) from Point G and the mental health care center for the Malian association for mental deficiencies (AMALDEME) in Bamako. In Point G, informed consent was obtained from the accompanying adults and patients who were 18 years old and older. Younger patients if not emancipated signed the assent forms. Compensation was given to each study participant. At the AMALDEME, after the approval of the director of the center, three research nurses who are the daily caregivers and guardians of the patients consented for the study and provided information on patients. Nurses were compensated individually, but the IRB approved rate for compensation was added up for the total number of study participants in the center. The equivalent in food (powder milk, sugar, snake, tooth paste and toothbrush) was handed to the director for the participants. The study participant recruitment lasted six months in Point G and one month at the AMALDEME. The filled SCQ questionnaires were reviewed and scored by the research team.

Each item in the SCQ requires a dichotomous “yes”/“no” response, and each scored item receives 1 point for abnormal behavior and 0 points for its absence or normal behavior. The first item—“Is she/he now able to talk using short phrases or sentences?”—is not scored, but rather determines whether six items for abnormal language are scored. Only “verbal” children (i.e., children with a “yes” response to the first question) are assigned the six items relating to abnormal language while “non-verbal” children (i.e., children with a “no” response to the first question) are not resulting in a 6-point total score difference between the two groups. The ASD risk was based on the following cutoff: score < 15 out of 40 (no risk) and score > 15 (at risk).

Anthropologic evaluation of M-CHAT-R and SCQ

Each of the 20 items on the M-CHAT-R and 40 items on the SCQ were evaluated for their appropriateness in the Malian sociocultural context. In other words, the question in each item of the M-CHAT-R and SCQ was evaluated with the following two concerns: (i) is the question as formulated easy to understand by mothers mostly illiterate? (ii) Are the objects (examples: airplane, vacuum cleaner, etc.…) or scenarios (example: mimicking feeding a doll) in the question known or commonly used by most Malians? When inappropriate, a proposition was made on how to rephrase the question in the item either by keeping its key words or replacing some with equivalent words (example: noise produced by a pestle in a mortar instead of that of a vacuum cleaner). Our evaluation was based on the vast majority of Malians seen at the CSCOM level meaning mostly uneducated and poor. The minority of well-educated and wealthy Malians leading a Western lifestyle were not taken into account since they are treated in ether private clinics or University hospitals or even abroad.

ASD awareness seminary

A 1-day ASD awareness seminary has been developed in the following format: (i) oral presentation on the general information on ASD followed discussion on the early ASD symptoms in toddlers (ii) testimonials of four Malian ASD families (iv) review of the items on M-CHAT-R and SCQ (v) attendees practice M-CHAT-R and SCQ on the available ASD families (vi) each attendee received a dozen of copies of M-CHAT-R and SCQ for autism screening at their respective health centers and give ideas on how early ASD screening could be implemented into the Malian health system.

Sample size

In total, 947 questionnaires were used for the M-CHAT-R among which only 17 out of 89 eligible took part to the M-CHAT-R/F and clinical evaluation. For the SCQ, the sample size was set at 120 study participants aged 4–20 years old (60 autistic and 60 age and sex matched controls with neuropsychiatric disorders other than ASD) based on feasibility. We calculated for both the M-CHAT-R and the SCQ the sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), positive likelihood (LR+) and negative likelihood (LR-), Area under the Curve (AUC) and the Youden's J.

Working definitions

Cutoff

The M-CHAT-R cutoff values were 1–2 for low risk, 3–7 for moderate risk and 8–20 for high risk. Only a score ≥ 8 was eligible for the M-CHAT-R/F and clinical evaluation. The SCQ cutoff value was 15 as suggested in the manual of SCQ, a total SCQ score > 15 meant positive or < 15 negative ASD diagnosis for each study participant [1].

Sensitivity or true positive rate

The proportion of toddlers or patients with ASD who are correctly identified as having ASD or at ASD risk. Specificity or true negative rate: the proportion of toddlers aged 16–30 months old or patients aged 4–20 years old without ASD who are correctly identified as not having ASD. Interpretation of the values: 1.0 = perfect; 0.9–1.0 = very good; 0.8–0.9 = good; 0.7–.0.8 = fair; <0.7 = poor [2].

Youden's J

It was calculated using the formula: Sensitivity + Specificity – 1 as an indicator to establish the optimal cutoff point. It gives equal weight to sensitivity and specificity. Values from +1 (indicates a perfect measure with neither false positives nor false negatives) to −1 (indicates a perfect inverse measure). A value of 0 indicates that the measure has no value [3].

Area under the ROC curve

A measure of how well the SCQ Total Score can distinguish between the presence of an ASD diagnosis and the absence of an ASD diagnosis. AUC varies between 0 and 1 (in normalized units), where the diagnostic accuracy is perfect (AUC = 1.0); very good (AUC is between 0.9 and 1.0); good (AUC is between 0.8 and 0.9); fair (AUC is between 0.7 and 0.8); poor (AUC is between 0.6 and 0.7); very poor (AUC is between 0.5 and 0.6); and non-discriminating (AUC = 0.5) [4].

Negative predictive value (NPV)

NPV is the probability of a person who receives a negative test result actually does not have the disease. Positive predictive value (PPV) is the probability of a person who receives a positive test result actually has the disease [5].

Likelihood ratios (LR), sensitivity and specificity data have been used to calculate the Likelihood ratio for positive test results (LR+) and Likelihood ratio for negative test results LR-. For both M-CHAT-R and SCQ, LR+ was calculated using the formula: sensitivity/(1-specificity) as an indicator for ruling in the diagnosis or risk of ASD where higher is better. Good diagnostic tests have LR+ > 10. LR− was calculated using the formula: (1-sensitivity)/specificity as an indicator for ruling out the diagnosis or risk of ASD where lower is better. Good diagnostic tests have LR- < 0.1 [1].

Ethical considerations

Study participation was voluntarily. Compensation was given to the study participants. No personal identifier, but a code was used in either questionnaire. The key to the code was written on a spreadsheet and stored in an excel database accessible only to the medical student, epidemiologist and the principal investigator. The questionnaires were stored in a locked cabinet at the FMOS, USTTB. Study participants suffering from epilepsy received free antiepileptic medications (valproid acid and carbamazepine).

Results

From our preliminary data (Table 1), the diagnosis of ASD may be relatively late in the childhood in Mali based on the age of first medical visit (7.64 ± 3.85 years) of autistic children in Bamako. All the qualified health professionals for ASD diagnosis are in Bamako, the capital city. The further away from Bamako the scarcer the health services implying an even later ASD diagnosis age in rural Mali. In addition, only 11.4% (n = 105) of autistic children were schooled in public education where teachers are unaware of ASD. The annual new patient rate of 7 and the hospital frequency of 4.5% (1 in 27) are underestimating the number of ASD cases in Bamako. Finally, the rate of first degree consanguinity of 13.9% and the history of psychosis on maternal and paternal families in 6.15–7.21% (Table 1) highlighted the potential for molecular ASD genetic research. Key to early ASD screening and diagnosis in the Malian context relies on the easy-to-use parent-reported ASD screening tools such as M-CHAT-R and SCQ.

Table 1

Study population description in the preliminary data and M-CHAT-R survey.

Preliminary data from the medical chart review	Values
Sex ratio	1.5
Age range	3–14 years old
Average age at the first outpatient visit	7.64 ± 3.85 years old
Schooling rate of autistic children	11.4% (n = 105)
Average number of new patients per year from 2000 to 2014	7 (peak of 22 in 2012)
ASD hospital frequency	4.5% (n = 2343)
First degree consanguinity rate (%) with 95% CI	13.19% IC95% = [11.86–14.6]
Family history of psychosis in paternal side	7.21% IC95% = [6.22–8.31]
Family history of psychosis in maternal side	6.15% IC95% = [5.23–7.17]
M-CHAT-R survey	Values
Sex ratio	1.08
Age range	16–30 months old
Rate of non-educated mothers	70% (n = 947)

Discussion

Interpretation and significance of the preliminary data for ASD research in Mali with an emphasis on late ASD diagnosis

We found an ASD hospital frequency of 4.5%. Due to stigma surrounding ASD, only few among many other parents seek care at the hospital or private medical clinic for their autistic children. ASD is still thought to be resulting from witchcraft or parental misdeed or sin. Traditional healers and hunters are more likely to be consulted to treat autistic children in rural even urban Mali. Perceptions on ASD and its treatment options widely vary across cultures [6]. The peak of 22 ASD patients in 2012 was due to the effect of the creation of the association of autistic families affiliated to the pediatrics department of the University hospital Gabriel Toure.

Study limitations (sample size, age ranges, rural study site)

The age ranges for both M-CHAT-R and SCQ were set at 16–30 months old and ≥ 4 years old, respectively. The choice of these two ASD screening tools will obviously leave children between 30 months old and 4 years old unscreened. There is so much work to be done in ASD screening and diagnosis in most African countries [7] that we had to start somewhere in Mali. This study was limited to the ethnically diverse and mixed population in Bamako therefore our results may not necessarily be extendable to some rural areas.

Anthropologic evaluation of the M-CHAT-R and SCQ

From one country to another or even between different ethnic groups in same country, cultural differences may greatly influence on how ASD is perceived and cared for worldwide [8]. We therefore had the M-CHAT-R and SCQ reviewed by anthropologists at the FMOS in Bamako for cultural appropriateness. While SCQ revealed no problematic item, four items (3, 4, 6 and 12) were inappropriate in the Malian sociocultural context. Coincidentally, those items were the only ones to register the highest failed response rates (Table 2). For instance, asking a question of the noise from a vacuum cleaner may mean nothing to most of the surveyed mothers who might not have heard of such thing before. Instead of a vacuum cleaner, they used the traditional wipers on a daily basis.

Table 2

Anthropologic evaluation of appropriateness in the Malian sociocultural context of the items on the M-CHAT-R.

Item# on the M-CHAT-R	Appropriateness in the Malian sociocultural context	Yes or No response	Frequency (%)N = 82**
3	No	No	20 (24.4%)
4	No	No	20 (24.4%)
6	No	No	40 (48.8%)
12	No	No	37 (45.1%)

*Items on M-CHAT-R were reviewed for the 82 of the 90 toddlers eligible for follow up and clinical evaluation. The frequency of failed responses and corresponding percentages were determined.

Note: All the 40 items on the SCQ were labeled appropriate in the Malian sociocultural context.

Discuss the significance of the validation especially 100% specificity of M-CHAT-R and 73% PPV of SCQ; discriminative capacity in ASD versus ASD + epilepsy

From our M-CHAT-R and SCQ validation data (Table 3), the sensitivity of M-CHAT-R was as low as 50% with a specificity of 100%. With limited ASD research funds per the 10/90 gap along the lack qualified human resources in Africa [9] as well as the stigma surrounding ASD in Mali, we prefer a screening that may be less sensitive, but very specific. We don't want to label a child autistic when s/he is not. The PPV for SCQ was 73% with a fair diagnostic accuracy (AUC = 0.70) which highlighted the need for a more accurate additional diagnostic tool not solely based on parental reports. Autism diagnostic observation schedule-2 (ADOS-2) will be the ideal ASD diagnostic aid tool to fill this gap.

Table 3

Validation data of M-CHAT-R and SCQ in Bamako, Mali.

Characteristics	M-CHAT-R	SCQ lifetime
Sex ratio (male/female)	1.08	1.6
Median age with extremes	24 months old with the extremes of 16–30 months old	10 years old with the extremes of 4 and 20 years old
Sample selection	Communitybased-screening	Health facility-based screening
ASD risk estimation	low risk (n = 857)moderate risk (n = 83)high risk (n = 7)	no risk (16 autistic patients and 42 controls) at risk (44 autistic patients and 18 controls)
Sample size and characteristics	M-CHAT-R (n = 947)M-CHAT-R/F (n = 90)Clinical evaluation (n = 17)	N = 120 (60 autistic patients⁎ and 60 age and sex matched controls)
Sensitivity	50%(2/4) IC95% [0.08–0.91]	71% (44/60) IC95% [0.61–0.82]
Specificity	100%(13/13) IC95% [0.77–1]	72% (42/60) IC95% [0.57–0.80]
Area Under the Curve (AUC)	Not applicable	0.70
Positive predictive value	100%(2/2) IC95% [0.17–1]	70%(44/62) IC95% [0.59–0.81]
Negative predictive value	87%(13/15) IC95% [0.62–0.97]	72%(42/58) IC95% [0.60–0.82]
Positive likelihood (LR+)	Not applicable	2.5
Negativelikelihood (LR-)	0.5	0.01
Youden's J	0.50	0.49

Among the 60 autistic patients, 26 (43.3%) had clinically diagnosed epilepsy and 43 (71.7%) were nonverbal.

Epilepsy is a frequent co-morbidity of ASD [10]. In our cohort, SCQ picked up about 1 in 5 ASD with epilepsy when the cutoff is at 15 (Table 4). Should the cutoff be lower in this specific patient population is worth of wonder, but it is obvious that epileptic crisis in which the child falls down unconscious, bites his/her tongue, leaks urine and have traumatism from the recurrent falls draw much more the attention of their parents than autistic symptoms.

Table 4

Discriminative capacity of the SCQ lifetime in diagnosing ASD in the presence or absence of epilepsy as a co-morbidity.

ASD screening tool	Score	Autistic	Non-Autistic	Total
SCQ	>15	44 (73.3%)	nnn(0%)	44 (73.3%)
	<15	0(0%)	16 (26.7%)	16 (26.7%)
	Total	44 (73.3%)	16 (26.7%)	60 (100%)
	Score	Epileptic	Non-Epileptic	Total
SCQ	>15	5 (19.2%)	0(0%)	5 (19.2%)
	<15	0(0%)	21 (80.8%)	21 (80.8%)
	Total	5 (19.2%)	21 (80.8%)	26 (100%)

Training of health professionals across in Bamako, Mali

The use of M-CHAT-R and SCQ for ASD screening has been part of our ASD awareness and research plan. After the validation of M-CHAT-R and SCQ, we worked with the regional direction of health (DRS) to train in ASD screening using M-CHAT-R and SCQ during a 1-day ASD seminary the physician in chief and the pediatrician at all the six district health centers (DRS) in Bamako. Work is underway to train in using M-CHAT-R and SCQ for ASD screening 180 technical directors of the centers (DTC) from all the community health centers (CSCOM) in Bamako. Our ultimate goal is to establish a referral system using a stepped care model in which toddlers aged 16–30 months old and autistic individuals aged ≥4 years old will be subsequently screened at the CSCOM and CSRef levels before those at risk are referred to our ASD research team at the FMOS in Bamako.

Conclusion

M-CHAT-R has a perfect specificity and SCQ a fair diagnostic accuracy for ASD in Mali. In the future, physicians including the DTC will be initiated into the use of M-CHAT-R/F and SCQ. In addition, other ASD screening tools in toddlers [11] and ADOS-2 will be validated and used to supplement M-CHAT-R/F and SCQ. Finally, these M-CHAT-R/F and SCQ validation data will be relevant and useful in many other West African countries.

Funding statement

“Dr. Modibo Sangare, MD, PhD was supported by a postdoctoral fellowship from a DELTAS Africa grant “DEL-15-007: Awandare” to “Professor Gordon Awandare”. The DELTAS Africa Initiative is an independent funding scheme of the African Academy of Sciences (AAS)’s Alliance for Accelerating Excellence in Science in Africa (AESA) and supported by the New Partnership for Africa’s Development Planning and Coordinating Agency (NEPAD Agency) with funding from the Wellcome Trust (107755/Z/15/Z: Awandare) and the UK government. The views expressed in this publication are those of the author(s) and not necessarily those of AAS, NEPAD Agency, Wellcome Trust or the UK government.” Dr. Modibo Sangare was also a grantee of the University of Sciences, Techniques and Technologies of Bamako (USTTB), Ministry of Innovation and Scientific Research of Mali.