ECHO Autism: Evaluation of Participants' Perceptions of Collaborative Telementoring Network.

Introduction: Primary care clinicians report inadequate training and lack of support regarding identification of early signs of Autism Spectrum Disorder (ASD), resulting in later diagnosis and poor outcomes. The objective of this study was to evaluate the perception of participants in ECHO Autism project regarding the value of the virtual collaborative as a mentoring tool.
Methods: We completed a retrospective cross-sectional study of post-virtual clinic surveys of all participants who attended Autism ECHO from September 2, 2015 to June 5, 2019.
Results: There were 87 Autism ECHO sessions held, with 83 didactic presentations. A total of 92 de-identified patient cases were discussed with 490 unique learners in attendance. Participants reported increase in self-efficacy in identifying ASD symptoms in children, assessing medical comorbidities, and learning new information. Discussion: Continuing education platforms in healthcare professions are increasingly embracing virtual live-interactive conferences. They provide opportunities to connect with experts, but also hear from peers regarding real-life case examples. During COVID-19 pandemic ECHO Autism was a lifeline for rural providers and will likely continue to increase in participation.

Introduction

Autism is one of the fastest growing neurodevelopmental disorders worldwide, occurring in all racial, socioeconomic and ethnic groups (1). The percentage of children diagnosed with autism spectrum disorder (ASD) has increased from 1.1% to 2.5% from 2009 to 2017 (2,3). Typical onset of ASD is at around 18 months of age, affecting language, sensori-motor skills, behavior and cognitive functioning, among other characteristics (4). There is no curative treatment for ASD, however, numerous studies have shown that early diagnosis is critical in order to provide timely interventions which can improve a child's long-term outcomes (5–8).

The use of an autism screening checklist at routine pediatric checkups has been shown effective in identifying early signs of ASD, but unfortunately almost 40 percent of children with ASD do not receive evaluation until 4 years of age (7,9,10). Isolating behavior can be viewed as shyness by both parents and clinicians, often causing delayed diagnosis (11,12). In addition, limited local diagnostic resources, mistrust, cultural factors, and costs have been found as significant barriers to early screening (13). Ethnic and racial disparities in autism diagnosis have also been reported – African–American Medicaid-eligible children are 2.6 times less likely than their white counterparts to be diagnosed with autism during their first specialty visit (14). African–American children and children of other ethnicities whose parents are recent immigrants are more likely than white children to be misdiagnosed with adjustment disorder, rather than autism, underlying the need for improved strategy for interactions between clinicians, patients, and families (14). Dosreis et al. found that only 8 percent of pediatricians practicing in general primary care routinely screened for autism, citing lack of familiarity with screening tools as a main reason for not screening (15). Centers for Disease Control also reported that screening numbers remain largely unchanged, with less than 40 percent of children receiving appropriate developmental screening (16). Inadequate and delayed screenings may result in later diagnoses, poor patient outcomes and increases in referrals to specialists. This often causes long wait times, disproportionally affecting patients from rural and underserved areas due to their unique geographic and travel challenges.

ECHO Autism

Faculty from The University of Missouri (MU) School of Medicine in partnership with Missouri Telehealth Network (MTN) implemented Extension for Community Healthcare Outcomes (ECHO) project for building capacity of primary care providers (PCPs) to screen, diagnose, and manage patients with autism (17). The ECHO model uses virtual collaborative learning, rooted in adult learning theory and case-based learning, to share evidence-based guidelines and best practices with participating PCPs (18,19). Expert didactic presentations and telementoring sessions are tailored to the unique learning needs of PCPs and focus on identification of early signs of ASD and implementation of appropriate screening tools in general pediatric clinic settings. The overarching goal of ECHO Autism is to guide participating PCPs in overcoming the reported barriers and provide early screening to their pediatric patients.

ECHO Autism sessions are held twice a month during 1.5-h videoconferences. Expert hub team consists of a pediatrician who specializes in development and behavior, a clinical psychologist, a child and adolescent psychiatrist, a pediatric dietitian, a community resource specialist, and a parent of a child with autism advocate (17). Mazurek et. al. reported that participants in ECHO Autism demonstrated improved self-efficacy in screening, identification, and treatment of patients with autism (20). This is an important finding, as 40 percent of participants reported that low self-efficacy was a major barrier to management of children with ASD (20).

The objective of this project was to determine the participants’ perception of the value of ECHO Autism as a learning tool. The primary hypothesis was that participation in ECHO Autism is perceived as an effective collaborative learning and knowledge-sharing virtual platform.

Methods

This was a retrospective cross-sectional study of online post- virtual clinic surveys. All completed surveys collected during the study period were included in the analysis. The project was acknowledged as quality improvement by the University of Missouri Institutional Review Board (IRB), number 2011517. One study staff (EW) collected raw survey data, and all other project members received de-identified aggregate data.

Study Participants

Study participants were all learners who attended ECHO Autism sessions from September 2, 2015 to June 5, 2019. Four hundred ninety (490) unique learners, with total of 2,262 instances of participation, attended sessions during the study period: 123 medical doctors (MDs) and doctors of osteopathic medicine (DOs), 42 nurses, 20 social workers, 61 clinicians (therapists and psychologists), 40 students (medical, nursing, health professions), 84 administrators, 26 researchers, 22 others, and 72 who did not specify their credentials (Table 1). Of the 123 MDs and DOs, 55 (45%) were pediatricians, 21 (17%) were specialists such as psychiatrists, neurologists, neonatologists, sleep disorder specialists and developmental pediatricians, and 47 (38%) did not specify their type of practice.

Table 1.

ECHO Autism Attendee Credentials.

Attendee credentials	Number (n)
MDs/DOs	123
Nurses	42
Social worker	20
Clinicians (therapists and psychologists)	61
Students	40
Administrators (CEOs, program directors, program coordinators)	84
Researchers	26
Others (family advocates, pharmacists, two medical assistants, communication consultants, and state representatives)	22
Unspecified	72

All 490 participants received a link to online survey after each session they attended. Inclusion criteria consisted of all surveys completed during the study period. We excluded partially or incomplete surveys from the analysis. A total of 128 unique (26% response rate), and 672 total (29%) responses were received. The post-virtual clinic survey was sent one time after each ECHO Autism clinic to all attendees, and completed survey was a pre-requirement for receiving continuing medical education (CME) credits. Of the 490 participants, most practiced at an academic setting (217), followed by health system (60), independent clinics (24), federally qualified healthcare clinic (20) and school-based clinic (10). Other attendees were from Indian Health Services, public health centers, social services, and insurers, among others.

Study Instrument

The 15-question post-virtual clinic survey was created by the ECHO Autism hub team experts with guidance from the MU School of Medicine Continuing Medical Education and Physician Lifelong Learning staff. The survey is divided into four separate pre-determined themes or categories: Improved self-efficacy (4 questions), knowledge increase (3 questions), enjoyment of the learning collaborative (5 questions), and other (question regarding effectiveness of technology and conflict of interest disclosure). The survey is a part of a program accredited by the Office of Continuing Medical Education and Physician Lifelong Learning and is consistent with the guidelines of the Accreditation Council for Continuing Medical Education (ACCME).

Fourteen questions used a 5-point Likert scale: 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree, and one question (conflict of interest disclosure) was a yes/no question.

Results

A total of 87 live-interactive ECHO Autism sessions were held during the study period, with 83 expert didactic presentations delivered (Table 2). A total of 261 unique learners from 37 US states and 9 countries attended, in addition to 229 learners from Missouri. Missouri learners came from 29 different counties and the independent city of St. Louis, making the ECHO Autism reach 26 percent of the 113 total Missouri counties. In addition, Missouri learners attended an average of 10.5 sessions, while other learners only attended an average of 3.7 sessions each. Students and residents were not required to attend ECHO Autism sessions, they chose to attend during their elective rotations. Non-Missouri learners were a part of different research study requiring them to attend at least three ECHO Autism sessions, which may have affected their lower attendance average.

Table 2.

ECHO Autism Learners.

ECHO Autism
Date	Sept 2, 2015–June 5, 2019
Number of live-interactive sessions	87
Number of hours of interaction	131.5
Number of didactic presentations	83
Number of real-life de-identified patient case discussions	92
Number of unique learners	490 learners
Total number of learners	2,262 learners
Total contact hours (1.5 h/session)	3,394.5 h
	Non-MO	MO
Number of unique learners	261	229
Total number of learners	648	1,614
Total contact hours (1.5 h/session)	972	2,421
Geographic reach	37 US states + 9 counties	29 counties + STL

Four hundred ninety (490) unique attendees attended multiple sessions during the study period and claimed a total of 886.3 CME hours were received (Table 2). After each completed session, they received a link to post-virtual clinic survey, and a total 672 surveys was received: 50 respondents did not claim any CME credits, 9 claimed less than 1 h of CME, with the average of 1.4 h of CMEs claimed. There was a total of 35.3 CME hours claimed from September 2, 2015–December 21, 2015, 239.4 in 2016, 306.1 in 2017 (50 percent increase from previous year), 201.75 (44 percent decrease from previous year) and 49.75 from January 1, 2019–June 5, 2019 (Table 3). Two overlapping occurrences might have affected this observed decreases in CMEs claimed – practicing providers need to claim 50 credits in a 24 month period, and are likely not completing surveys after they have claimed the required credits, and the MTN modified timing of the survey collection in 2018 from annual to bi-annual.

Table 3.

CME Hours.

Year	Sessions	Unique attendees	Hours of interaction	Survey respondents	CME hours awarded
2015	8	61	159	22	21
2016	22	149	784.5	190	142.5
2017	22	174	1,015.5	262	284.7
2018	24	177	987	159	149.75
2019	11	103	451.5	37	42.5
Total	87	478	3,394.5	672	640.45

Discussion

Theme: Improved Self-efficacy

Self-efficacy is described as the perception of one's abilities to “perform a particular behavior in a specific situation” (21,22). Enhancing healthcare provider self-efficacy is linked to improved ability to diagnose and provide education to at-risk patients (23).

The CME survey in this study included four questions in the self-efficacy theme (Table 4). Over 98% of respondents agreed or strongly agreed that their self-efficacy in identifying symptoms of autism in children increased, and almost 96% agreed or strongly agreed that they increased self-efficacy in treating common medical comorbidities. The questions that received the lowest self-efficacy score, 91.9%, was assessing and treating common psychiatric comorbidities.

Table 4.

Survey Response Frequencies.

QUESTION	Mean	Standard deviation	Mode
Improved self-efficacy (SE)	4.61	0.62	5
1. Increased SE in identifying autism symptoms in children	4.67	0.53	5
2. Increased SE in assessing and treating common medical comorbidities	4.60	0.61	5
3. Increased SE in assessing and treating common psychiatric comorbidities	4.56	0.68	5
6. Better able to care for patients with autism and behavioral health concerns	4.61	0.63	5
Knowledge increase	4.67	0.57	5
4. Learned new information	4.61	0.60	5
7. Learning best-practice care	4.74	0.52	5
14. Likely to have an impact on practice	4.67	0.59	5
Enjoyment of the learning collaborative	4.78	0.50	5
5. Effective way to learn	4.73	0.52	5
8. Connected with peers whose opinion I respect	4.74	0.58	5
9. I respect the knowledge of the specialists	4.85	0.41	5
11. Knowledgeable presenters	4.81	0.47	5
13. The presenters allowed feedback and discussion	4.79	0.47	5
Other	4.59	0.68	5
10. Effective technology	4.58	0.65	5
15. Conflict of Interest Disclosure (CID) made	4.59	0.72	5

Theme: Enjoyment of the Learning Collaborative

Five questions were in the “Enjoyment of the learning collaborative” theme of the survey: “Effective way to learn,” “Connected with peers whose opinion I respect,” “Knowledgeable presenters,” and “The presenters allowed feedback and discussion” (Figure 1). These questions were all scored primarily as strongly agree or agree. Only one question (Knowledgeable presenters) received 1% of strongly disagree responses. Both Knowledgeable presenters and I respect the knowledge of the specialists both received over 99% of agree and strongly agree responses.

Figure 1.

Survey responses.

Theme: Knowledge Increase

Three questions were in the “Knowledge increase” theme of the survey: “Learned new information,” “Learning best-practice care,” and “Likely to have an impact on practice” (Figure 1). Over 95 percent of respondents indicated that they agree or strongly agree that what they learned will have impact on their practice. In addition, almost 98 percent of respondents agreed or strongly agreed that they learned best-practice care in ECHO Autism.

Conclusion

The goal of continuing medical education is to “maintain and improve clinical performance” (24,25). Medical practices and the way care is delivered continue to evolve and change, as healthcare providers increasingly adopt telehealth or hybrid models. Similarly, distance continuing medical education has become a more widely accepted model for busy providers, as it allows them to maximize their clinic time and access education virtually.

The goal of this study was to evaluate the perception of the value of a virtual and collaborative learning ECHO Autism network. We completed the study by analyzing three themes in the post-virtual clinic survey: enjoyment of the learning collaborative, knowledge increase, and self-efficacy. A total of 490 learners completed the survey, which was distributed after each of 87 live-interactive sessions held from 2015 to 2019.

We found an overwhelmingly positive responses to all survey questions, indicating that respondents value the ECHO Autism platform as a learning tool. Findings from this study provide insight that aids in future ECHO Autism session design and implementation. For example, the expert hub team will focus additional attention to didactics and case presentations that specifically address mental health comorbidities, which was one of the lowest scored responses received.

As one of ECHO pillars is to “move knowledge, not people,” ECHO Autism is centered around sharing resources, expertize, and experiences. This approach may not only have positive effect on participating PCPs knowledge and skills, but also their patients’ healthcare outcomes. PCPs practicing in remote and isolated areas will be better equipped to provide early screenings as they receive guidance from the specialty hub team. Early screening will likely result in access to early intervention for patients, which has been shown to improve cognitive and intellectual ability and reduce interfering behaviors.

Since the Show-Me ECHO project also offers Child Psychiatry ECHO sessions, we recommend that both ECHO Autism and Child Psychiatry ECHOs provide learning opportunities that address this specific topic in order to reach a wider audience of PCPs.

This was a one site study, and the results may not be generalizable to other states with a more diverse participant learner population. In addition, learning needs and expectations from ECHO Autism learners may differ from other ECHO Autism hubs not focusing on rural and underserved populations. ECHO Autism is about pediatric clinical cases brought forward by primary care clinicians. To date, a self-identified autistic primary care clinician has not presented a clinical case. Nor has a self-identified autistic professional shared an experience related to their own early childhood diagnosis or early intervention experience. We incorporate the lived experience caregiver voice on the hub team and integrate the self-advocate voice in ECHO Autism sessions when discussing adolescents or adults on the spectrum.

Until introduction of the ECHO approach, continuing education was primarily held in person or through webinars, focusing on individual, independent learners. ECHO fundamentally changed this landscape, by providing virtual non-judgmental learning networks where “all teach and all learn” (18). Although implemented in many specialties across the world since 2007, the COVID-19 pandemic has required a significant increase in adoption of ECHO and ECHO-like platforms, to facilitate successful continuation knowledge-sharing during the global pandemic.

Autism spectrum disorder (ASD) is one of the fastest growing neurodevelopmental disorders worldwide

Early detection is critical, and can improve child's long-term outcomes

Primary care providers (PCPs) lack adequate training to identify early signs of ASD

PCPs report many barriers, such as limited local diagnostic resources and lack of familiarity with screening tools as reasons for not screening patients

Continuing medical education platforms are increasingly adopting live-interactive virtual sessions

Expert telementoring of primary care providers (PCPs) through Extension for Community Healthcare Outcomes (ECHO) Autism project results in high satisfaction and increased self-efficacy in identification and management of patients with autism

PCPs benefit from de-identified real-life case discussions with peers and autism specialists