Testing Our Children When the World Shuts Down: Analyzing Recommendations for Adapted Tele-Assessment during COVID-19.

In 2017, the National Association of School Psychologists described tele-assessment as the least researched area of telehealth. This became problematic in 2020 when COVID-19 curtailed the administration of face-to-face assessments. Publishers began to offer computer-adapted tele-assessment methods for tests that had only previously been administered in person. Recommendations for adapted tele-assessment practice had to be developed with little empirical data. The current study analyzed recommendations from entities including professional organizations, test publishers, and governmental offices. The samples for each were small, but the findings were noteworthy. Test publishers were unanimous in recommending the use of their face-to-face assessments through adapted tele-assessment methods (either with or without caution). Governmental agencies were more likely to recommend not using adapted tele-assessment methods or to use these methods with caution. Finally, professional organizations were almost unanimous in their recommendations to use adapted tele-assessment but to do so with caution. In addition to deviations in the types of recommendations provided, entities varied in how the information was distributed. About one-fifth (23.5%) of all entities surveyed provided no recommendations at all. About 45% of the remaining entities provided recommendations on their Web sites. The rest provided information through shared documents, online toolkits, peer-reviewed journals, and emails. Implications for the field of psychology's future crisis management planning are discussed in response to these findings.

Psychology is considered a “soft science” (Blumenthal-Barby, 2014, p. 531) in part because the field lacks assessment methods that produce psychometrically precise results. Specifically, psychological assessment results are plagued by errors associated with validity (accuracy), reliability (consistency), and appropriate normative (comparisons) information (Sattler, 2018). The more error in the test results, the more inaccurate are decisions based on those results (Matuszak & Piasecki, 2012). Ideally, to reduce error in assessments, the testing experience should be standardized (e.g., same items, same instructions, and similar setting; Sattler, 2018). However, the recent COVID-19 pandemic has forced the use of instruments designed for traditional, standardized face-to-face testing in a tele-assessment environment, thus increasing the risks of testing errors (Krach & Sattler, 2018).

Practitioners considering the use of adapted tele-assessment methods in this situation seek guidance. Adapted tele-assessment methods are defined here as the administration of assessments that were designed to be administered face to face in a computer-mediated manner. Research related to generic tele-assessment is scarce (National Association of School Psychologists, 2017) but further dwindles to only a handful of studies regarding the specific area of adapted tele-assessment (Wright, 2018a, 2018b, 2020).

Given the dearth of available research, guidance regarding the use of adapted tele-assessment had to come from national associations (e.g., American Psychological Association [APA] and National Association of School Psychologists [NASP]), from governmental regulatory groups (e.g., U.S. Department of Education and US Social Security Administration), and from test publishing companies (e.g., Mental Health Systems [MHS] and Pearson). However, these recommendations, when forthcoming, seemed to differ based on the source and the means of recommendation.

The goal of this article was to analyze the professional recommendations related to adapted tele-assessment disseminated during the early days of the COVID-19 crisis. In particular, the goals were to examine: (1) what types of recommendations were provided, (2) differences between recommendations by the type of entity providing them, and (3) dissemination methods of these recommendations to practitioners in the field. The findings from this study may help current psychologists interpret the recommendations that they have received. More importantly, the findings can help future leaders in the field of psychology to make changes to field-wide planning for crisis management.

Tele-Assessment

The term “telehealth” includes distance-based mental health and physical health services in the areas of assessment, prevention, and intervention. Perle and Nierenberg (2013) described the need for telehealth methods to address shortages of services to people who may not otherwise have had access to them in their communities. But with the onset of the COVID-19 pandemic, the sudden increased need for distance services resulted in an exponential surge in telehealth implementation for all clients. Tele-assessment is no longer limited to the few individuals who were receiving telehealth services before the pandemic (Wosik et al., 2020).

“Telehealth” comprises a broad field. For the purposes of this article, “tele-assessment” is defined as “diagnostic, psychological assessment procedures administered to (an) individual(s) who is not in the same room as the examiner through the use of telecommunication technologies.” Telecommunication technologies include, but are not limited to, telephone conferences, online video conferencing (e.g., Zoom video conferencing), and web-based assessments. Furthermore, as documented by Jordan and Shearer (2019), “information transmitted between patients and providers may be occurring synchronously (occurring in real time) or asynchronously (occurring at different time points) and may include text, graphics, video, sounds, or other data” (p. 2).

Some tests have been specifically designed to be administered using a telecommunication device (e.g., the Gibson Test of Cognitive Skills, 2nd Edition [GTCS-2]; Moore & Miller, 2016); therefore, using them as tele-assessments does not break standardization, and such assessments are not the focus of the current article. Instead, this article examines the recommendations regarding the use of adapted tele-assessment. “Adapted tele-assessment” is the administration using telecommunication technology of tests that were standardized for face-to-face administration. Adaptive tele-assessment creates a higher risk of introducing error because it breaks an instrument’s special standardization, testing environment, and norms. Without the consistency and standardized procedures with which the tests were designed, raw scores can no longer be compared to one another with any assurance of psychometric accuracy (Kline, 2015)

Previous Research on Adapted Tele-Assessment

Ideally, any test adapted to a tele-assessment format from a paper-based and/or a face-to-face method should demonstrate psychometric equivalency. The requirements for psychometric equivalency were provided by the American Psychological Association Committee on Professional Standards and Committee on Psychological Tests and Assessment (1986) in their Guidelines for Computer-Based Tests and Interpretations and the joint commission Standards for Educational and Psychological Testing (American Education Research Association [AERA], American Psychological Association [APA], & National Counsel on Measurement in Education [NCME], 2014). These equivalency standards require the following: (a) inter-version equivalency correlations, (b) mean score differences that are not statistically significantly different with small effect sizes, and (c) the score dispersion shapes are not statistically significantly different from one another. Two additional guidelines should also be considered: (d) demographic equivalency between the study sample and the original normative sample (Grosch, Gottlieb, & Cullum, 2011; Hodge et al., 2019; Krach, McCreery, Dennis, Guerard, & Harris, 2020) and (e) sufficient sample size to ensure the statistical power to perform the equivalency analyses (Cohen, 1988; Farmer et al., 2020a). All these criteria must be demonstrated to state definitively that two forms of a test are equivalent.

Unfortunately, the literature is scarce regarding the use of face-to-face, diagnostic, direct assessments through an adapted tele-assessment method. Only the following tests have been evaluated for this purpose: the Woodcock-Johnson Tests of Cognitive and Achievement, Fourth Edition (WJ-IV: COG and WJ-IV: ACH; Wright, 2018b), the Reynolds Intelligence Assessment Scale (RIAS; Wright, 2018a), and the Wechsler Intelligence Scale for Children, Fifth Edition (WISC-V; Wright, 2020). None of these studies met all the requirements set forth to definitively support version equivalency.

Although equivalency research examining Pearson’s paper-based and Q-Interactive (Qi) methods (Daniel, 2012a, 2012b, 2013a, 2013b, 2013c; Daniel, Wahlstrom, & Zhang, 2014a; Daniel, Wahlstrom, & Zhou, 2014b) exists, these studies have all been performed using in-person (face-to-face) and not adapted tele-assessment methods (Cayton, Wahlstrom, & Daniel, 2012). In these studies, both the test taker and test administrator were in the same room.

Challenged Beliefs and Practice (COVID-19)

COVID-19 challenged entities and individuals to change their minds and their practices regarding adapted tele-assessment (Farmer et al., 2020a, 2020b). One clear example comes from the NASP. In their Considerations for Delivery of School Psychological Telehealth Services (2017), NASP provided a list of concerns related to tele-assessment, specifically reporting, “most publishers do not currently have well-established or well-vetted systems established for conducting assessments virtually.” Their recommendation at that time was that school psychologists only use “validated assessment tools and methods” (pgs. 4–5, & 25). Given the findings discussed above, none of the adapted tele-assessments described in this article would meet the criteria. In essence, NASP stated that adapted tele-assessment should not be used. They reiterate this when stating that adapted tele-assessment “results may not hold up in a legal proceeding, since test construction and norming samples did not include a sample of those who were administered the assessments remotely” (p. 22).

In 2020, during the height of the COVID-19 crisis, NASP provided an updated set of recommendations for virtual service delivery. In this updated document, they emphasized caution in administering and interpreting assessment results provided through adapted tele-assessment. Specifically, they state that “high-quality evidence” is needed prior to using any assessment adapted for delivery using only “platforms designed for that purpose” (p. 2). They add that any alteration to the standardized form of assessment should be considered carefully when making diagnostic decisions and clearly documented in any resulting assessment report. Therefore, they move from stating not to use traditionally available tests as tele-assessments (National Association of School Psychologists, 2017) to suggesting cautious use.

Farmer and colleagues (2020) go into additional detail as to how these types of policy/recommendation shifts manifest. A shift may have occurred when practitioners were faced with limited choices for conducting direct assessments unless they adapted to tele-assessment methods. Therefore, NASP’s position altered when practitioners were only given two choices: (1) delay Individuals with Disabilities Education Improvement Act [IDEA] (2004) child find requirements or (2) use methods of questionable validity.

How Information Was Disseminated during COVID-19

Initial recommendations regarding adapted tele-assessment became available in mid-March 2020. On March 16, 2020, the Office of Civil Rights posted a white paper to their Web site suspending face-to-face testing for all P-12 schools until schools reopened. Shortly after, on March 30, Pearson provided a webinar that gave step-by-step directions on how to use their Q-Interactive products in an adapted tele-assessment format (Henke, 2020). Pearson’s webinar indicated no restrictions to the use of their products. In addition, on April 3, the American Psychological Association (APA, Wright, Mihura, Pade, & McCord, 2020) published an article on its Web site reiterating that the use of face-to-face tests in adapted tele-assessment was fine as long as caution was used when interpreting the results. On the other hand, the American Psychiatric Association [ApA] (2020) produced a similar white paper indicating no restrictions at all. For practitioners, it must have seemed that adapted tele-assessment recommendations would change depending on what you read, when you read it, who stated it, and how you received it.

These recommendation differences continued as school buildings remained closed and as states extended social distancing recommendations into the summer and fall of 2020, and the methods by which practitioners might access these recommendations varied widely as well. Distribution to practitioners might best be described as haphazard. Some recommendations were provided as announcements on the entities’ Web sites (e.g., MHS; Wheldon, 2020), some were available through multimedia presentations (e.g., podcasts, webinars, and virtual workshops; e.g., Henke, 2020; Sharp, Mcfadden, & Morera, 2020), and still others were shared documents stored in cloud drives (e.g., Wyoming State Board of Education, 2020). Methods of distributing the recommendations included advertisements and sharing on professional listservs, through online NASP communities and through personal and professional email correspondence.

The current authors asked three main research questions related to these recommendations: (1) what information did practitioners receive regarding adapted tele-assessment methods for tests that were traditionally administered as face-to-face, in-person assessments, (2) was there a difference in the recommendation regarding adapted tele-assessment based on what type of entity was providing the information (i.e., test publisher, government agency, or professional organization), and (3) how were recommendations disseminated during the time of the COVID-19 crisis?

Methods

Entities

Three categories of entity recommendations were evaluated. Psychological and mental health–related professional associations (e.g., APA and NASP) were chosen at the national and state levels. Test publishers were chosen if they provide commonly used diagnostic, direct assessments (e.g., Pearson and MHS). Finally, government agencies were chosen if they provide funding or legal guidelines for psychological services (e.g., U.S. Departments of Education and US Social Security Department). All related national professional organizations and test publishers were included.

In addition to the national entities described above, a stratified randomized sample of 15 additional state entities was also included. Given that COVID-19 has been reported as a national pandemic by the Center for Disease Control (2020, https://www.cdc.gov/), a nationally differentiated sample was chosen. In preparation for doing a chi-squared analysis, each group needed a minimum of five entities per cell (McHugh, 2013). Therefore, using US Census data, five state governmental and/or professional agencies were randomly selected by using a random number generator after separating states into one of three potential groups: (1) the 10 states with the highest populations, (2) the 10 states with the lowest populations, and (3) the states whose populations did not fall in either of these two categories.

Procedures

Recommendations were identified through a comprehensive Google search of entity Web sites, announced trainings/webinars through listservs, Facebook links from psychological assessment groups, and email responses directly from the entity. Information was pulled only from sources that were approved agents for disseminating information on behalf of the entity involved. Direct quotes related to adapted tele-assessment were pulled from all the following types of sources: emails, peer-reviewed and non-peer–reviewed journal articles, documents published online, official organizational webpages, blog and news posts, and toolkits specifically dedicated to the dissemination of COVID-19 resources and information. All data were collected between May 1 and July 1 of 2020. Table 1 provides information on the type of information evaluated as well as how the information was disseminated.

Table 1.

Visual Analysis of Recommendations.

	Severity Categorization
Entity	Do Not Use	Caution	No Concern	No Advice
Professional organizations
ApA1		X
ApA2				X
ApA3				X
ApA4		X
APA5		X
APA6		X
APHA7				X
CRS8			X
NASP9		X
NRHSP10				X
NRHSP11				X
California ASP12	X
Louisiana SPA13	X
Montana ASP14		X
ASP of Pennsylvania15	X
South Carolina ASP16	X
Tennessee ASP17	X
Wisconsin SPA18	X	X
Government agencies
Alaska DEED19	X	X
Alaska DEED20		X
Alaska DEED21		X
Alaska DEED22		X
Idaho SESTA23
Louisiana DOE24	X	X
Louisiana DOH25		X
Louisiana state BEP26
Maryland BEP27	X
Maryland public schools28	X
Mississippi DOE29	X
Montana OPI30	X
Nebraska DOE31	X
Public schools of North Carolina32	X	X
North Carolina psychology board33
Commonwealth of Pennsylvania34				X
Pennsylvania DOE35	X	X
South Carolina BEP36
South Carolina DOE37	X
Tennessee DOE38	X
U.S. DOE39	X
U.S. VA40				X
U.S. VA41				X
U.S. VA42				X
U.S. Social security administration43				X
Centers for medicare and medicaid services44		X		X
Virginia DOE45
Wyoming SBP46		X		X
Test publishers
PAR47		X
Pearson/PsychCorp48			X
Riverside/Harcourt49		X
MHS50			X
WPS51		X

Note 1. ApA = American Psychiatric Association; APA = American Psychological Association; APHA = American Public Health Association; CRS = Congressional Research Service; NASP = National Association of School Psychologists; NRHSP = National Register of Health Service Psychologists; ASP = Association of School Psychologists; SPA = School Psychological Association; DEED = Department of Education and Early Development; SESTA = Special Education Support and Technical Assistance; DOE = Department of Education; BEP = Board of Examiners of Psychologists; DOH = Department of Health; OPI = Office of Public Instruction; VA = Department of Veteran’s Affairs; SBP = State Board of Psychology.

Note 2. Hilty et al. (2016)1; American Psychological Association [ApA] (2020)2; American Psychological Association [ApA] & American Telehealth Association (2018)3; Hafften (2016)4; Wright et al. (2020)5; Joint Task Force for the Development of Telepsychology Guidelines for Psychologists, 20136; American Public Health Association [APHA] (2020)7; Elliott (2020)8; National Association of School Psychologists (2020)9; Alvord, Siegel, and Harris (2019)10; Dalton (2020)11; California Association of School Psychologists [CASP] (2020)12; Louisiana School Psychological Association [SPA] (2020)13; Montana Association of School Psychologists [ASP] (2020)14; Association of School Psychologists [ASP] of Pennsylvania (2020)15; South Carolina Association of School Psychologists [ASP] (2020)16; Tennessee Association of School Psychologists [ASP] (2020)17; Wisconsin School Psychological Association [SPA] (2020)18; Alaska Department of Education and Early Development [DEED] (2020d)19; Alaska Department of Education and Early Development [DEED] (2020a)20; Alaska Department of Education and Early Development [DEED] (2020b)21; Alaska Department of Education and Early Development [DEED] (2020c)22; Idaho Special Education Support and Technical Assistance [SESTA] (2020)23; Louisiana Department of Education [DOE] (2020)24; Louisiana Department of Health [DOH] (2020)25; Louisiana State Board of Examiners of Psychologists [BEP] (2015)26; Maryland Board of Examiners of Psychology [BEP] (2018)27; Maryland Public Schools (2020)28; Mississippi Department of Education [DOE] (2020)29; Montana Office of Public Instruction [OPI] (2020)30; Nebraska Department of Education [DOE] (2020)31; Public Schools of North Carolina (2020)32; North Carolina Psychology Board (2020)33; Commonwealth of Pennsylvania (2010)34; Pennsylvania Department of Education [DOE] (n.d.)35; South Carolina BEP (2020)36; South Carolina Department of Education [DOE] (n.d.)37; Tennessee Department of Education [DOE] (2020)38; United States Department of Education (2020)39; United States Department of Veteran Affairs (2017)40; United States Department of Veterans Affairs (2018)41; Lum and Tambyah (2020)42; Verma (2020)43; Centers for Medicare and Medicaid Services (2020)44; Virginia Department of Education [DOE] (2020)45; Wyoming State Board of Psychology (2020)46; PAR (2020)47; Henke (2020)48; Walker et al. (2020)49; Wheldon (2020)50; WPS, 202051.

Instrumentation

Categorical determination analysis was conducted by a team of four individuals. One of the team members is a professor in the field of school psychology. Two of the others are psychology doctoral students who had completed a minimum of six semester hours of coursework on assessment administration and interpretation. The final rater is a graduate from an Ed.S. degree program in Clinical Mental Health Counseling who previously completed a minimum of three semester hours of coursework and 12 semester hours of internship on assessment administration and interpretation.

A first round of data analysis was completed by the team. In this first round analysis, all four team members examined a subset of about 20 quotes on their own and assigning a descriptor/label to each. These descriptors/labels were compared thematically. Five categories were identified from these themes in this first round analysis. Three of these five categories remained unchanged for the final analysis: (1) “do not use,” (2) “use with no concerns/restrictions,” and (3) “no advice given.” In the first-round analysis, two additional categories listed “mild caution” and “moderate caution” were initially identified. Due to the lack of interrater agreement, these two categories were condensed into a single category of “caution.” Raters then developed an agreed-upon set of rules/definitions for each of the four categories prior to the final analysis.

The final four categories were: (1) do not use, (2) use with no concerns/restrictions, (3) use with caution, and (4) no advice given. The rules/definitions for each of the categories are as follows. For the “no advice given” category, the entity may or may not have provided guidance on telehealth, but they were included in this category if the advice was not specific to direct, psychology-based, adapted tele-assessment. For the “no concerns/no restrictions” category, the entity must have provided guidance on adapted tele-assessment with no caveats as to how the tests should be used. For the “do not use” category, the entity explicitly stated not to use this method of assessment. Finally, the “use with caution” category was chosen if adapted tele-assessment was discussed with caveats about the administration and/or the interpretation of the tests. Provided are two specific examples of categorical decisions.

Example 1 comes from a fact sheet posted on the Web site for the United States Department of Education (2020, pg. 3). “If an evaluation of a student with a disability requires a face-to-face assessment or observation, the evaluation would need to be delayed until school reopens.” In this example, the recommendation was categorized as do not use.

Example 2 comes from the Web site position paper for the APA (Wright et al., 2020, para 2). In this situation, the recommendation was categorized as caution:

However, the situation is more challenging with assessment services that have standardized administration procedures that require in-person contact. In considering these challenges, some psychologists may choose to pause their psychological assessment services during this time; however, there are others who do time-sensitive, high-need, and/or high-stakes assessments that really need to continue. Most current and emerging telehealth guidelines largely focus on psychotherapy, and as such, tele-assessment guidance is necessary.

Each statement was read aloud to the group in a Zoom meeting. The members submitted their votes to the professor once after she had voted. Votes were submitted through the Zoom chat feature with permission set for only the professor to view the responses. For the four categories, first-round, interrater, unanimous agreement was 90% (46 out of 51 cases). The remaining five cases had only one or two of the five members in disagreement. When this happened, the final decision was based on the majority opinion.

Analysis

What recommendations were provided?

To address the first research question, findings were tabulated in a graphical manner for easy reference (Tables 1 and 2).

Table 2.

Frequency Counts for Recommendations by Type of Entity.

	Entity Type
Recommendation	Professional Organization	Government Agency	Test Publisher	Total
No advice	5	7	0	12
Do not use	5	12	0	17
Caution	7	9	3	19
No concern	1	0	2	3
Total	18	28	5	51
Total removing no advice	13	21	5	39

Was there a difference in recommendations across entities?

To address the second research question, a chi-square analysis was attempted with entity type as the grouping variable: professional organizations, government agencies, and test publishers. The categorical variable included: (1) do not use, (2) use with no concerns/restrictions, and (3) use with caution. However, to run a chi-square analysis, two assumptions must first have been met (McHugh, 2013): (1) a frequency of at least one response per cell and (2) 80% or more cells should have a frequency of at least five.

The first chi-square assumption was violated due to issues with the publisher and the government agency grouping variables. None of the test publishers indicated “do not use” in their recommendations, creating the first null set. In addition, none of the government agency entities indicated that they had “no concerns,” resulting in a second null set. The second assumption was also violated in that 55.6% of the cells had counts of less than five per cell. Given the nature of these recommendations, it is possible that collecting additional data would continue to result in at least one null set. So, instead of running the chi-square analysis, a graphical representation of the data was provided in a cross-tabulation in Table 2. Percentages were calculated from all entities that provided advice and reported in the results section.

How were the recommendations disseminated?

Tables 3 and 4 provide information about the manner in which data were provided. From this, percentages were calculated and presented in the results section.

Table 3.

Recommendation Dissemination Method.

Entity	COVID-19 Specific?	Material Published	Dissemination Method
Professional organizations
ApA1	No	Journal article	Peer-reviewed journal
ApA2	Yes	Guidelines/recommendations	Toolkit
ApA3	No	Guidelines/recommendations	Webpage, publication
ApA4	No	Video recording	Toolkit
APA5	Yes	Blog/news	Webpage
APA6	No	Journal article	Peer-reviewed journal
APHA7	Yes	Facts	Toolkit
CRS8	No	FAQ	Document and congressional report
NASP9	Yes	Guidelines/recommendations	Webpage and toolkit
NRHSP10	No	Video recording and blog/news	Webpage
NRHSP11	Yes	Blog/news	Webpage
California ASP12	Yes	Position paper	Webpage and document
Louisiana SPA13	Yes	Position paper	Webpage and document
Montana ASP14	Yes	Statement	Email
ASP of Pennsylvania15	Yes	Statement	Email
South Carolina ASP16	Yes	Statement	Email
Tennessee ASP17	Yes	Guidelines/recommendations	Document
Wisconsin SPA18	Yes	Statement	Webpage and document
Eligibility entities/Service providers
Alaska DEED19	Yes	Guidelines/recommendations	Webpage
Alaska DEED20	No	Guidelines/recommendations	Document and email
Alaska DEED21	Yes	FAQ	Document and email
Alaska DEED22	Yes	Video recording	Webpage
Idaho SESTA23	Yes	Guidelines/recommendations	Toolkit and Document
Louisiana DOE24	Yes	Guidelines/recommendations	Document
Louisiana DOH25	Yes	Guidelines/recommendations	Toolkit and document
Louisiana state BEP26	No	Guidelines/recommendations	Webpage and document
Maryland BEP27	No	Policy/regulation/ethical code	Webpage and document
Maryland public schools28	Yes	FAQ	Document
Mississippi DOE29	Yes	Guidelines/recommendations	Toolkit
Montana OPI30	Yes	Guidelines/recommendations	Toolkit
Nebraska DOE31	Yes	FAQ	Document
Public schools of NC32	Yes	FAQ	Document and toolkit
NC psych board33	Yes	FAQ	Web site
Commonwealth of PA34	No	Policy/regulation/ethical code	Document
Pennsylvania DOE35	Yes	FAQ	Webpage
South Carolina BEP36	No	Statement	Document
South Carolina DOE37	Yes	FAQ	Toolkit
Tennessee DOE38	Yes	Guidelines/recommendations	Document
U. S. DOE39	Yes	Facts	Document
U. S. VA40	No	Facts	Document
U. S. VA41	No	Blog/News	Webpage
U.S. VA42	Yes	Journal article	Peer-reviewed journal
U. S. SSA43	Yes	Blog/news	Webpage
Centers for medicare & medicaid44	Yes	Facts and blog/news	Document and webpage
Virginia DOE45	Yes	FAQ	Webpage
Wyoming SBP46	Yes	Guidelines/recommendations	Document and webpage
Test publishers
PAR47	Yes	Guidelines/recommendations	Webpage and toolkit
Pearson/PsychCorp48	Yes	Video recording	Webpage
Riverside/Harcourt49	Yes	Video recording	Webpage
MHS50	Yes	Statement	Document and email
WPS51	Yes	Statement	Webpage

Note 1. Journal article = published, peer-reviewed article; guidelines/recommendations = specific advice given to be used by practitioners and parents; video recordings = webinars, conference presentations, and other recorded video or audio; blog/news = blog and news posts; facts = fact sheets and webpages; FAQ = frequently asked questions; position paper = position of entity regarding tele-assessment; statement = statement by entity on tele-assessment or call to action on tele assessment; policy/regulation/ethical code = legal and ethical guidance on tele-assessment.

Note 2. Email = email communications with stakeholders; peer-reviewed journal = published article that was peer reviewed; publication = article published by an entity that was not in a peer-reviewed journal; document = working documents, PDFs, word documents, etc. published online; webpage = webpages, Web sites, and blogs; toolkits = webpages specifically dedicated to the dissemination of COVID-19 resources and information.

Note 3. ApA = American Psychiatric Association; APA = American Psychological Association; APHA = American Public Health Association; CRS = Congressional Research Service; NASP = National Association of School Psychologists; NRHSP = National Register of Health Service Psychologists; ASP = Association of School Psychologists; SPA = School Psychological Association; DEED = Department of Education and Early Development; SESTA = Special Education Support and Technical Assistance; DOE = Department of Education; BEP = Board of Examiners of Psychologists; DOH = Department of Health; OPI = Office of Public Instruction; VA = Department of Veteran’s Affairs; SBP = State Board of Psychology.

Note 4. Hilty et al. (2016)1; American Psychological Association [ApA] (2020)2; American Psychological Association [ApA] & American Telehealth Association (2018)3; Hafften (2016)4; Wright et al. (2020)5; Joint Task Force for the Development of Telepsychology Guidelines for Psychologists, 20136; American Public Health Association [APHA] (2020)7; Elliott (2020)8; National Association of School Psychologists (2020)9; Alvord, Siegel, and Harris (2019)10; Dalton (2020)11; California Association of School Psychologists [CASP] (2020)12; Louisiana School Psychological Association [SPA] (2020)13; Montana Association of School Psychologists [ASP] (2020)14; Association of School Psychologists [ASP] of Pennsylvania (2020)15; South Carolina Association of School Psychologists [ASP] (2020)16; Tennessee Association of School Psychologists [ASP] (2020)17; Wisconsin School Psychological Association [SPA] (2020)18; Alaska Department of Education and Early Development [DEED] (2020d)19; Alaska Department of Education and Early Development [DEED] (2020a)20; Alaska Department of Education and Early Development [DEED] (2020b)21; Alaska Department of Education and Early Development [DEED] (2020c)22; Idaho Special Education Support and Technical Assistance [SESTA] (2020)23; Louisiana Department of Education [DOE] (2020)24; Louisiana Department of Health [DOH] (2020)25; Louisiana State Board of Examiners of Psychologists [BEP] (2015)26; Maryland Board of Examiners of Psychology [BEP] (2018)27; Maryland Public Schools (2020)28; Mississippi Department of Education [DOE] (2020)29; Montana Office of Public Instruction [OPI] (2020)30; Nebraska Department of Education [DOE] (2020)31; Public Schools of North Carolina (2020)32; North Carolina Psychology Board (2020)33; Commonwealth of Pennsylvania (2010)34; Pennsylvania Department of Education [DOE] (n.d.)35; South Carolina BEP (2020)36; South Carolina Department of Education [DOE] (n.d.)37; Tennessee Department of Education [DOE] (2020)38; United States Department of Education (2020)39; United States Department of Veteran Affairs (2017)40; United States Department of Veterans Affairs (2018)41; Lum and Tambyah (2020)42; Verma (2020)43; Centers for Medicare and Medicaid Services (2020)44; Virginia Department of Education [DOE] (2020)45; Wyoming State Board of Psychology (2020)46; PAR (2020)47; Henke (2020)48; Walker et al. (2020)49; Wheldon (2020)50; WPS, 202051.

Table 4.

Information Dissemination Methods.

	Dissemination Method
Type	Webpage	Document	Toolkit	Journal	Email
Professional organization (n = 18)	8	6	4	2	3
Government agencies (n = 28)	11	17	6	1	2
Test publisher (n = 5)	4	1	1	0	1
Total recommendations	23	24	12	3	6

Note. Total number of entities = 51.

Results

What Recommendations Were Provided?

Tables 1 and 2 provide information on the types of recommendations that were provided.

Was There a Difference in Recommendations across Entities?

About half (53.8%) of professional organizations suggested caution when using adapted tele-assessments; whereas, over a third (38.5%) recommended that these types of assessments not be used at all. Over half (57.1%) of government agencies recommended that practitioners not use adapted tele-assessments and 42.9% recommended that they be used with caution. The majority (60%) of test publishers suggested caution when using their instruments, with fewer than half (40%) indicating no concerns. Test publishers never advised against the use of their products. Government agencies never advised the use of adapted tele-assessment without concerns.

How Were the Recommendations Disseminated?

Given that some entities provided recommendations through multiple methods, the percentages will not equal 100%. For professional organizations, 44.4% provided information on their Web sites compared to 39.3% of government agencies and 80% of test publishers. Tables 3 and 4 provide more specific information on other methods.

Discussion

At this time, no formal ethical codes exist specific to the use of adapted tele-assessment. The American Psychological Association’s (APA) (2017)Ethical Principles of Psychologists and Code of Conduct, Standard 9.02, provides some guidance related to adapted assessments in general. They emphasize that any adapted instrument should be used in a manner supported by research-based evidence and add the need to disclose any issues related to score reliability, test validity, or other potential limitations. In 2014, Luxton, Pruitt, and Osenback published “Best Practices for Remote Psychology Assessment via Telehealth Technologies.” In this article, the authors mention the need to maintain standardization when adapting existing measures for use in a tele-assessment, recommending to choose a different assessment method if standardization cannot be maintained. They specifically list in-person, Wechsler tests as poor candidates for tele-assessment adaptation.

Teleconferencing options have changed since this publication, as have the reasons for the need for adapted tele-assessment, so practitioners should depend on more current recommendations for how to move forward. To fill this gap, about four-fifths of the entities evaluated in this study offered some official statement on adapted tele-assessment; all of the test publishers had an official statement. In general, these recommendations fell in three categories: “do not use,” “use with caution,” or “use with no restrictions.” Unfortunately, the recommendations were inconsistent across and within entity type (i.e., professional organizations, governmental agencies, and test publishers). Government agencies were unanimous in their recommendation that adapted tele-assessment be conducted with some caution or not at all. Test publishers were unanimous in their recommendation that adapted tele-assessment with their instruments be conducted (either with or without caution).

One hypothesis for these findings is that use decisions are, in some part, made based on the financial stake held by each type of entity (Vitell, Singhapakdi, & Thomas, 2001). For example, test publishers are likely to lose money if no online equivalent to face-to-face testing is available for their products. Therefore, it would make sense that they would recommend adapting instruments for tele-assessment. Second to test publishers in terms of potential financial loss are assessment practitioners. If practitioners cannot provide assessments, they have the potential to lose some or all of their incomes. Possible financial loss may apply more to private practitioners; however, given that the only national, legal requirement for districts to hire school psychologists over other mental health practitioners involves testing (Individuals with Disabilities Education Improvement Act [IDEA], 2004), this could potentially apply to school-based practitioners as well. Although individual practitioners were not evaluated in this study, the professional organizations that represent these practitioners were included. These organizations must balance the financial/practical needs of their members with the ethical needs of their clients. Therefore, it would make sense that they would recommend using adapted tele-assessments but with caution.

Finally, governmental agencies may have more to lose from potential lawsuits resulting in any deviation from established policy and/or regulatory procedures, so it makes sense they would advise against using deviations from standard practice, such as adapted tele-assessment. However, without specific data regarding the rationale behind each recommendation, the reasons for these differences cannot be determined for certainty.

Another, more troubling finding from the evaluations of these recommendations is that there appears to be no systematic method of disseminating recommendations to professionals in a time of crisis. Table 3 shows that although many of the proffered recommendations were specific to the pandemic, many were recycled from previous, more generic policy statements. In addition, often there was no single source that individuals could use to find what they needed. Information came from many different sources, such as Web sites, congressional reports, emails, video recordings, blogs, journal articles, and stored documents. This is problematic, especially when the same entity may offer contradictory recommendations within their own publicly available materials (e.g., Alaska DEED). Given this, entities may need to reevaluate the information dissemination portions of their crisis management plans. If such a plan does not exist, then the current writers suggest that one needs to be forthcoming.

Call for Future Research

In 2017, NASP described tele-assessment as, “the least explored area of service in telehealth” (pg. 7). This still holds true in 2020. There is no current research on how many practitioners are using tele-assessment methods nor is there any into how practitioner and/or clients view adapted tele-assessment techniques. It is clear that additional study is needed, while social distancing restrictions are present.

In addition, if practitioners continue to use adapted tele-assessment methods post-COVID-19 closures, further equivalency studies are a necessity. The field of tele-assessment is relatively new (Brearly et al., 2017), but it does pre-date COVID-19. Barak was evaluating easily available tele-assessment tools as early as 1999 because psychologists have been looking for ways to reach individuals who live a considerable distance from generic and specialized services. Brearly et al. (2017) provided a meta-analysis on methods of providing neuropsychological tele-assessments. Cobb and Sharkey (2007) analyzed different technology modalities to assess individuals with complex motor disabilities. McCreery et al. (2019) investigated game-based technology as a direct method of assessing behaviors that had only previously been tested indirectly. Finally, the possible use of computers as translators (Karpińska, 2017) or as mediation devices for administering translated tests is an enticing concept (Farmer et al., 2020b).

Such studies must examine multiple aspects of equivalency, including: (a) inter-version correlation, (b) mean score differences, (c) score distribution equivalences (American Educational Research Association [AERA] et al., 2014), (d) sample demographic and setting equivalencies (Grosch et al., 2011; Henke, 2020; Hodge et al., 2019; Krach et al., 2020), and include (e) a large enough sample size (Cohen, 1988; Farmer et al., 2020a). Currently, no published studies adequately meet these criteria. In addition to psychometric equivalency, consequential equivalency/validity should also be evaluated. Consequential equivalency would examine if the score variability between administration formats contributes to different diagnostic conclusions (Matuszak & Piasecki, 2012; Ruskin et al., 1998).

Limitations

Unfortunately, there was not enough statistical power available to accurately run a chi-square analysis with this sample, so no statistical significance results are available. However, given the null sets provided by both test publishers and government agencies, it is possible that having a larger sample of groups would still result in violations in chi-square assumptions.

Also problematic is the in-flux nature of adapted tele-assessment recommendations. The current analysis provides a snapshot of how entities responded during the early months of the crisis. It is possible/probable that the recommendations evaluated in this study have already been replaced by new guidelines as of the print date for this article. Problems associated with changing recommendations are compounded by the fact that the field of psychology does not have a singular distribution method for crisis information.

One final concern about this study is that the recommendations here are specifically focused on assessments traditionally conducted by clinical and school psychologists. It is possible that these findings may differ for other telehealth areas (e.g., counseling and consultation) or other tele-assessment methods (e.g., career counseling and neuropsychological assessment).

Conclusion

Given the perceived rarity of the need for tele-assessment services, any push for adapted tele-assessment research was minimal prior to COVID-19. Tele-assessment has been a convenient way to provide assessments at reduced cost to individuals who may not have traditional access to psychological diagnostic options (National Association of School Psychologists, 2017; Luxton, Pruitt, & Osenback, 2014), including rural or homebound clients (Carlbring et al., 2007). Tele-assessment has also been an expedient way to provide services by specialized practitioners (e.g., bilingual psychologists or autism specialists) who are located some distance from clients. These were considered useful but inessential tools for practitioners, until the need naturally exploded when most direct assessments could no longer safely be administered in a face-to-face manner due to the global pandemic. The ensuing research gap has resulted in the widespread use of adapted tele-assessment, but the latter’s lack of standardization potentially renders invalid any results from such assessment.

At the time of this writing, COVID-19 is still considered a public crisis. Decisions by school districts and individual psychologists regarding adapted tele-assessment are ongoing and ever-changing. It is no easy task for practitioners to make decisions regarding tele-assessment, given that there is no singular directive providing a clear map for adapted tele-assessment practice. This lack of clear direction puts both practitioners and the general public at risk.

When practitioners base their practice on not using adapted tele-assessments, clients who need immediate services may not be identified in a timely manner (Individuals with Disabilities Education Improvement Act [IDEA], 2004). On the other hand, practitioners who follow guidance to conduct adapted tele-assessments with no restrictions may risk making inaccurate client diagnoses due to faulty assessment data (National Association of School Psychologists, 2017, 2020). These opposing needs apply undue stress to the mental health and special education systems. In the end, individual practitioners will have to follow their own understanding of professional ethics and best practice, as well as any legal mandates of their governing bodies, while the field awaits future and more consistent recommendations.

Given that entities provided different recommendations across different formats, the main findings from this study are mostly historical in nature. These findings indicate the need for psychologists to speak in a singular voice using an easily identifiable/discoverable platform when crises occur. It is clear that our field was not prepared for this type of crisis; we had no plan in place for what to do. More importantly, we did not have a plan in place to disseminate urgent/emergency information. The fields of psychology and school psychology need to expand their emergency plans to include: (1) unified standards for making decisions regarding adapted tele-assessments and (2) a unified dissemination plan for any crisis-generated recommendations going forward. Hopefully, this study will provide information to future leaders for use when developing such a plan.