Survey Findings About the Experiences, Challenges, and Practical Advice/Solutions Regarding Teleneuropsychological Assessment in Adults.

OBJECTIVE: In the wake of the coronavirus pandemic, teleneuropsychology utilization has increased. There is a need to characterize the first-hand experiences of individuals using teleneuropsychology, identify the common teleneuropsychology challenges, and devise practical strategies for mitigating/resolving these challenges.
METHOD: Survey data were collected from U.S. based neuropsychologists and other individuals (e.g., graduate students and research assistants) who were engaged in remote cognitive assessment with adults (n = 87). Frequency analyses were conducted to examine: how respondents used teleneuropsychology (e.g., duration of use, types of measures and devices/technology platforms used); challenges that were encountered with different technology platforms and teleneuropsychology use; and advice for navigating these challenges.
RESULTS: Most respondents began using teleneuropsychology relatively recently in the context of the coronavirus pandemic, with home-to-home or clinic-to-home settings being the most frequently reported teleneuropsychology settings. Zoom®, Doxy.Me®, and MyChart® were the most frequently used platforms, largely due to workplace mandates and/or Health Insurance Portability and Accountability Act-compliant features. Common challenges with teleneuropsychology included internet connection issues and environmental distractions in examinees' homes, and examinees having limited-to-no access to technologies requisite for teleneuropsychology. Providing clear instructions to the examinee prior to the teleneuropsychology visit was the most common advice for ensuring a successful teleneuropsychology evaluation. Similar response patterns were generally noted for those who used TeleNP for clinical or research purposes.
CONCLUSIONS: These survey results reflect common experiences and challenges with teleneuropsychology and identify priority targets for increasing its feasibility, reliability, and validity. Findings provide context for the development of formal teleneuropsychology competencies.

Introduction

The SARS-CoV-2 (COVID-19) pandemic has generated a tremendous interest in the use of remote means for cognitive assessment. Teleneuropsychology (TeleNP) is broadly defined as the “application of audiovisual technologies to enable remote clinical encounters with patients,” (Bilder et al., 2020, p. 648). Despite the extant research on the reliability and validity of TeleNP, especially in adult populations (Chapman et al., 2019; Cullum et al., 2014; Eilam-Stock et al., 2021; Fox-Fuller et al., 2021; Marra, Hoelzle, et al., 2020b; Parks et al., 2021; Wadsworth et al., 2018; Chapman, Gardner, et al., 2020a), the field is still split on the use of TeleNP. Prior to the COVID-19 pandemic, the Medicare/Medicaid reimbursement policy discouraged TeleNP services delivered to patients in their homes; this policy was relaxed during the pandemic (U.S. Congress, 2020). Early surveys of neuropsychologists conducted immediately after the start of the COVID-19 pandemic-related lockdowns found there was initially relatively low initial uptake of TeleNP for the purpose of administering cognitive assessments, with many respondents solely engaging in remote clinical interviews and feedback (Chapman, Ponsford, et al., 2020b; Hammers et al., 2020; Marra, Hoelzle, et al., 2020b). In the USA by July 2020, however, one study found that nearly two-third of the surveyed U.S. licensed neuropsychologists began using TeleNP for the purposes of remote cognitive assessment (Rochette et al., 2021). Initially slow uptake of TeleNP for the purposes of remote cognitive assessment was due to several concerns, including: deviation from in-person delivery of neuropsychological services; limited information on preferred measures to use for a cognitive assessment, including concerns about reliability and validity of measures administered remotely; restricted funding or institutional support for TeleNP practice; and inadequate knowledge or familiarly with telehealth technologies on the part of the examiner and/or examinee (Chapman, Ponsford, et al., 2020b; Hammers et al., 2020; Marra, Hoelzle, et al., 2020b).

Despite this hesitation, given the nature and temporality of the pandemic, as well as the ethical need to provide continued clinical services and/or need to continue grant-funded research, several clinicians and investigators began to use TeleNP, especially in home-to-home or clinic-to-home settings. The delivery of neuropsychological testing virtually, especially to examinees in their homes, was found to be generally acceptable to many examinees while allowing examiners to achieve the goal of the encounter the vast majority of the time (Parsons et al., 2021). However, a review paper on TeleNP assessment in adults (Marra, Hamlet, et al., 2020a) found that most TeleNP research prior to the pandemic had been conducted in clinic-to-clinic settings, where some of the confounds of TeleNP (e.g., environmental distractors) were more easily controlled (e.g., Cullum et al., 2014; Wadsworth et al., 2018). Comparatively less research has been conducted in home-to-home or clinic-to-home settings (Abdolahi et al., 2016; Park et al., 2017; Stillerova et al., 2016), with a handful of new studies having been published recently during the COVID-19 pandemic (Fox-Fuller et al., 2021; Parks et al., 2021; Seghezzo et al., 2021). Such a discrepancy in the actual practice of TeleNP in settings different from how previous TeleNP research has been conducted may have heightened the concerns about the reliability and validity of TeleNP as a means for remote cognitive assessment (Rochette et al., 2021; Scott et al., 2021).

In the immediate aftermath of the COVID-19 lockdowns, several neuropsychological professional societies collaboratively developed guidance to help navigate the rapid shift from in-person assessment to TeleNP. For example, the Inter Organizational Practice Committee (IOPC) recommendations (Bilder et al., 2020; Postal et al., 2021), as well as those advanced by other groups—such as the Neuropsychology Service within the Department of Neurology at the Emory University Brain Health Center (Hewitt et al., 2020; Hewitt & Loring, 2020) and the Center for Cognitive Neuroscience and Aging at the University of Miami in Miami, Florida (Kitaigorodsky et al., 2021)—have responded to this need by focusing largely on the clinical use of TeleNP. However, the perspectives of the broader community of clinicians and researchers conducting cognitive assessments using TeleNP has not been well characterized to date, with only one recently published survey of licensed U.S. neuropsychologists by Rochette et al. (2021), to our knowledge, more fully exploring the first-hand experiences of TeleNP users. By examining the experiences of individuals using TeleNP for remote cognitive assessment—including the perspectives of nonlicensed individuals conducting remote cognitive assessments (e.g., graduate students, postdoctoral fellows, psychometricians, research assistants, and allied health professionals)—more focused guidelines can be developed to address the specific challenges that are being encountered with remote cognitive assessment.

We developed the Practical Advice Regarding Tele-Neuropsychology Evaluations in Research (and Clinical) Settings (PARTNERS) survey to characterize the first-hand experiences and take into consideration the challenges faced by individuals conducting TeleNP evaluations with adults. Additionally, we gathered information on the ways in which these challenges were navigated by the respondents. The study’s findings will thus provide valuable information on some practical solutions that can be used as a starting point for those wishing to utilize TeleNP. In summary, the three overarching objectives of this survey study were to:

Understand the current state of TeleNP practice, including the frequency of use, types of measures administered, the setting/s in which it is conducted, the devices used, and the platforms commonly relied upon;

Identify the challenges associated with the use of specific platforms for TeleNP, and more generally with the practice of TeleNP;

Elicit practical advice about mitigating or navigating the TeleNP challenges experienced by the respondents.

Additionally, a final exploratory aim of the study was to characterize the nature of TeleNP use and challenges faced in clinical versus research contexts.

Methods

Surveys

Two surveys were conducted, which were identical in format but targeted different samples. IRB approval was obtained through Columbia University Irving Medical Center.

Survey 1

Between December 8, 2020 and March 1, 2021, neuropsychologists in the USA were invited over list-servs and social media (e.g., Twitter and LinkedIn) to participate in the PARTNERS study. Colleagues of the authors were also emailed and were requested to circulate the survey among their neuropsychology networks (e.g., clinics, labs, and collaborators). The sampling frame targeted multiple neuropsychology member organizations, including: national neuropsychology groups (the American Academy of Clinical Neuropsychology, the National Academy of Neuropsychology, and Division 40 of the American Psychological Association), cultural neuropsychology groups (the Hispanic Neuropsychological Society, Asian Neuropsychological Association, and Society for Black Neuropsychology), state neuropsychology organizations (e.g., the Massachusetts Neuropsychological Society), and other organizations with memberships potentially involved in remote cognitive assessment of adults (e.g., the Gerontological Society of America). The population of interest for the first survey was licensed neuropsychologists in the USA who have conducted one or more remote cognitive assessments with adults. The survey consisted of multiple-choice and free-response questions (see supplement for example survey items; please contact the corresponding author for the entire survey). Ninety-seven individuals initiated the survey, 28 respondents did not complete the entire survey, whereas 13 were identified in the screening process as ineligible (i.e., self-reported they were not a U.S. neuropsychologist or did not engage in TeleNP). The total sample from survey 1 consisted of 56 respondents.

Survey 2

Next, between February 16, 2021 and March 1, 2021, nonlicensed individuals who reported engaging in TeleNP with adults in the USA as part of their work or training (graduate students, postdoctoral fellows, psychometricians, research assistants, and allied health professionals) were invited to participate using the same methods as survey 1. Out of the 50 respondents who initiated this survey, 31 met the eligibility criteria (working/training in the USA and not as a licensed neuropsychologist) and completed all the survey elements. The 19 ineligible respondents on survey 2 consisted of six individuals who self-reported as licensed U.S. neuropsychologists (i.e., they were eligible and redirected to survey 1), two individuals who did not report using tele-health technology for remote cognitive assessment, and 11 individuals who did not complete the entire survey.

Data Analysis

Statistical analyses were performed in SPSS version 26 (IBM SPSS Statistics, 2019). We used frequency analyses to investigate the three overarching objectives:

Specific TeleNP Characteristics: Specifically we sought to understand how long respondents had been engaging in TeleNP evaluations, how many TeleNP evaluations they had completed, and what types of assessments were being administered over TeleNP; the setting of TeleNP utilization (e.g., clinic-to-clinic) and the typical number of devices used by examiners and examinees; and which technologies were being used for TeleNP evaluations, and the reasons these technologies were utilized.

Challenges with TeleNP: The specific issues that respondents encountered with each technology used (e.g., with Zoom®) were documented, as well as the general problems encountered with TeleNP.

Practical Advice and Suggestions: Respondents’ specific solutions to the challenges associated with remote assessments and their general advice about TeleNP were obtained via free-response, text format questions. The first author (JTFF) thematically coded the responses into shared clusters which were cross-verified by the second author (SR). Any discrepancies between the coders were resolved by a discussion among all authors.

TeleNP for clinical versus research purposes: To examine TeleNP use and challenges associated with its use in research and clinical contexts, we used a survey question to split the groups. Specifically, participants were asked “At your peak level of your use of tele-neuropsychology, what was/is the percentage of your clinical work that was conducted using tele-health technologies?” The same question was repeated for obtaining the percentage of their research work that was conducted using tele-health technologies. For each question, participants were asked to indicate the percentage of use (ranging from 0% to 100%) by placing a slider in the appropriate place. Only those respondents who endorsed using TeleNP 100% of the time for clinical work at their peak level of TeleNP use were put into a “clinical group.” Similarly, those reporting using TeleNP 100% of the time for research work at their peak level of TeleNP use were put into a “research group.” The group responses were compared qualitatively due to the limited sample size in each group

Results

Participant Characteristics

Overall there were 87 participants (n = 56 from survey 1, and n = 31 from survey 2). Participant demographics are outlined in Table 1. Respondents represented 24 U.S. states with the largest representation from CA (n = 15, 17.2%), followed by NY (n = 12, 13.8%), then MA (n = 11, 12.6%), TX (n = 8, 9.2%), FA (n = 8, 9.2%), and PA (n = 6, 6.9%).

Table 1

Sample demographics

Characteristic	n or mean (% or SD)
Respondent type  Licensed neuropsychologists in the USA Other individual engaged in cognitive assessment in the USA	56 (64.4%)31 (35.6%)
Role of “other individuals” (n = 31) Clinical psychology doctoral student School counseling doctoral student Counseling psychology doctoral student Postdoctoral fellow Research assistant/research staff Allied health professional	14 (45.2%)a1 (3.2%)a1 (3.2%)a5 (16.1%)a9 (29.0%)a1 (3.2%)a
Age (years)  Licensed U.S. neuropsychologist Other individual engaged in cognitive assessment	42.7 (9.6)b29.0 (4.5)a
Race  Caucasian/White Asian/Asian American Latino/a/x Black/African American	73 (83.9%)10 (11.5%)7 (8.0%)2 (2.3%)
Ethnicity  Hispanic Non-Hispanic Other	10 (11.5%)75 (8.6%)1 (1.2%)
Geographic setting(s) of practice/work  Urban Suburban Rural	54 (62.1%)33 (37.9%)9 (10.3%)
Years of work post licensing (neuropsychologists)  >10 years 5–10 years 1–4 years <1 year	22 (39.3%)b15 (26.8%)b15 (26.8%)b4 (7.1%)b
Years of work in the field of cognitive assessment (other individuals)  5–10 years 1–4 years <1 year	13 (41.9%)a15 (48.4%)a1 (3.2%)a
Language of practice/work (without use of interpreter) c  English Spanish Hindi Mandarin Tagalog Other (1 respondent only per language; e.g., Russian, French)	87 (100%)12 (13.8%)3 (3.4%)2 (2.3%)2 (2.3%)5 (5.7%)
Practice/work in any language with interpreter assistance  Yes No	38 (43.7%)49 (56.3%)
Primary work setting(s)  Academic medical center Private practice Academia/university setting/small liberal arts college VA medical center Private hospital Rehabilitation hospital Industry/commercial company Community clinic Multidisciplinary group practice Other	32 (36.8%)21 (24.1%)16 (18.4%)14 (16.1%)7 (8.0%)3 (3.4%)2 (2.3%)2 (2.3%)1 (1.1%)4 (4.6%)
Adult population(s) of practice/work  Memory disorders (e.g., Alzheimer’s disease) Psychiatric disorders Acquired brain injuries (e.g., stroke) Concussion/traumatic brain injury Movement disorders (e.g., Parkinson’s disease) Healthy adults Epilepsy Developmental disorders Forensic cases Special populations (e.g., immigrants, language-specific consults) Oncology cases Other	58 (66.7%)45 (51.7%)45 (51.7%)43 (49.4%)38 (43.7%)30 (34.5%)28 (32.2%)26 (29.9%)13 (14.9%)12 (13.8%)8 (9.2%)9 (10.3%)

Specific TeleNP characteristics

TeleNP uptake, frequency of TeleNP use, and types of measures administered remotely

Regarding temporality, most respondents (n = 71; 81.6%) had used TeleNP only since the COVID-19 pandemic began, whereas nine participants (10.3%) endorsed using TeleNP for more than 1 year, 6 (6.9%) for 2–4 years, and one respondent (1.1%) for 5–10 years. Eighty-two respondents (94.3%) reported using cognitive measures over TeleNP (e.g., tests of memory, executive functioning, and language). Seventy-five participants (86.2%) reported utilizing behavioral/symptom checklists and self-report questionnaires, and 53 participants (60.9%) reported using cognitive screening tools (e.g., the Montreal Cognitive Assessment [Nasreddine et al., 2005], the Telephone Interview for Cognitive Status [Brandt et al., 1988], and the Mini Mental State Examination [Folstein et al., 1975]) over TeleNP. About half the respondents (n = 42, 48.2%) stated they had completed more than 25 TeleNP evaluations (either directly or through supervision of a trainee/psychometrician) since they began doing remote cognitive assessments. 17 (19.6%) respondents reported completing between 11 and 24 TeleNP assessments, 22 (29.9%) had completed between 2 and10 TeleNP evaluations, and two (2.3%) had completed only one TeleNP evaluation.

Setting of TeleNP service delivery and the type/number of devices typically used

A majority of respondents (n = 71, 81.6%) reported engaging in home-to-home TeleNP evaluations at some point or presently, with 39 (44.8%) engaging in clinic-to-home assessments. 19 (21.8%) and 9 (10.3%) respondents, respectively, engage/engaged in TeleNP in clinic-to-clinic (same building, different rooms) and clinic-to-clinic (satellite center) settings. Fifty-three participants (60.9%) used two or more technologies (e.g., computers, tablets, telephone, etc.) during a typical TeleNP assessment. Nearly all participants (n = 86, 98.9%) endorsed using a computer during a TeleNP evaluation as an examiner, 36 (41.4%) used a telephone (audio call), 28 (32.2%) used a tablet, and 14 (16.1%) used smartphone applications. A majority of the respondents (n = 69, 79.3%) stated their examinees used a single device during a typical TeleNP assessment, with the predominant device used by examinees being computers (n = 73, 83.9%), followed by tablets (n = 45, 51.7%). Additionally, 37 (42.5%) of participants reported that examinees used telephone (audio calls) during TeleNP evaluations, and 32 (36.8%) reported their examinees utilize smartphone applications.

Type of technologies used for TeleNP evaluations and the reasons for their use

Table 2 details the specific technology platforms used for TeleNP by respondents and other information, including: the length of their use by the respondents; and the respondents’ reasons for using these technology platforms. Zoom® was the most commonly used technology (n = 54, 62.1%), followed by Doxy.Me® (n = 18, 20.7%) and MyChart® (n = 16, 18.4%). Across nearly all technologies used for TeleNP evaluations, the majority of respondents (>50%; see Table 2) reported duration of use of less than 1 year, reflecting uptake in the context of the COVID-19 pandemic. The selected technology was most often chosen because it was the mandated option or the only available option at work or due features relating to Health Insurance Portability and Accountability Act (HIPAA) compliance.

Table 2

Technologies used in remote cognitive assessment in adult populations in the USA

Technology	Number of users (% of total sample, n = 87)	Duration of use of the specific tele-health technology	Reason(s) for use	Challenges encountered, if any	Resolutions, if any	Mean recommendation level (1–10)
Zoom	54 (62.1%)	<1 year: 53.7%1–4 years: 38.9%5–10 years: 7.4%	Mandated/only option at work: 57.4%HIPAA Compliance: 68.5%Cost (most economical): 9.3%Ease of use for examiner: 31.5%Ease of use for examinee: 31.5%Did not know about other options: 3.7%Popularity: 31.5%Other: 3.7%	No challenges: 51.9%Connectivity problems (nonspecific to Zoom): 33.3%Patient/participant difficulty with platform: 13.0%Technological problems specific to Zoom (e.g., time-limit on free version and nonintegration with some software): 13.0%Issues with video/audio quality (specific to Zoom): 5.6%	Do a practice call with the patient (or have administrative staff do a practice call); improvise or call examinee to walk them through difficulties; reconnect/restart meeting; reschedule to in-person or use a backup technology; use only audio for the visit; purchase pro Zoom account	8.17 (SD = 1.44); range: 5–10
Doxy.Me	18 (20.7%)	<1 year: 88.9%1–4 years: 11.1%	Mandated/only option at work: 50%HIPAA compliance: 61.1%Cost (most economical): 33.3%Ease of use for examiner: 16.7%Ease of use for examinee: 22.2%Popularity: 5.6%Other (good compatibility with document camera/good screen sharing): 5.6%	No challenges: 33.3%Poor audio/visual quality: 33.3%Issues with technology (specific to Doxy.Me): 16.7%Patient/participant difficulty with platform: 5.6%	Use audio only; use telephone call for backup audio; restart visit; have participant switch web browsers	6.39 (SD = 2.66), range: 1–10
MyChart (or EPIC Equivalent)	16 (18.4%)	<1 year: 56.2%1–4 years: 31.3%5–10 years: 6.3%>10 years: 6.3%	Mandated/only option at work: 100%HIPAA compliance: 43.8%Ease of use for examiner: 6.3%	No challenges: 25%Connectivity problems (nonspecific to MyChart): 25%Patient/participant difficulty with platform: 37.5%Poor audio/visual quality: 12.5%Issues with technology (specific to MyChart): 18.8%	Use hospital administrative/IT help to resolve problems; use telephone call or a different platform as backup; reschedule the visit to in-person for testing; call examinee ahead of time to inform them of instructions about how to join visit	6.81 (SD = 2.26), range: 1–10
VA Video Connect	14 (16.1%)	<1 year: 71.4%1–4 years: 21.4%5–10 years: 7.2%	Mandated/only option at work: 92.8%HIPAA compliance: 50%Cost (most economical): 7.2%Ease of use for examiner: 21.4%Ease of use for examinee: 21.4%	No challenges: 14.3%Connectivity problems (nonspecific to VA Video Connect): 35.7%Patient/participant difficulty with platform: 35.7%Poor audio/visual quality: 14.3%Issues with technology (specific to VA Video Connect): 14.3%	Use telephone call; use alternate telehealth technology; reschedule for in-person testing	6.43 (SD = 2.07), range: 2–9
Facetime	8 (9.2%)	<1 year: 50%1–4 years: 12.5%5–10 years: 25%>10 years: 12.5%	HIPAA compliance: 12.5%Cost (most economical): 12.5%Ease of use for the examiner: 25%Ease of use for the examinee: 75%Popularity: 25%	No challenges: 50%Cannot block personal cell phone number: 12.5%Concerns about security of the session/limitations of the platform for visual test stimuli: 37.5%	Facetime used as a last resort if examinee not able to use other platform; examiner blocks examinee’s number on phone after session; only use a limited test battery	6.38 (SD = 2.39), range: 3–10
Microsoft Teams	5 (5.7%)	<1 year: 20%1–4 years: 80%	Mandated/only option at work: 100%HIPAA compliance: 40%Cost (most economical): 20%Ease of use for examiner: 20%Ease of use for examinee: 20%Popularity: 40%Integration into software at work: 20%	No challenges: 60%Poor audio/video quality: 20%Loss of visibility of examinee when sharing screen: 20%	Removal of timed-exposure tasks or administration of these tasks at a later date in-person; use telephone call as backup	7.20 (SD = 0.84), range: 6–8
Skype	2 (2.3%)	<1 year: 50%>10 years: 50%	Popularity: 50%Other reason (other technologies did not work): 50%	Not always functional: 50%General connectivity problems: 50%	Reschedule assessment; use telephone as backup	3.50 (SD = 2.12), range: 2–5Note: 2 raters only
Other technologies: (Doximity, Google Meet/Gsuite, WebEx, and other miscellaneous technologies)	22 (25.3%)	<1 year: 77.2%1–4 years: 18.3%5–10 years: 4.5%	Mandated/only option at work: 54.5%HIPAA compliance: 59.1%Cost (most economical): 36.4%Ease of use for examiner: 22.7%Ease of use for examinee: 27.3%Other (scheduling/sharing documents): 13.6%	No challenges: 50%Connectivity problems (nonspecific to the telehealth technology): 18.2%Patient/participant difficulty with platform: 13.6%Technology problems (specific to the telehealth platform): 13.6%Poor audio/video quality: 9.1%	Restart session; reschedule testing for in-person session; use telephone for backup audio; switch to a different telehealth technology; troubleshoot on internet search engine	n/a—various technologies included in “other”

Commonly used TeleNP platforms and challenges/resolutions associated with their use

Specific technology-related challenges

Table 2 depicts the challenges encountered by the respondents with the technologies and the manner in which the challenges were resolved. Seventy respondents (80.5%) reported that they had not changed the technology/technologies they were using for remote cognitive assessment or had not used any technologies for remote cognitive assessment prior to their currently used platform(s). Of the other respondents, seven participants reported they had stopped using Zoom®, five stopped using Skype®, three stopped using Facetime®, three stopped using MicrosoftTeams®, three stopped using Doxy.Me®, two stopped using VA Video Connect, and one stopped using VSee®. Across all platforms, the most frequently endorsed reason for changing platforms was a change in workplace policies, followed by concerns with HIPAA-compliance. Some platforms had uniquely endorsed issues, with one respondent describing poor audiovisual quality on Doxy.Me®, and another respondent noting that the VA Video Connect servers were overwhelmed with traffic at the start of the COVID-19 pandemic.

Common general challenges with TeleNP

Table 3 depicts the nature and frequency of the general challenges associated with using TeleNP. The most frequently endorsed problems included: internet connectivity problems in the examinee’s home (n = 72, 82.8%); environmental distractions in the examinee’s home/satellite testing center (e.g., pets, children, partners, unexpected loud noises) (78.2%); unknown sources of internet connectivity problems (58.6%); examinee’s limited access to technology (i.e., examinee has a smartphone but does not own a computer) (57.5%) or complete lack of access (35.6%); lack of audio clarity (55.2%); examinee’s lack of familiarity with the videoconferencing technology (52.9%); and lack of ability to easily conduct visuoconstructional tasks (52.9%).

Table 3

Teleneuropsychology challenges endorsed by survey respondents

Problem	# of respondents endorsing problem; n (% of total sample)
Internet connectivity problems in the examinee’s home	72 (82.8%)
Environmental distractions in the examinee’s home/satellite testing center (e.g., pets, children, partners, unexpected loud noises)	68 (78.2%)
Unknown source of internet connectivity problems	51 (58.6%)
Examinee has limited access to technology (i.e., they have a smartphone but do not own a computer)	50 (57.5%)
Lack of audio clarity	48 (55.2%)
Examinee’s lack of familiarity with the videoconferencing technology	46 (52.9%)
Lack of ability to easily conduct visuoconstructional tasks	46 (52.9%)
Issues adapting and finding norms for testing over videoconferencing	41 (47.1%)
Examinee does not own the required technology to complete remote cognitive assessment (i.e., they have no access at all)	31 (35.6%)
Internet connectivity problems in the examiner’s home	31 (35.6%)
Lack of video clarity	30 (34.5%)
Environmental distractions in the examiner’s home/place of work (e.g., pets, children, partners, unexpected loud noises)	29 (33.3%)
Concerns about test security	29 (33.3%)
Examinee’s discomfort using the videoconferencing technology	21 (24.1%)
Lack of the examiner’s familiarity/lack of ability to troubleshoot technological issues that arise during internet videoconferencing	17 (19.5%)
Internet connectivity problems in the examiner’s place of work	14 (16.1%)
Use of technology with examinees who are non-English speakers	14 (16.1%)
Internet connectivity problems at the examinee’s satellite testing location	5 (5.7%)
Examinee only has access to required technology to complete remote cognitive assessment in public locations (e.g., going to a library to use a computer, or going to a cafe to use WiFi)	3 (3.4%)
Other (examinee is more informal, such as doing their interview in bed)	1 (1.1%)

Practical advice about mitigating/resolving TeleNP challenges

Details of the general TeleNP advice themes given by respondents are provided in Table 4. The more common thematic suggestions that emerged included: providing the examinee with clear instruction about the virtual visit and expectations regarding the session ahead of time; reserving TeleNP for clinical interviewing or brief cognitive screening while having the examinee come to the clinic for in-person assessment; and having a backup or assistive plan ready should technological difficulties arise (e.g., use a telephone for instructions).

Table 4

Teleneuropsychology advice themes given by survey respondents

Advice	# of respondents endorsing this theme in the free-response question
Provide as much instruction about the virtual visit ahead of time, and clarify the expectations of the visit	8
Use TeleNP solely for clinical interview, brief cognitive screening, and behavioral measures; have examinee complete testing in-person	3
Have backup plan ready (e.g., phone call)	3
Do a practice call with the examinee before the day of testing	2
Use sparingly in clinical settings, and only use tests for which there is normative data for virtual visits, as future research is needed	2
Create standard operating procedures for TeleNP in the clinic/setting, and document issues that arise and how they were resolved	2
Develop a sense of flexibility and patience for if/when challenges are encountered	2
Use a smaller, targeted battery than what you would use in-person	2
Use certain devices to augment the experience (e.g., an external headset and microphone)	2
Technology companies should be contacted about how to streamline the process of logging onto visits for examinees (especially for individuals with low technological literacy)	2
If working with an older adult or more impaired examinee, be sure to have a carepartner available to troubleshoot technology problems, if at all possible	2
Break into multiple sessions	1
Screen-share visual stimuli over the platform versus holding the stimuli up to the camera	1
Do your own pseudoevaluations with a friend/colleague before you attempt an evaluation with a patient or participant	1
Minimize environmental distractions on both ends of the call (e.g., mute all phones, ensure examinee is in a private space)	1
Technologies should be created to adjust more seamlessly to different broadband availabilities	1
Consider if TeleNP is appropriate for the referral, especially if the examinee has any sensory impairments (e.g., vision or hearing loss)	1
When possible, use a wired internet connection	1

Characterizing TeleNP for clinical and research purposes

At their peak level of TeleNP use, out of the 87 respondents, 42 respondents (including 30 licensed U.S. neuropsychologists) and 30 respondents (including 14 licensed U.S. neuropsychologists) reported using TeleNP for 100% of their clinical work and research work, respectively. A majority of respondents in both groups used formal cognitive measures over TeleNP (clinical = 40/42 [95.2%]; research = 30/30 [100%]); followed by behavioral symptom checklists/questionnaires (clinical = 37/42 [88.1%]; research = 29/30 [96.7%]) and cognitive screening tools (clinical = 27/42 [64.3%]; research = 22/30 [73.3%]). In both groups, a majority of examiners were or still are conducting evaluations in home-to-home (clinical = 36/42 [85.7%]; research: 29/30 [96.7%]) or clinic-to-home (clinical = 19/42 [45.2%]; research = 7/30 [23.3%]) settings. However, relative to respondents in the research group (1/30 [3.3%]), more respondents in the clinical group (12/42 [28.5%]) were or still are conducting TeleNP evaluations in clinic-to-clinic settings, particularly wherein the examiner and examinee were in different rooms at the same clinic.

In both groups, most respondents endorsed using two or less devices as an examiner (clinical = 34/42 [80.1%]; research = 27/30 [90.0%]), and computers were overwhelmingly used by examiners in both groups (clinical = 41/42 [97.6%]; research = 29/30 [96.7%]), followed by telephone audio calls (clinical = 16/42 [38.1%]; research = 11/30 [36.7%]), tablets (clinical = 11/42 [26.2%], research = 7/30 [23.3%]), and smartphone applications (clinical = 10/42 [23.8%]; research = 6/30 [20.0%]). A majority of respondents in both groups reported that their patients or participants typically used one device during a TeleNP encounter (clinical = 36/42 [85.7%]; research = 23/30 [76.7%]). In both groups, respondents reported that the examinees most commonly used computers during the TeleNP evaluation (clinical = 37/42 [85.7%]; research = 22/30 [73.3%]), followed by tablets (clinical = 23/42 [54.8%]; research = 14/30 [46.7%]), telephone audio calls (clinical = 20/42 [47.6%]; research = 14/30 [46.7%]), and smartphone applications (clinical = 13/42 [31.0%]; research = 12/30 [40.0%]). The most typically used platform for TeleNP evaluations was Zoom® in both groups (clinical = 26/42 [61.9%]; research = 21/30 [70.0%]). The TeleNP challenges encountered by the clinical versus research groups were generally the same across the groups and overall consistent with what was found for the full sample (see Table 3).

Discussion

Though general practice recommendations have been developed to aid in conducting TeleNP for cognitive assessments (e.g., Bilder et al., 2020), to our knowledge this is among the first studies to empirically examine the first-hand experiences and challenges experienced by TeleNP users. We also sought information on the common strategies that TeleNP users employed to navigate the challenges associated with TeleNP use, and based on the data we have proposed some practical solutions to address specific challenges. Although a recent study by Rochette et al. (2021) used a similar survey design, our study is unique because we included information not reported elsewhere, such as the inclusion of information related to the use of TeleNP for clinical versus research purposes and the reasons TeleNP users were using or switched between TeleNP technologies. We also elicited the perspectives of non-neuropsychologists in the USA (e.g., graduate students, research assistants, psychometricians) who use/used TeleNP. The findings from our study and other surveys can thus be used as a starting point for those considering to engage in TeleNP and can be incorporated into discussions about formalized TeleNP competencies. Below we first highlight the primary findings from our study, and then propose recommendations synthesized from the survey’s respondents while integrating the TeleNP literature.

Experiences with TeleNP

The primary findings of the study were that most respondents using TeleNP were relatively new to remote cognitive assessment, with a majority endorsing TeleNP use for less than a year, and only half of the respondents reporting completion of 25 or more remote cognitive evaluations (see Table 1). Home-to-home and clinic-to-home were the most common settings in which TeleNP was conducted, contrary to the majority of extant TeleNP research which occurred in clinic-to-clinic settings (e.g., Chapman et al., 2019; Chapman, Gardner, et al., 2020a; Cullum et al., 2014; Marra, Hoelzle, et al., 2020b; Wadsworth et al., 2018). This underscores the need to develop setting-specific TeleNP guidelines for examiners and focus more efforts on investigating the reliability and validity of cognitive tests in settings in which the examinee is in their home. During a typical TeleNP session, a majority of survey respondents endorsed using two or more devices simultaneously as examiners, with the use of computer being the most common, followed by the use of telephone. Relative to in-person neuropsychological assessment, using multiple devices during a TeleNP evaluation, while necessary in some cases, may increase working memory demands on examiners (Scott et al., 2021), and also in examinees, especially those with cognitive impairments and limited technological knowledge. Therefore, future studies should also consider the number and type of devices used during TeleNP assessments and conduct psychometric studies in this context.

Zoom®, Doxy.Me®, and MyChart® were the platforms used by the majority of respondents to conduct remote cognitive assessments, similar to what was found by Rochette et al. (2021) in their survey study. The use of certain technologies for TeleNP was largely driven by organizational-related mandates and/or due to the HIPAA-compliant features of these technologies (see Table 2). The majority of the respondents had not discontinued using the first platform(s) they had used, likely reflecting the relative recency of TeleNP uptake in this sample and/or respondents’ limited experiences with or knowledge of alternative platforms. For the minority of respondents who had changed technology platforms, the two most common driving factors were changes in workplace policies or concerns about HIPAA-compliance.

Challenges Encountered with TeleNP

The most frequently reported general challenge with TeleNP use was poor internet connectivity in the examinee’s home or an unknown source of internet connectivity problems. This challenge is a barrier to all forms of remote cognitive assessment requiring live internet connections. The IOPC recommendations and other commentaries have suggested screening for both the internet connectivity speed in the examiner’s and examinee’s locations using simple and accessible websites, such as https://www.speedtest.net/ (Bilder et al., 2020; Hewitt et al., 2020; Hewitt & Loring, 2020). By testing internet connectivity speed in advance (preferably), or at least at the start of the session, potentially problematic connections can be detected and alternative plans for cognitive testing can be discussed and implemented.

Other common TeleNP concerns endorsed by respondents included environmental distractions (e.g., pets, family members interrupting the visit, and unexpected noises) in the examinee’s home and/or the examiner’s home/place of work (see Table 3). These environmental distractions likely contribute to concerns about test reliability and validity (Bilder et al., 2020; Scott et al., 2021). During the COVID-19 pandemic when the use of TeleNP has substantially grown, clinicians and investigators have also been faced with balancing the trade-off of test reliability and validity concerns associated with home-to-home or clinic-to-home settings with those related to in-person, pandemic-related testing protocols. Some examples include the utilization of masks, gloves, and plexiglass barriers, all of which are deviations from the procedures in which in-person cognitive assessments were designed and standardized (Postal et al., 2021).

Lastly, one of the most frequently reported challenges with TeleNP assessment was concern that some examinees have limited technological access (i.e., they may have a smartphone but not a computer) or have no access to the technology requisite for remote cognitive assessment. Previous research suggests that older individuals and individuals with lower levels of technological knowledge are less likely to engage in accessing online health portals (Wildenbos et al., 2017) and that access to broadband internet within one’s neighborhood or community was a major contributor to usage of online health portals (Perzynski et al., 2017). However there is also evidence that older adults prefer to use online technology for some aspects of their everyday functioning, such as money management (Sunderaraman et al., 2020). Emerging research on the acceptability of TeleNP in various contexts during the COVID-19 pandemic suggests that many examinees find TeleNP services satisfactory (Appleman et al., 2021; Gardner et al., 2021; Parsons et al., 2021). Prior telepsychiatry research also suggests that patients, especially those with limited access to in-person healthcare services in their communities, are highly satisfied with telehealth services, and in some cases prefer telehealth appointments over in-person visits (Chong & Moreno, 2012; Hilty et al., 2007; Shore, 2013).

TeleNP Use and Challenges for Clinical versus Research Purposes

Although the goals of clinical and research evaluations differ, generally there were not many notable differences in TeleNP use and challenges encountered between clinical- and research-focused respondents. One specific difference that was noted was that more clinically focused individuals reported conducting remote cognitive evaluations in clinic-to-clinic settings relative to research-focused respondents; both groups, however, reported a relatively equal number of examiners whom were/are also conducting TeleNP evaluations in home-to-home or clinic-to-home settings as well. Across groups, TeleNP uptake occurred relatively recently in the context of the COVID-19 pandemic. Regardless of the purpose, during a TeleNP encounter typically about 1–2 devices were being utilized by the examiners, while one device (typically a computer) was utilized by the examinee. Zoom® was the primary platform used for TeleNP. TeleNP challenges did not differ between the research and clinical groups and were consistent with those reported in the full sample.

Practical Solutions and Advice About TeleNP Use

Based on respondents’ advice about how they navigated challenges encountered with TeleNP (Table 4), subsequently we outline a set of practical recommendations and solutions to aid those conducting remote cognitive assessments. We also suggest ways in which clinical work and research can potentially be enhanced by TeleNP, as well as potential avenues for innovative TeleNP research.

Ethical considerations and informed consent

Particularly in clinical settings, it is important to weigh ethical considerations when determining if TeleNP in lieu of in-person assessment is an appropriate avenue of care for the examinee. The expectations for TeleNP evaluations, including their potential limitations, can be included in an informed consent form that can be shared with examinees prior to or at the start of the TeleNP encounter. A recent paper highlighted the need to carefully consider the APA standards and ethical principles governing the practice of remote assessments (Scott et al., 2021), including factors related to examinee’s:

Pre-existing medical conditions and comorbidities

Urgency of clinical care

Familiarity with the technology used for remote assessment

English-language proficiency

Geographic distance from the clinic or a satellite clinic and access to transportation; and

Availability of technical and administrative support

Depending on the situation and its demands, one possibility is for the examiner to visit the examinee and conduct the testing in the examinee’s home. Such an approach is more frequently used in research than in clinical practice, especially for those who may have barriers to engaging in TeleNP. When preparing for a TeleNP evaluation, verifying in advance the examinee’s precise location, including their residential address if their current location varies from their home address, is essential. This process will ensure that the practitioner is not violating laws governing clinical practice across states (or countries). In the event the examinee experiences a medical or behavioral emergency or discloses information that requires mandated reporting, having the confirmed address of the examinee and a tiered contingency plan will allow the examiner to facilitate an appropriate intervention with local emergency personnel. The consent form should also detail this process.

Develop and expand examiner competencies

As TeleNP continues to grow in clinical and research settings, ensuring the competencies of examiners is of the utmost importance. Competencies do not simply include an understanding of which tests have the strongest research base for TeleNP, but also how to successfully navigate across different technologies which can be used to conduct a TeleNP evaluation and troubleshoot problems when they arise during an assessment. As has been suggested (Bilder et al., 2020), examiners interested in conducting TeleNP evaluations can advance their competencies by:

Attending webinars and conference presentations about TeleNP (as of this report’s publication, a list can be accessed through the IOPC website at: https://iopc.online/teleneuropsychology-training).

Consulting with experienced colleagues about best practices.

Seeking out and developing platform-specific protocols and scripts.

Practicing the administration of tests over TeleNP with colleagues prior to working with patients or participants.

Prepare for the session in advance

One of the most frequently suggested recommendations by respondents was that examiners should prepare in advance for TeleNP visits. This preparation may resemble standard preparation for in-person cognitive assessment, though there are additional factors to consider, including:

Preparing tests which have alternate forms in case the first version becomes spoiled (e.g., an environmental distraction occurs during a word list-learning paradigm).

If possible, using a brief and targeted TeleNP test battery relative to what is common during in-person assessment, which may help prevent fatigue (including “Zoom fatigue,” or the type of fatigue specific to prolonged videoconferencing interactions).

Reaching out to examinees ahead of time to set expectations about the virtual visit, such as asking the examinee to find a quiet space free from distraction in the home and, ideally, to be seated at a table or a desk (vs. being in their bed or on the couch).

Providing examinees in advance with step-by-step instructions for how to operate the chosen technology used for the assessment (e.g., screenshots).

When possible, examiners should conduct a brief technology setup and internet connectivity check session prior to the day of the TeleNP evaluation; this will reduce the risk of the examiner and examinee becoming fatigued by any troubleshooting challenges that may arise on the day of testing.

Create and be prepared to execute a backup plan if the planned virtual visit fails, including:

Switching to a platform that does not require a web application download (e.g., GoToMeeting®, Doxy.Me®, WebEx®) or switching to a platform with which the examinee may be more familiar (while maintaining HIPAA compliance).

Assessing prior to the visit which platform(s) the examinee is familiar with and utilizing one of their preferred platforms, while ensuring that it is HIPAA-compliant; obtaining advance permission from the examiner’s institution may be necessary.

In case of poor audio quality, supplementing the video feed with a telephone audio call.

Conducting part of the visit (e.g., interviews/questionnaires) on a smartphone.

Rescheduling cognitive testing for a later date or in-person (if feasible).

If available, involve a family member or friend with more technological experience to help set up the visit.

For examinees with auditory issues, ask them to wear their hearing aids (if available) or the use of high quality, noise-cancelling headphones can be suggested.

Select the tests and prepare the test stimuli in advance

If TeleNP use is anticipated, having a ready list of digitally friendly cognitive assessments will be the first step in ensuring a successful testing session (see https://iopc.squarespace.com/teleneuropsychology-research for suggestions). Additional considerations include:

For in-person tests that need to be administered remotely, one could ship the test stimuli (e.g., Seghezzo et al., 2021) to the participant with appropriate written consent to not disclose or violate copyright issues. Examiners should also include instructions asking the examinee not to open the materials until prompted by the examiner during the testing session.

For visuospatial tasks, utilizing programs that present images (Adobe Acrobat, Microsoft PowerPoint, etc.) and screen sharing the visual stimuli is preferable for preserving stimuli quality as opposed to an examiner holding up the stimuli in front of the camera. Screen sharing frees up examiners hands to engage in other tasks (e.g., scoring the test or recording behavioral observations) and the presentation of timed stimuli can also be automated in some programs. A document camera can also be considered as a way to share stimuli that is not available digitally, though using a document camera may be more cumbersome to set up than screen sharing for some examiners. Moreover, both approaches—screen sharing and using a document camera—may not necessarily mitigate concerns with online sharing of stimuli due to copyright issues.

It is important to establish the examinee’s ability to view what is being shared; make sure the examinee is viewing the stimuli in the fullest window possible on their device and that nothing (e.g., a video thumbnail of the examiner and examinee) is blocking any part of the stimuli or is otherwise distracting the examinee. Some platforms, such as Zoom®, allow the examiner to remotely control the examinee’s window and ensure an optimal viewing experience. Alternatively, examiners should be prepared to explain to examinees how to minimize and move the video thumbnails out of the way. Device specific variations should be explored and expected (e.g., Zoom® settings may vary on Windows® versus Macintosh® devices).

The use of a nonscored sample set of slides at the beginning of the session (e.g., one sentence to be read out loud; naming colors, shapes) can help ensure that the examinee can see exactly what is intended to be shared, as well as introduce the examinee to how the examiner will be transitioning between tasks. Additionally, such procedures can screen for individuals with poor vision or color blindness which may preclude the administration of certain visual tasks.

As with in-person testing, it is essential during a TeleNP evaluation to carefully document any issues that arise during testing which may confound results. For example, a delayed internet connection during a processing speed measure or a word list-learning paradigm can directly contribute to poor test performance.

Collect tech-specific data

During and prior to TeleNP evaluations, it may be advisable to collect data on examinees’ technological knowledge. Examples include the examinee’s prior experience with specific platforms and devices and their confidence with technology skills. As has been suggested (Hewitt & Loring, 2020), with TeleNP it may become standard to collect such data about examinees in order to best plan for remote cognitive assessment. This data collection can typically be executed in a relatively simple fashion (e.g., brief prescreening questionnaires with multiple-choice and Likert-scale items). Such data can also be used to account for unexpected test findings and serve as a way to check for possible confounds in the results.

Anticipate and develop a plan to mitigate environmental distractions

Environmental distractions were one of the most frequently endorsed challenges identified by the respondents (both in the examinee’s environment, and less frequently in the examiner’s environment). A suggested solution by some respondents involved providing clear instructions about the expectations of a TeleNP evaluation to patients or participants in advance (i.e., over a brief previsit phone call or video chat). Expectations for virtual visits can include discussion about how the examinee should treat the assessment as seriously as they would if they were being tested in the clinic (e.g., if possible, the examinee should be seated at a table or desk in a quiet, private room in their home or office where they will not be disturbed; the examinee should silence phones and other distractors; when possible, arrangements should be made for the care of pets or dependents during the evaluation). Examiners should also appraise and document the level of distractibility for each test administered during the TeleNP session, which may provide valuable information about the validity of each test and aid in the interpretation of data.

Some other suggestions are:

Provide examinees and examiners with noise cancellation devices (e.g., if already mailing or loaning materials, consider also sending noise-cancelling headsets or white noise machines to be put outside the examinee’s room during the session). Though not explicitly stated as a concern by any respondent in this survey, examinees’ hearing issues may be partially mitigated through both examiners and examinees using noise-cancelling headphones, which enhance the audio quality and reduce background noise on both ends.

As mentioned previously, select and prepare to use assessments which have alternate forms, when possible, for use in the event that administration of the first form is spoiled due to environmental distractions.

Give the examinee frequent breaks, which will aid in reducing fatigue for both the examiner and examinee.

Be prepared to cancel and reschedule the session, and/or budget extra time for testing than what is typical for an in-person evaluation.

As noted by Dr Kira Armstrong during a workshop presentation at the 2021 AACN Virtual Conference (McPherson et al., 2021), in some situations examiners may wish to ask examinees to set up multiple cameras during the TeleNP visit. Whereas the main computer or device’s webcam can be used to communicate directly with the patient (i.e., seeing the examinee’s face), having the examinee log into the meeting with one or two additional devices can help capture additional valuable information pertinent to behavioral observations and test-taking behaviors (e.g., restlessness, being fidgety with hands, having a pet seated in the examinee’s lap, etc.). To set up a device, the examinee can log into the same meeting with their smartphone and set it up on a chair or any available piece of furniture across the room and make sure to set this additional device’s microphone on mute to prevent echo. An additional camera can also be set up by an examinee at an angle that can allow examiners to see the process of how the examinee draws or writes. However, setting up multiple devices may require the examinee to own or borrow such devices, or have devices loaned to them. Help from another family member or friend may be required, especially for examinees with severe cognitive impairments or those who are technologically naïve.

Increase equitable access to TeleNP

Technology unfamiliarity by examinees, including technology-specific concerns (Kruse et al., 2012), is one driver of the digital health divide. This barrier persisted in the COVID-19 pandemic with examinees’ lack of confidence with their own technology skills being a common barrier to willingness to complete a TeleNP evaluation in lieu of on-site evaluation in a university medical center setting (Hewitt & Loring, 2020). Both our survey and another recent survey (Rochette et al., 2021) identified examinees’ lack of access to requisite technology as a common barrier to TeleNP. For examinees with limited technological knowledge and low social support to help with the setup required for many TeleNP evaluations, some alternatives may be to adopt a telephone-based cognitive assessment or use a hybrid model in which the clinical interview and questionnaires can be conducted via remote methods (e.g., using whatever device is accessible to the examinee), with cognitive testing conducted at a later point in-person.

For those who have broadband internet access but have limited access to specific devices (i.e., they do not own a computer, but do own a smartphone), one way to increase equity of access to TeleNP may involve loaning the necessary equipment (e.g., computer or tablet) to the examinee. This strategy is now being deployed in several active research studies (e.g., the Long Life Family Study, the Integrative Longevity Omics Study, and the Longevity Consortium Centenarian Project), and, as mentioned by others (Scott et al., 2021), is emerging in some clinical settings, such as the Veteran Health Administration’s smart-device loaning program (). Such an approach may increase standardization procedures, particularly because the devices can be preconfigured to standard settings and include setup for remote operation by examiners. However, this is costly to implement in clinics or for investigators with limited budgets, and may also confuse some examinees who may be used to working with their own devices. Moreover, this approach may be unfeasible or require significant technological support from the examiner or the examiner’s support staff.

Additionally, increased and creative advocacy efforts are required by neuropsychologists, social workers, lobbyists, and others on behalf of individuals who could benefit from increased broadband internet access and, more broadly, the expansion of telehealth services in general. In particular, if performances on TeleNP (especially in contexts in which the examinee is in their home) and in-person neuropsychological evaluations are found to be largely comparable, and/or if the unique relevance of TeleNP—especially under certain circumstances (e.g., those having difficulty traveling for in-person evaluations)—is underscored, neuropsychologists will need to lobby the U.S. Congress to continue to permit reimbursement for TeleNP evaluations by Medicare/Medicaid.

Future directions—old and new avenues of TeleNP research

(i) TeleNP Settings: The home-to-home and clinic-to-home settings were the most frequently endorsed setting in which TeleNP was being conducted in our survey, for both clinical and research purposes. This likely reflects common practices during the COVID-19-related lockdowns (e.g., Parsons et al., 2021). The practice of remote cognitive assessment with an examinee in their home has several nonpandemic-related benefits (e.g., no travel needed for the examinee and the examiner can get a valuable glimpse into the examinee’s home life), and many neuropsychologists anticipate using TeleNP in some form in the future (Rochette et al., 2021). Therefore, as has been noted by others (Bilder et al., 2020; Marra, Hamlet, et al., 2020b), more research is needed to specifically explore the reliability and validity of tests in home-to-home and clinic-to-home settings compared with traditional in-person face-to-face assessment and clinic-to-clinic TeleNP, which have stronger research bases. In the context of the COVID-19 pandemic, a handful of research studies have begun to address these concerns (Fox-Fuller et al., 2021; Parks et al., 2021; Seghezzo et al., 2021).

An added advantage of conducting research in the context of an examinee’s home is that data can be combined with other forms of digital data. For example, recently research studies have begun to collect naturalistic data from passive sensors that are incorporated in a participant’s daily life (e.g., pedometer [Beattie et al., 2020; Rabin et al., 2019]). TeleNP cognitive data collected in the home-to-home setting can thus be used in parallel to study the ecological validity of how cognitive test performances in one’s everyday environment relate to other aspects of health and daily functioning. Some research groups have also begun testing models of administering cognitive tests which are automated in smartphone apps, requiring no examiners (Weizenbaum et al., in press).

(ii) Platforms and devices: There is little data scrutinizing the variability associated with the type of platform being used for remote cognitive assessment or on how the type and number of devices being used may affect cognitive demands placed on examinees. For example, in the last author’s (PS) anecdotal research experience, the use of Zoom® versus TeamViewer® was found to affect the speed of tasks requiring a remote control feature (Teamviewer® being faster than Zoom®). However, formal studies comparing the impact of different platforms on the collection of cognitive data have not yet been conducted, presenting an important avenue for future research. This is a particularly important question given that some studies use a single platform (e.g., Fox-Fuller et al., 2021; Parks et al., 2021), thereby constraining all examinees through a shared experience. Other TeleNP studies have allowed examinees to use their preferred platform and preferred device (e.g., Seghezzo et al., 2021), which places fewer demands on examinees to engage with a single and potentially unfamiliar device, but may introduce more noise and error. Some evidence suggests that variability across device features (e.g., screen size) and the type of devices (e.g., android versus iOS; laptop versus tablet) can influence the outcomes of cognitive assessments (Germine et al., 2019), making the study of device features an issue of critical importance for TeleNP research.

Similarly, the use of multiple devices to conduct cognitive assessments (with and without video feature) is another aspect of TeleNP that has not been well characterized. Some cognitive tasks, such as the National Institutes of Health (NIH) Toolbox (www.nihtoolbox.org), have the examiner administer the test to the participant via an iPad® in which the video feature (to look at the examinee’s face) is absent due to the iPad’s® functionality during the administration of this task. This can be problematic because unlike the “share screen” feature on some platforms (e.g., Zoom® and TeamViewer®), the examiner administering the NIH Toolbox on an iPad® is unable to view the stimuli as seen by the examinee or see if the examinee is engaging in unwanted test behaviors (i.e., “cheating”). In the last author’s (PS) experience, when the “share screen” feature is enabled by pairing it with another device (iPad® + computer), at times the visual stimuli (e.g., those associated with List Sorting Working Memory Test from the NIH Toolbox) are not displayed in a standard manner (e.g., some stimuli are skipped, or the rate of presentation between two stimuli is dissimilar). Such aspects related to TeleNP assessment are problematic because examiners may not recognize that these errors are happening and the examinee (given the novel nature of the task) may not identify this irregular presentation as problematic. By being able to see the stimuli as it gets presented to the examinee, the examiner can determine when to discontinue the test and thereby safeguard the integrity of the test data.

There are serious clinical and research implications if stimuli are being presented inconsistently to examinees. The nonstandardized administration of tests and interference due to technological variation can result in potentially noisy, if not invalid, data when not caught by the examiner. Of even greater concern is the possibility of arriving at erroneous conclusions about the examinee’s cognitive status based on noise introduced by the technology used for TeleNP evaluation. One method for ensuring a standardized test experience is to send a preconfigured device to the examinee, as is being done in several research studies (see previously mentioned point above about increasing equitable access to TeleNP). However, the cost to do this may be prohibitive in a clinical setting and in some research settings with less funding.

Future directions—clinical TeleNP training

Our findings, along with recent studies, suggests that TeleNP practice is likely to continue in the future under some contexts. For the present, clinics and research labs may wish to develop their own TeleNP certification processes for psychometricians and staff administering tests remotely. In the future, formal certifications in TeleNP could also be created by neuropsychology organizations (e.g., AACN, INS, NAN, SCN) as a way for both trainees and professionals to receive formalized training in TeleNP based on current literature. Clinical psychology graduate programs, especially those with neuropsychology tracks, should also consider updating existing neuropsychological coursework or creating new courses about TeleNP assessment to provide neuropsychology trainees with foundational TeleNP concepts and knowledge prior to clinical rotations. These revised or new courses could cover the TeleNP literature, as well as provide an opportunity for trainees to learn how to administer neuropsychological tests virtually in a low-stakes environment (i.e., with classmates). Finally, incorporation of TeleNP practice as a formal competency of neuropsychology could also be considered during updates to the Houston Conference Guidelines (Hannay et al., 1998). The inclusion of TeleNP as a competency in a revision to the Houston Conference Guidelines would be analogous to AACN’s 2050 Relevance Initiative, which seeks to incorporate specific diversity-related competencies into the revised Houston Conference Guidelines given projected trends in U.S demographics (Stringer et al., 2021).

Limitations of the Present Study

There are some limitations to the current study. First, we intentionally restricted our sampling frame to individuals working with adults to reasonably limit the scope of our study, given that individuals engaged in remote cognitive assessment with children have multiple additional factors to consider (Ransom et al., 2020). Similarly, we limited our sample to individuals working in the USA to preserve a reasonable scope and focus on responses that may be more relevant to practice in the USA (e.g., insurance billing). Future research should look at the experiences of individuals who engage in remote cognitive assessment with children, as well as the cross-cultural experiences of individuals from other nations and regions of the world where the practice of TeleNP may be substantially different relative to the USA. Additionally, there were other related questions that were not included in our survey which limits the present study, such as the nature of telehealth consent forms used by the respondents.

This study also asked participants to self-identify as licensed neuropsychologists (or non-neuropsychologists engaged in remote cognitive assessment), which could contribute to sampling bias. Participants were highly encouraged, though not required, to list their name and affiliations, and for those who did report this information, we did not find evidence of oversampling of certain neuropsychology group practices or hospital services. The investigators also used Qualtrics to run the survey, which records IP addresses associated with responses. This approach of documenting IP addresses, at least in part, mitigates the chances of duplicate responses from the same respondent, though we cannot rule out the possibility that respondents could have submitted multiple responses to the survey on different devices or that participants inaccurately self-reported their eligibility, as is the case in all anonymous survey research.

Some characteristics of the sample may have limited the generalizability of our study. Our sample was relatively small relative to the true population of licensed U.S. neuropsychologists and other individuals engaged in remote cognitive assessment with adults (Sweet et al., 2021). Our sample was also heterogenous and spanned many different practice/trainings settings. After examination of pooled responses, additional subdivision of our sample into those using TeleNP for clinical versus research purposes further restricted our sample size. Quantitative analysis to compare the clinical versus research groups was not conducted because this was not the primary focus of the study. However, we wanted to explore this issue given that no specific literature, to our knowledge, exists on this topic. The clinical versus research groups were identified based on a single item in which the percent of TeleNP use for clinical versus research work at the peak of their TeleNP practice was obtained. This meant that the same individual could have endorsed 100% clinical and 100% research use if they were using TeleNP 100% of the time. Sixteen individuals (including nine licensed U.S. neuropsychologists) reported that at their peak TeleNP use 100% of their research and clinical work was conducted using telehealth technologies, i.e., these individuals overlapped in both the “research” and “clinical” groups. Although it would have been ideal to have neuropsychologists who are exclusively involved in either clinical or research work, the practical reality is that it is often hard to dissociate between these two groups given the natural overlap that exists in clinical and research work in our field. Moreover, given the nature of research versus clinical work, the composition of the groups differed with mostly nonlicensed psychologists including graduate students and research assistants endorsing TeleNP use for research purposes, whereas licensed psychologists endorsed using it for clinical reasons. Therefore, the comparison between the two groups must be interpreted with caution. Future researchers may consider explicitly asking the participants about their primary roles (research, clinical, or both contexts) to more clearly compare TeleNP use among the various groups. However, we believe that information we gathered is still meaningful and highlights the commonalities associated with TeleNP use and its challenges. Future work directly addressing this question will be required to replicate the current observations. Lastly, our respondents also mostly worked in urban and suburban environments. Although the area of typical practice does not necessarily preclude the examiner from practicing TeleNP in rural areas as well, it is important to note this limitation given that more frequent access to broadband internet exists in urban areas relative to rural areas.

Summary

The COVID-19 pandemic has resulted in growing interest in remote means of cognitive assessment for clinical and research purposes. Survey findings of individuals conducting TeleNP evaluations in the USA with adults suggest that for the majority of the field, TeleNP was initiated largely in the context of the COVID-19 pandemic and was, at least for a time, occurring largely in settings in which the examinee was in their home for the assessment, which is a departure from the clinic-to-clinic model used by most prior TeleNP research studies. Issues with internet connectivity, environmental distractions, and examinees’ limited or lack of access to requisite technologies for TeleNP evaluations were among the top general challenges reported with TeleNP. Based on these challenges and the respondents’ suggestions for mitigating these difficulties, we provided a synthesized list of recommendations that can help guide individuals using TeleNP in clinical or research settings. The findings from this survey represent the perspectives from the community of individuals engaged in work involving remote cognitive assessment and can be incorporated into discussions about formalized TeleNP competencies.

Click here for additional data file.