Availability of researcher-led eHealth tools for pain assessment and management: barriers, facilitators, costs, and design.

INTRODUCTION: Numerous eHealth tools for pain assessment and management have been developed and evaluated with promising results regarding psychometric properties, efficacy, and effectiveness. Although considerable resources are spent on developing and evaluating these tools with the aim of increasing access to care, current evidence suggests they are not made available to end users, reducing their impact and creating potential research waste.
METHODS: This study consisted of 2 components: (1) a systematic review of eHealth tools for pediatric pain assessment and/or management published in the past 10 years, and (2) an online survey, completed by the authors of identified tools, of tool availability, perceived barriers or facilitators to availability, grant funding used, and a validated measure of user-centeredness of the design process (UCD-11).
RESULTS: Ninety articles (0.86% of citations screened) describing 53 tools met inclusion criteria. Twenty-six survey responses were completed (49.06%), 13 of which (50.00%) described available tools. Commonly endorsed facilitators of tool availability included researchers' beliefs in tool benefits to the target population and research community; barriers included lack of infrastructure and time. The average cost of each unavailable tool was $314,425.31 USD ($3,144,253.06 USD total, n = 10). Authors of available tools were more likely to have followed user-centered design principles and reported higher total funding.
CONCLUSION: Systemic changes to academic and funding structures could better support eHealth tool availability and may reduce potential for research waste. User-centered design and implementation science methods could improve the availability of eHealth tools and should be further explored in future studies.

1. Introduction

The growing field of eHealth, defined as the use of information and communications technology for health,[135] has the potential to provide numerous benefits for consumers and health systems, such as improving the accessibility and cost-effectiveness of health care.[37,81] A plethora of eHealth tools (eg, mobile applications and online interventions) have been developed for a range of health conditions and symptoms,[90,98,102] including acute and chronic pain,[54,69,117] with promising results regarding their effectiveness.[81] Although considerable grant funding may be spent on the development and evaluation of evidence-based eHealth tools for pain with the rationale that they will improve access to care, current evidence suggests that end users (ie, targeted users of the tool) do not have access to these tools outside research studies, reducing their real-world impact.[81] For example, one systematic review identified 47 articles describing 34 pain-related mobile applications (apps), none of which were publically available in app stores.[66] Reviews of publically available pain-related mobile apps found that 0/283 apps were reported on in published scientific literature[66] and only 1/279 apps was empirically evaluated.[68]

The possibility that significant research funding is spent developing eHealth tools that are never made available to end users is a concerning example of potential “research waste” (ie, avoidable waste in the research process[21]). Barriers to making research outputs available (eg, through commercialization) that are commonly endorsed by researchers include lack of financial support, dedicated time, industry partnerships, and sufficient institutional infrastructure.[77,129] Tool availability may be facilitated by researchers' intrinsic motivations,[70] user-centered design processes (ie, methodology based on the premise that involving end users throughout the tool design process improves user experience and effectiveness[136]), and implementation efforts targeted to eHealth contexts and users.[43,76]

Understanding the barriers and facilitators associated with the availability of pain-related eHealth tools to end users is critical for improving tool availability and reducing potential research waste. This study systematically reviewed eHealth tools for pediatric pain across tool types (eg, mobile apps and online interventions), goals (eg, assessment or management), and pain types (eg, acute, chronic, and disease-related pain). It focused on pediatric pain because pain is a common experience for children and adolescents[35,63] and eHealth tools are often the preferred modality among this age group.[110] This study makes novel contributions to the field by examining potential barriers and facilitators (including user-centered design processes) to pediatric pain eHealth tool availability and quantifying funding used to develop and evaluate available and unavailable tools.

This study consisted of 2 components: (1) a systematic review of eHealth tools for pediatric pain assessment and/or management published in the past 10 years, and (2) a survey completed by authors of the identified tools regarding their eHealth tool's availability, barriers or facilitators to availability, tool design process, and grant funding used. The 10-year timeline was chosen given the rapid changes in technology in the field of eHealth tools. It was hypothesized that: (1) many researcher-developed eHealth tools meeting inclusion criteria would be unavailable to end users in any form, (2) researcher-reported barriers to tool availability would include outdated technology and system-level barriers (eg, lack of funding and insufficient institutional infrastructure), and facilitators would include researchers' individual beliefs about the importance of tool availability, and (3) considerable research funding would be spent on tools that are unavailable to end users.

2. Method

2.1. Systematic review

The systematic review was conducted following established guidelines for this method[55,87] and was registered with Prospero (registration: CRD42017069910).

2.1.1. Search strategy

A systematic search was conducted in 4 databases (PubMed, PsycINFO, CINAHL, and EMBASE) from May 2 to 3, 2017. The search strategy included terms for eHealth (eg, eHealth and mHealth), pain (eg, pain, chronic pain, headaches, and needles), assessment and management (eg, assessment, intervention, and therapy), and children (eg, child and pediatrics) (see Appendix A for full search strategy, available at http://links.lww.com/PR9/A31). Medical subject headings (MeSH terms) and other key terms were also used. The search strategy was developed in consultation with a librarian and experts in the fields of pediatric pain and eHealth tool development, and followed previous research validating optimal search terms for identifying pediatric research.[73] Additional hand searching of previous reviews on similar topics[66,114] and reference sections of included full-text articles was also completed.

2.1.2. Eligibility criteria

Articles were eligible for inclusion in the review if they met the following criteria: (1) the article described an empirical study (published study or unpublished dissertation) written in English and published within the past 10 years (ie, January 1, 2007–May 3, 2017), (2) the article described the development of an eHealth tool for pediatric pain assessment or management and/or evaluated the benefits of its use in the target population, and (3) the tool was studied in children and adolescents (aged 0–18 years, sample median age <19 years) or their parents/caregivers. The 10-year timeline for the systematic search was chosen given the rapid evolution of eHealth technology that can quickly lead to developed eHealth tools becoming out of date.

2.1.3. Study selection and data extraction

Citations identified in the systematic search were imported into Covidence systematic review software[31] and results were deduplicated. Two study authors (K.S.H. and P.R.T.) independently reviewed the titles and abstracts of each remaining citation to determine whether they met inclusion criteria; discrepancies were resolved by consensus. Next, a full-text screen was completed of all remaining articles (by K.S.H. and P.R.T.) to determine final inclusion and discrepancies were again resolved by consensus. Before completing data extraction, a list of all included articles, organized by tool, was developed by one author (K.S.H.) with any unclear situations resolved through consultation with the other team members.

A data extraction manual was developed by the authors for this review. During data extraction, information was collected in the following areas: study identifiers (eg, title, corresponding name and contact author, and name of eHealth tool described), tool focus (eg, pain assessment, management, or combination of both; aimed at children/adolescents or their parents/caregivers; and type of pain tool was intended for), characteristics of the tool (eg, type of tool, theory on which it was based), and information on the study described (eg, study design, participants, and results). Whether or not each article evaluated the following aspects of the eHealth tool was recorded: functionality (ie, whether the tool worked correctly and did the functions it was designed to do), usability (ie, ease of use, ease of learning to use the tool, and degree to which the tool could be used by the target population to achieve outcomes with effectiveness and efficiency), accessibility (eg, compliance with Web Content Accessibility Guidelines 2.0 [WCAG 2.0[133]], guidelines for making web content accessible for individuals with various disabilities developed collaboratively with international input from industry, governments, researchers, and disability organizations), user experience (ie, participants' experiences with using the tool, and perceptions and responses resulting from using the tool), and feasibility (ie, whether the tool was suitable to use in the target population or practical for use in everyday life). When one article described multiple studies or samples, data from each were extracted separately. Data were extracted from each article by 2 independent coders (K.S.H. and P.R.T.) and entered into an Excel spreadsheet; disagreements were resolved by consensus, consistent with established systematic review guidelines.[55] Descriptive statistics were used to summarize the results of the data extraction process.

After identification of the included eHealth tools, tool availability was examined by searching Google and in app stores (Apple App Store, Google Play, Windows Store, and Blackberry World, following methods from previous reviews[66,68]). These searches served to gather availability information for all tools meeting inclusion criteria and supplemented the voluntary tool author survey data. A tool was considered available if any means of accessing the tool was evident based on these searches (eg, available for free or for purchase, with a specific user account or health system, from a website, app store, or other location, etc.). These searches were conducted independently by 2 individuals, with discrepancies discussed and resolved by a third party (K.S.H.). Availability status and study of particular characteristics of the eHealth tools (functionality, usability, accessibility, user experience, and feasibility) were compared across tool types (pain assessment tools, pain management tools, and tools combining assessment and management functions) using χ2 tests.

2.2. Author survey

2.2.1. Participants

Corresponding author contact information was identified from each included publication and authors were invited by email to take part in the survey. If the email address provided was no longer active, a coauthor whose contact information was available was invited to participate. Where the same eHealth tool was described in multiple articles, only the corresponding or alternate coauthor of the original article was invited to participate in the study.

2.2.2. Measures

Authors were invited to complete the survey either by telephone or online based on their preference; all participants chose to complete the survey online. The survey was developed for the current study and included questions modified from previous surveys addressing similar topics in different research fields.[3,77,129] The survey also included a previously developed validated measure of the user-centeredness of a tool's design process, the User-Centered Design-11 (UCD-11[128]). Survey questions addressed the following topics: (1) availability of the eHealth tool and, if available, its form (eg, mobile app, online intervention) and cost (free or paid), (2) barriers or facilitators to making the tool available (questions modified from Refs. 77,129; authors rated the extent to which each of 17 items was a barrier/facilitator of making their tool available on a 5-point scale ranging from strongly disagree to strongly agree), (3) the amount and currency of grant funding secured for developing and/or evaluating the tool and making it available to the public, if applicable, (4) the user-centeredness of the tool's design process (UCD-11[128]; 11 yes/no items regarding use of user-centered design strategies in tool development, original study α = 0.74, current study α = 0.85), and (5) author demographic information (questions modified from 3). A copy of the survey is available in Appendix B (available at http://links.lww.com/PR9/A31). To protect participant privacy, survey responses were not linked to a particular eHealth tool (ie, the survey was designed to be anonymous); however, participants were informed in the consent procedures that anonymity could not be guaranteed given the small, publically known pool of potential participants for the study.

2.2.3. Procedure

The study protocol was approved by the IWK Health Centre Research Ethics Board. Authors of included articles were invited to complete the survey or to forward the invitation to a coauthor if they believed they could better report on the described eHealth tool (eg, corresponding author was a trainee and felt their supervisor could better report on the tool). The email invitation introduced participants to the study and included a link to an online consent form with additional information about the study. Authors were also informed that they could decline to participate in the study and would not be contacted further. Reminder emails were sent 2 and 3 weeks after the initial invitation to those authors who had not yet contacted the research team (eg, to set up a telephone interview or to decline study participation).

All participants opted to complete the survey online. They did so by clicking on a link provided in the invitation email. They were provided with an online version of the study consent form and asked to click “I agree” to indicate their consent to participate in the study. Participants could return to previous pages to change their responses if desired. The online survey process took approximately 20 minutes to complete.

No compensation was provided for participating in the study. It is considered common practice by researchers in many fields to share additional information about their publications when requested by other researchers (eg, to provide study summary statistics for inclusion in meta-analyses). This survey invitation was considered to be an extension of this type of academic collaboration.

2.2.4. Data analysis

Descriptive statistics were used to describe the results of the author survey. Survey responses were separated into 2 groups based on whether they pertained to tools available or unavailable to end users. The frequency with which each barrier and facilitator of tool availability was endorsed at each level and the average score for each item was calculated to determine the most strongly endorsed barriers or facilitators in each sample subgroup. All reported grant funding amounts were converted to US dollars (USD) using exchange rates on November 13, 2017, before calculating total and average amounts of funding spent on the development and evaluation of available and unavailable tools. Authors' responses to the UCD-11 were totaled, and an independent-samples t test was used to compare available and unavailable tools on their total UCD-11 scores.

3. Results

3.1. Systematic review

The screening process for the systematic review is illustrated using the PRISMA model[87] in Figure 1. The systematic search identified 14,285 citations; 3,817 duplicates were removed, resulting in 10,468 unique citations. The titles and abstracts of the 10,468 citations were screened, resulting in 215 articles identified as relevant for full-text review. The full-text review resulted in 90 articles meeting inclusion criteria, describing 53 unique eHealth tools across 97 studies. For descriptive information on the included eHealth tools and articles describing them, see Table 1.

Figure 1.

PRISMA diagram.

Table 1

eHealth tools and studies included in systematic review.

Twenty-nine tools (54.71%) addressed pain management and 16 tools (30.19%) focused on pain assessment. Eight tools (15.09%) addressed both pain assessment and management. Thirty-four tools (64.15%) were intended for use by children and/or adolescents, 9 tools (16.98%) were intended for use by parents/caregivers of children and adolescents, and 10 tools (18.87%) were aimed at both. Half of the included eHealth tools were focused on chronic pain conditions (n = 26; 49.06%; eg, headaches, juvenile idiopathic arthritis); among the remainder, tools were intended for acute procedural pain (n = 7; 13.21%), cancer-related pain (n = 5; 9.43%), pain related to sickle cell disease (n = 5; 9.43%), postoperative pain (n = 4; 7.55%), or pain in children with cerebral palsy (n = 1; 1.89%). For 5 tools, a particular pain type was not specified (n = 5; 9.43%). Thirty-two tools could be used on computers (60.38%), 23 could be used on mobile devices (43.40%), and 12 could be used on other devices (22.64%; eg, personal digital assistants and devices developed by the authors); note that tools could be used on more than one device (both computers and mobile devices, n = 11; both computers and other devices, n = 2; and both mobile devices and other devices, n = 1). Twenty-eight tools were reported to be web-based (52.83%); others were reported to be iPhone/iOS apps (n = 5; 9.43%), Android apps (n = 1; 1.89%), other app types (n = 1; 1.89%), or other tool types (n = 22; 41.51%; eg, standalone devices and computer software programs). Seventeen tools reported being based on a particular theoretical model (32.08%; eg, cognitive behavioural therapy[12]; gate-control theory of pain[82]; Pender's Health Promotion Model[99]).

Among the 97 included studies, the most common study type was a validation study (n = 28; 28.87%), followed by randomized controlled trials (n = 21; 21.65%), observational studies (n = 18; 18.56%), “other” study types (n = 11; 11.34%; eg, implementation studies, single-case experimental designs, and feasibility studies), usability studies (n = 13; 13.40%), and pre–post designs (n = 6; 6.19%). None of the included studies described case studies or case series. The 97 studies described within 90 articles included a total of 9,035 children and adolescents, 3,314 parents/caregivers, 214 health care professionals, and 33 other participants (eg, researchers and survey respondents who did not specify a role). Fifty-seven studies described assessing tool feasibility using various methods (58.76%; eg, adherence to tool use during study, completion rates, and time required to complete interventions). Forty-nine studies reported assessing outcomes related to user experience (50.52%; eg, parent and child reports of acceptability or satisfaction with the tool, preference for eHealth tool over standard tools, and feedback on tool functions). Usability was described as being assessed in 30 studies (30.93%; eg, children's ability to use the tool effectively, children's understanding of tool functions, and ratings of ease of use), and 20 studies reported assessing functionality (20.62%; eg, rates of tool malfunctioning or usage errors). None of the studies described assessing accessibility (eg, WCAG 2.0 compliance). The majority of studies (96/97, 99.00%) reported at least some positive results regarding the eHealth tool examined (ie, results supporting the tool's efficacy or effectiveness for at least one outcome; results supporting the tool's usability, feasibility, etc.), and all tools (53/53, 100.00%) had studies reporting positive results on them.

Web searches conducted to examine the availability of each tool indicated that only 15 of the 53 identified eHealth tools were available to end users in some form (28.30%). Among the 15 available tools, 9 (60.00%) required some type of permission to gain access (eg, applying for the ability to access the tool; tool was only available to patients of particular clinics or research participants in particular studies). Four tools (26.67%) were found to be available through the Apple App Store, 4 through Google Play, and one through Windows Store.

There was no significant difference in the proportion of tools found to be available (as examined by web searches) based on tool type (pain assessment, pain management, or combined assessment/management tools), χ2(2) = 0.650, P > 0.05. Regarding tool characteristics studied, a lesser proportion of pain assessment tools (44.8%) had at least one study examining their user experience compared with pain management tools (75.0%) or combination tools (87.5%), χ2(2) = 6.821, P < 0.05. A lesser proportion of pain management tools had at least one study examining usability (12.5%) compared with the other tool types (assessment: 51.7%; combination: 62.5%), χ2(2) = 8.244, P < 0.05. There were no significant differences in proportions of tool types having had functionality (χ2(2) = 0.810, P > 0.05), feasibility (χ2(2) = 0.119, P > 0.05), or accessibility (0 studies of any tool type) examined.

3.2. Author survey

Corresponding authors for each of the 53 unique eHealth tools were invited to participate in the survey. Authors (n = 4) who were corresponding author for more than one eHealth tool were asked to complete the survey once for each of their eHealth tools. Twenty-six responses to the online survey were received (49.06% response rate).

3.2.1. Availability of eHealth tools

Among the 26 responses received, 13 tools (50.00%) were identified as currently being available in some form. These tools were reported to be available to the general public (n = 5; 38.46%) or to patients of a particular clinic or health system (n = 8; 61.54%). Ten tools were reportedly available on websites (76.92%), 3 on the Apple app store (23.08%), one through Android app store (7.69%), and one through specialty clinic (7.69%; tools could be available in more than one location). Twelve available tools (92.31%) were reported to be free of cost; one respondent abstained from responding about cost. Regarding ownership, available tools were reported to be owned by the author's institution (n = 9; 69.23%), by both the author and their institution (n = 2; 15.38%), by the author themselves (n = 1; 7.69%), or by creative commons license (n = 1; 7.69%). Only 4 authors of available tools reported attempts to commercialize their tool (30.78%).

3.2.2. Facilitators and barriers to eHealth tool availability

3.2.2.1. Available tools

Descriptive statistics summarizing author responses regarding facilitators of tool availability are provided in Table 2 (note that authors only completed this section of the survey if they reported that their tool was currently available to end users in some form). The most commonly endorsed facilitators (ie, those with the highest mean scores where 5 = strongly agree and 1 = strongly disagree) were (1) belief in benefit to society/target population, (2) belief that making tool available is important to your research field, (3) belief that making tool available is important to academia, (4) tool had promising clinical/commercial application, and (5) financial support.

Table 2

Frequency of author responses regarding facilitators of eHealth tool availability (n = 13 available tools).

3.2.2.2. Unavailable tools

See Table 3 for frequencies of author responses regarding barriers impeding tool availability (note that authors only completed this section of the survey if they reported that their tool was not currently available to end users in any form). The most commonly endorsed factors (ie, those with the highest mean scores) were (1) lack of infrastructure to support tool availability, (2) lack of time, (3) not aware of how to commercialize or make tool available, (4) lack of industry partners, and (5) outdated technology of tool.

Table 3

Frequency of author responses regarding barriers to eHealth tool availability (n = 13 unavailable tools).

3.2.3. Funding

3.2.3.1. Available tools

Authors of 12 of the 13 available tools answered the survey questions about grant funding amounts; a total of $5,699,146.46 USD in grant funding was reportedly spent on the development and testing of all these eHealth tools combined (n = 12 tools with completed funding information; for one tool, grant sources but not amounts were reported). The average amount of grant funding reportedly spent on development and testing of each eHealth tool was $474,928.87 USD (n = 12 tools included in calculation; median = $157,971.96, range = $2,508,250.00, interquartile range = $671,033.90). When asked about the funding spent specifically on making the tool available to end users, 8 authors reported a total of $165,900.00 USD was used; 5 authors reported that they preferred not to answer this question. For 4 of 13 tools (30.77%), the principal investigator had reportedly budgeted in the original grant for work that would make the tool available to end users. Six of the responses (46.15%) indicated that making the tool available to end users was an expectation of the grant funding. Five responses (38.46%) indicated that additional funding was secured for making the eHealth tool available to end users.

3.2.3.2. Unavailable tools

Authors of 10 unavailable tools answered the survey questions about grant funding amounts; a total of $3,144,253.06 USD in grant funding was reportedly spent on the development and testing of all these eHealth tools combined (n = 10 tools with completed funding information; for one tool, grant sources but not amounts were reported, and 2 authors indicated that they preferred not to answer this question). Three tools were reported to have had no grant funding used in the development and testing of the tool. The average amount of funding reportedly spent on each unavailable tool was $314,425.31 USD (n = 10 tools included in calculation; median = $11,336.50, range = $2,207,260.00, interquartile range = $347,975.54).

3.2.4. Design process

3.2.4.1. Available tools

Among the 13 eHealth tools available to end users, 7 (53.85%) were reportedly developed by a contracted third party, including private companies (n = 3), a combination of private company and in-house development (n = 2), other organizations (n = 1), or freelancers (n = 1). Six tools (46.15%) were reportedly developed in-house. Steps followed in the user-centered design process are depicted in Table 4. Available tools scored significantly higher on user-centeredness of the design process (ie, UCD-11 total score) compared with the unavailable tools, t(16.01) = 2.33, P < 0.05.

Table 4

Author responses to questions regarding user-centeredness of eHealth tool design process.

3.2.4.2. Unavailable tools

Among the 13 unavailable eHealth tools, 7 were reportedly developed in-house (53.85%) and 6 (46.15%) were contracted out (n = 5 to a private company; n = 1 to a student). Steps followed in the user-centered design process are shown in Table 4.

3.2.5. Author demographics

3.2.5.1. Available tools

Among the 13 tools reportedly available to end users in some form, eleven had authors who identified primarily as researchers; 2 identified as “other.” Most authors were primarily affiliated with an academic institution (n = 7), followed by a hospital (n = 4) or research centre (n = 2). Authors were most commonly situated within the fields of nursing (n = 7), psychology (n = 4), or other (n = 2; pain, computer science). Available tools most often had authors who self-identified as mid career (10–20 years; n = 7), early career (less than 10 years; n = 4), or senior career (more than 20 years; n = 2). No available tools were authored by trainees or postdoctoral fellows.

3.2.5.2. Unavailable tools

Of the 13 tools reported as unavailable to end users, eleven (84.62%) had authors who identified their primary role as researchers, and 2 (15.38%) had authors who identified primarily as clinicians. These authors were most commonly affiliated primarily with academic institutions (n = 10) or hospitals (n = 3). Tools were most commonly authored by individuals in the fields of psychology (n = 5), nursing (n = 4), anesthesia (n = 1), or other (n = 3; pain, pediatrics, surgery), and by authors identifying as mid career (n = 4), followed by early career (n = 3), senior career (n = 3), postdoctoral fellow (n = 2), or trainee (n = 1).

4. Discussion

This study provides a systematic review of the recent empirical literature on eHealth tools for pediatric pain assessment and management and makes several novel contributions to this field. Results build on past research by describing author-reported barriers and facilitators to tool availability, funding secured for development and evaluation, and authors' use of user-centered tool design. As hypothesized, few pediatric pain–related eHealth tools reported in the published literature are available to end users (15/53 tools, 28.30%). This finding is consistent with previous research on mHealth tools for pain[66] and web-based interventions for other health conditions,[102] but must be considered in light of the fact that the current study included only published research. Thus, the extent to which unpublished tools are available to end users could not be determined.

Facilitators of tool availability most commonly endorsed by authors of available tools included personal beliefs in the importance of making their tools available to end users. This is consistent with past research on the role of researchers' intrinsic motivations in their engagement in research commercialization.[70] Many of the barriers that authors of unavailable tools endorsed were system-level issues such as lack of infrastructure to support making tools available, consistent with previous studies of researcher-reported barriers to making research outputs available through commercialization.[77,129] These results suggest that although some researchers may be motivated by strong personal beliefs to make their tools available, they are impeded from doing so because of systemic barriers and lack of support. These results contribute to the literature on the availability of pain-related eHealth tools by identifying potential targets for supporting researchers in making their eHealth tools available to end users.

Analysis of grant funding showed that on average, over $300,000 USD was spent to develop and/or evaluate each tool that was subsequently unavailable to end users (total of $3,144,253.06 USD, n = 10 tools included). This expenditure represents potentially wasted research resources and lost impact for end users. Results of this study showed that user-centered design processes were associated with tool availability, and thus, with reducing potential for research waste. This is consistent with the hypothesis that using user-centered design processes for eHealth tools may optimize tool design and adoption by end users[136,137] and is a novel finding of the current study. These results may also reflect that user-centered design is more likely to be used by teams with greater funding availability or overall higher research quality, given that the consultations and iterative design required may take more time and resources than other methods.

Taken together, the results of this study suggest there may be little focus on implementation and commercialization processes by researchers who develop eHealth tools and perhaps by the academic and granting institutions in which they are situated. There are several possible contributing factors to this situation. Researchers may be most focused on demonstrating the efficacy or effectiveness of their tools; efforts to implement eHealth tools and other health interventions are often haphazard or ineffective.[43] In addition, researchers may not be rewarded in career-relevant ways (eg, in consideration of tenure and promotion) for efforts such as implementation or commercialization, which do not map onto traditional metrics (eg, number of publications). Although commercialization efforts are often well considered by academic institutions, most tools are made available to users at no cost, thus realizing no financial benefit for the developer or their institution. Funding agencies may not prioritize knowledge translation and implementation and, thus, not provide appropriate budgets to support the initiation and maintenance of tool availability. Although the current study shows that academic eHealth tool developers are driven by intrinsic motivations for making their tools available to end users, it is likely difficult for many researchers to achieve this alone.

Several recommendations for improving the availability of eHealth tools for pediatric pain assessment and management can be made based on the results of the current study. It is important to recognize that individual researchers, end users, and larger systems (eg, academic institutions, granting agencies, and health systems) all have roles to play in improving eHealth tool availability and reducing potential research waste.[86,91] Increased engagement of end users and other stakeholders from the earliest stages of the research process would allow researchers to better understand their priorities and perspectives and to develop more effective tools for end users, whether based in eHealth technology or not, because “sometimes the best solution is a human solution, not a technological one.”[101] Researchers may be able to increase the chances of their tool becoming available and being effectively used by relevant end users by using user-centered design processes.[109,136,137] Incorporating implementation research methods and planning for potential tool availability earlier in the research cycle may also be helpful. Research has been conceptualized as a pipeline[11] from efficacy studies (examining the effect of interventions on clinical outcomes as studied under highly controlled conditions[30]) to effectiveness studies (examining the effect of interventions under less controlled, more realistic conditions[108]) to implementation research (studies of methods to promote uptake of research evidence or products into practice in intended contexts[11,100]). Given the rapid changes in technology that can lead to eHealth tools becoming out of date, reported as a barrier by developers of unavailable tools in the current study, efficient means of moving through the research pipeline while still covering each important component are needed. Increased use of study designs that facilitate efficient study of implementation outcomes earlier in the research process (eg, hybrid effectiveness-implementation designs, which explore tool effectiveness and implementation simultaneously[13,33]) may be particularly helpful to eHealth researchers. Researchers should consider such research designs as a potential avenue for moving research forward at a pace that better fits changing technology in the field of eHealth while maintaining thorough examination of tools' efficacy and effectiveness. Researchers developing eHealth tools may also benefit from additional training in patient-centered research, knowledge translation, and implementation science methods.[10] Similarly, researchers should incorporate collection of cost-effectiveness data into research designs; lack of information regarding initial and maintenance costs has been identified as a reason that eHealth tools may not be adopted by end users.[46] Use of these strategies may facilitate researchers in establishing the efficacy, effectiveness, safety, and usability of their eHealth tools, which are critical to establish before tool availability, in an efficient way that fits the pace of technology change in the field of eHealth.

Within academic institutions, reward structures could be altered to appropriately reward researchers for their engagement in the extensive effort required to make their eHealth tools available to end users. These efforts should be recognized in the assessment of research outcomes and researchers' performance measures. Such shifts have begun to occur in some organizations (eg, Faculty of Medicine, University of Toronto[9]). Academic institutions could also increase the culture of innovation and entrepreneurship among all faculties and better support researchers in exploring various pathways to tool availability, including commercialization. On the level of funding agencies, increased funding opportunities aimed at supporting tool availability (both initially and long term) and commercialization processes are needed. Granting agencies should require researchers to demonstrate plans for tool availability and sustainability in their proposals and should assist in developing partnerships between researchers and others (eg, industry partners) to enhance tool availability. Researchers have debated whether increased focus on commercialization is appropriate within the academic environment and the impact of this focus on research integrity.[20] Other models of making tools available to end users, outside simply pursuing commercialization and profit exploitation, should be explored.

Results must be considered in light of several limitations. First, response bias likely impacted author survey results, as a greater proportion of authors of available tools completed the survey compared with authors of unavailable tools. Also, authors of unavailable tools who completed the survey may have greater personal interests in the topic than authors of unavailable tools who did not complete the survey. Given the limited response rate to the survey, it is unclear how the results of the study generalize to the broader population of researchers who have developed eHealth tools for pediatric pain. Second, our analytic approach was somewhat limited due to concerns about participant confidentiality within a small, publically known pool of potential participants. As such, survey responses and data extracted from published articles were not linked, relationships between tool characteristics (eg, focus on pain assessment vs management, target user population) and availability could not be explored, and author-reported grant information could not be verified. The extent of evidence supporting each tool was not examined, and thus the relationship between evidence base and tool availability could not be examined. Although all tools had at least some positive results reported on in the included studies, it may be that tools with less evidence supporting their use were less likely to become available to end users. Finally, this project focused on pediatric pain tools and may differ from what might be discovered among eHealth tools for adult pain or other health conditions.

Future research should extend beyond barriers and facilitators of eHealth tool availability by exploring the perspectives of other stakeholders involved in eHealth tool design, evaluation, and dissemination. End users, industry partners, and policy decision makers have important perspectives that should be explored.[109] Currently, little is known about the demand for pediatric pain–related eHealth tools. Preferences for eHealth tools have been documented in some samples,[109,122] and the prevalence of children's pain[63,93,111] and access to technology[26] suggests that eHealth tools may be useful in this population; however, study of the needs of target end users is required to better understand whether eHealth tools are appropriate for various pediatric pain populations. Similarly, research on the best methods for making tools available to pediatric populations (eg, standalone eHealth tools provided directly to end users vs eHealth tools provided as part of larger eHealth systems implemented within health care systems) is also needed. Future research should prospectively examine specific predictors of tool availability, such as the use of user-centered design processes and implementation research methods, to ensure that eHealth tools actually do benefit users and not contribute to potential research waste.

Disclosures

The authors have no conflict of interest to declare.

K.S. Higgins is supported by a CIHR Doctoral Research Award, a Maritime Strategy for Patient-Oriented Research (SPOR) Support Unit Student Award, and a Nova Scotia Health Research Foundation Scotia Support Grant awarded to C.T. Chambers. P.R. Tutelman is supported by a CIHR Vanier Canada Graduate Scholarship. C.T. Chambers is supported by a Tier 1 Canada Research Chair and is the senior author on this article. C.T. Chambers' research is also supported by the Canadian Institutes of Health Research (#139704). H.O. Witteman is supported by a Fonds de recherche du Québec—Santé Research Scholar Junior 1 career development award. The infrastructure for this study was provided by a Canada Foundation for Innovation grant to C.T. Chambers. Open access publication was funded by a Maritime SPOR Support Unit Open Access Publication Bursary awarded to K.S. Higgins.