Development of a rehabilitation researcher survey of knowledge and interest in learning health systems research.

Introduction: LeaRRn, an NIH-funded rehabilitation resource center, is dedicated to developing learning health systems (LHS) research competencies within the rehabilitation community. To appropriately target resources and training opportunities for rehabilitation researchers, we developed and pilot tested a survey based on AHRQ LHS research core competencies to assess the training needs of rehabilitation researchers interested in LHS research.
Methods: Survey items were developed by the investigative team and iteratively refined with the assistance of an expert panel using two rounds of content validation. Survey items addressed knowledge of, ability to apply, and interest in LHS research competencies. The survey was pre-pilot tested with six rehabilitation professionals, refined again, and then pilot tested. Time to complete the survey was measured. Spearman correlations examined relationships between knowledge and ability.
Results: A 78-item survey was pilot tested. Forty-five individuals completed the pilot survey in full (71% female, 84% white, and 93% non-Hispanic). Due to concerns about response burden (mean 15 minutes to complete) and strong correlation between "knowledge" and "ability" ratings (all rho >0.57), "ability" was dropped, resulting in a 55-item survey assessing "knowledge" and "interest" in LHS research competencies. Conclusions: We developed a survey of knowledge and interest in LHS research competencies for rehabilitation researchers. The resulting survey may be used to assess training needs and guide LHS research content development by educators, programs directors, and other initiatives within the rehabilitation research community.

BACKGROUND

The imperative for developing Learning Health Systems (LHS) has been endorsed by many since the publication of the original Institute of Medicine (IOM) reports and the widespread adoption of electronic health records. At the same time, there has been clear recognition of the need to train a new cadre of individuals who can conduct LHS research. According to the Agency for Healthcare Research and Quality (AHRQ), a LHS researcher is an individual “who is embedded within a health system and collaborates with its stakeholders to produce novel insights and evidence that can be rapidly implemented to improve the outcomes of individuals and populations and health system performance.”

Conducting rigorous LHS research requires researchers, embedded within health systems, to acquire new skills and competencies. LHS research draws on theoretical and applied methods from a variety of fields. An AHRQ‐funded technical expert panel identified seven key domains and 33 core competencies required for LHS researchers (Table 1). The seven domains include: (a) systems science, (b) research questions, (c) research methods, (d) informatics, (e) ethics of research, (f) improvement and implementation science, and (g) engagement, leadership and research management.

TABLE 1

Original core competencies, initial item generation and process of revision resulting in the final LHS research survey for rehabilitation researchers: organized by LHS research domain

A
Systems Science: To understand how health systems are financed and operate and how to apply systems theory to research and implementation
Original Core Competency	Round 1 item	Action a	Round 2 item	Action a	Pre‐pilot item	Pilot item	Action a	Final item
Demonstrate knowledge of how systems theories can be used to understand how the interactions of the parts of health systems operate to produce value for stakeholders.	Systems theory (conceptual frameworks addressing how the parts of a health system interact to produce value for stakeholders)	Rev	Conceptual frameworks addressing how the parts of a health system interact to produce value for stakeholders	Rev	Conceptual frameworks addressing how the parts of a health system interact to produce value for stakeholders (eg, Complex Adaptive Systems, Social‐Ecological Framework)	Conceptual frameworks addressing how the parts of a health system interact to produce value for stakeholders (eg, Complex Adaptive Systems, Social‐Ecological Framework)	Keep	Conceptual frameworks addressing how the parts of a health system interact to produce value for stakeholders (eg, Complex Adaptive Systems, Social‐Ecological Framework)
Demonstrate systems thinking in the design and conduct of research and implementation of its findings within the context of complex health systems.	Designing research for complex health systems	Rev	Designing and conducting rehabilitation research with health systems	Keep	Designing and conducting rehabilitation research with health systems	Designing and conducting rehabilitation research with health systems	Keep	Designing and conducting rehabilitation research with health systems
	Conducting research in complex health systems	Rev	Conducting rehabilitation research in health systems	Com	‐	‐	NA	‐
	Implementing research findings in complex health systems	Rev	Implementing research evidence in health systems	Keep	Implementing research evidence in health systems	Implementing research evidence in health systems	Keep	Implementing research evidence in health systems
Demonstrate knowledge of the financing, organization, delivery, and outcomes of health care services and their interrelationships.	Financing of rehabilitation services	Keep		Keep	Financing of rehabilitation services	Financing of rehabilitation services	Keep	Financing of rehabilitation services
	Organization and delivery of rehabilitation services	Rev	Organization of rehabilitation services (facilities, equipment, team composition, and training methodology)	Rev	Organization of rehabilitation services (facilities, equipment, team composition, and training)	Organization of rehabilitation services (facilities, equipment, team composition, and training)	Keep	Organization of rehabilitation services (facilities, equipment, team composition, and training)
	Outcomes of rehabilitation services valued by health systems	Rev	Outcomes of rehabilitation services important to health systems	Keep	Outcomes of rehabilitation services important to health systems	Outcomes of rehabilitation services important to health systems	Keep	Outcomes of rehabilitation services important to health systems
	The interrelationships between financing, organization, delivery, and outcomes of rehabilitation services	Keep	The interrelationships between financing, organization, delivery, and outcomes of rehabilitation services	Keep	The interrelationships between financing, organization, delivery, and outcomes of rehabilitation services	The interrelationships between financing, organization, delivery, and outcomes of rehabilitation services	Keep	The interrelationships between financing, organization, delivery, and outcomes of rehabilitation services
Demonstrate ability to assess the extent to which research activities will likely contribute to the quality, equity or value of health systems.	Assessing the extent to which research activities will likely contribute to the quality of health systems	Rev	Assessing the extent to which research activities will improve the quality of health systems	Keep	Assessing the extent to which research activities will improve the quality of health systems	Assessing the extent to which research activities will improve the quality of health systems	Keep	Assessing the extent to which research activities will improve the quality of health systems
	Assessing the extent to which research activities will likely contribute to the equity of health systems	Rev	Assessing the extent to which research activities will improve the equity of health systems	Keep	Assessing the extent to which research activities will improve the equity of health systems	Assessing the extent to which research activities will improve the equity of health systems	Keep	Assessing the extent to which research activities will improve the equity of health systems
	Assessing the extent to which research activities will likely contribute to the value of health systems	Rev	Assessing the extent to which research activities will improve the value of health systems	Keep	Assessing the extent to which research activities will improve the value of health systems	Assessing the extent to which research activities will improve the value of health systems	Keep	Assessing the extent to which research activities will improve the value of health systems

Actions: Rev, Revised; Keep, Keep with no changes; Com, Combined; Drop, Dropped; RH, Dropped, red herring; DLP, Dropped, low priority; NA, Not applicable, item was omitted.

The Learning Health Systems Rehabilitation Research Network (LeaRRn), an NIH‐funded rehabilitation resource center is dedicated to developing LHS research competencies in rehabilitation researchers. This network was established with the long‐term goal of accelerating the translation of rehabilitation research evidence to practice by creating a cadre of LHS researchers that could conduct embedded research within health systems. Despite increasing evidence on the effectiveness of rehabilitation, our knowledge of the barriers to dissemination, implementation, scalability, and sustainability of effective interventions in real‐world care is quite limited. LHS research competencies are needed in rehabilitation to help disseminate and implement best practices across the spectrum of providers and patients. , Value‐based payment models and other payment policy changes for rehabilitation services also require health systems to understand how best practices can be incorporated while de‐emphasizing low value or adverse practices.

Rehabilitation care is particularly suited to a LHS framework as rehabilitation care providers (eg, physical therapists, occupational therapists, speech language pathologists, and rehabilitation psychologists) often see patients over several visits and routinely collect data on impairments and patient‐centered outcomes during an episode of care. With the adoption of EHRs in most health systems and efforts to standardize outcome measurement, rehabilitation care providers now have access to vast amounts of data that can be used to transform into a learning health system (LHS). However, they frequently lack the knowledge and expertise to participate in LHS efforts.

Part of LeaRRn's work in the first year of funding was to survey rehabilitation researchers to identify knowledge gaps and interest in LHS research competencies in order to appropriately target resources and training opportunities. In this paper, we describe our process of developing and pilot testing the survey to assess the needs of rehabilitation researchers interested in LHS research.

METHODS

Overview

The survey was developed using the framework of LHS core research competencies. Survey items were written by the investigative team and were iteratively refined with the assistance of a multidisciplinary expert panel (who had not been involved in initial item writing) using two rounds of content validation. The survey was then tested (pre‐pilot test) on a small group of rehabilitation professionals, refined, and then pilot tested. A summary of the survey development process is shown in Figure 1. A detailed synopsis showing initial item generation and changes made through the item refinement process is shown in Table 1. The final survey contains 55 items addressing level of knowledge and interest in learning more about LHS research competencies.

FIGURE 1

Overview of the survey development process

Creation of the expert panel

We identified rehabilitation professional associations through personal contacts and an internet search. We solicited nominations for an “expert panel” from 17 professional associations that represented a wide variety of rehabilitation disciplines. Ten members representing eight rehabilitation professions (Prosthetics and Orthotics, Speech Language, Physical Therapy, Occupational Therapy, Medicine Research, Psychology Research, Rehabilitation Counseling, and Audiology) were included.

Specifying unique and mutually exclusive LHS competencies

The investigative team reviewed the wording and content of each of the 33 LHS research core competencies as identified by Forrest et al. The team wrote separate items for those core competencies that included more than one concept. For example, the content of the original core competency, “Demonstrate the ability to employ specific quality improvement methods to reduce avoidable variation in clinical processes and outcomes in routine practice” was divided into two items: A)”Quality improvement methods to reduce avoidable variation in clinical processes,” and B) “Quality improvement methods to reduce avoidable variation in clinical outcomes.” This process resulted in 72 items.

Additionally, the team added seven “red herring” items (one per domain) to help us verify that respondents were paying careful attention to the survey content. (Appendix S1). These items were related to health care delivery or research, but were not considered by investigative team members to be relevant for LHS rehabilitation research. Thus, the survey used for content validation (described below) contained a total of 78 items.

Content validation—Round 1

Expert panel members were asked to review the 78 survey items and to rank each item on a three‐point scale based on its clarity (1 = not at all, 2 = somewhat, 3 = very), relevance (1 = not at all, 2 = somewhat, 3 = very), and priority (1 = low, 2 = medium, 3 = high) in relation to LHS in rehabilitation. They were also asked to add any comments on the items to aid in improving the survey and its contents.

We utilized content validation indices to help determine the final instrument content. , Rankings of expert panel members were tallied and separate content validation index (CVI) scores were generated for item clarity, relevance, and priority. Scores were then dichotomized as shown in Table 2. For clarity, items rated as very clear were assigned a value of 1 and items rated as not at all clear or somewhat clear were assigned a value of 0. For relevance and priority, items rated as somewhat or very relevant or medium or high priority were assigned a value of 1 and ratings of not at all relevant or low priority were assigned a value of 0. These dichotomizations were chosen to ensure that the items that the majority of the expert panel rated as not at all or only somewhat clear were revised and that only items that were rated as not at all relevant or low priority were dropped.

TABLE 2

Method of dicohotomizing scores for content validation

	Definition	CVI Dichotomization
		0		1
Clarity	The ease of understanding the wording of the item (ie, the meaning of the item is clear and is free of overly vague language or unnecessary jargon)	Not at all	Somewhat	Very
Relevance	The usefulness and necessity of this item for conducting rehabilitation research within LHS	Not at all		Somewhat	Very
Priority	The importance of training addressing this item for advancing rehabilitation research in LHS	Low		Medium	High

Study co‐investigators (J.S., J.F., L.R., K.P.) reviewed the comments and CVI scores to reach consensus on wording and decisions about which items to exclude for the next round. For all domains, items with content validation index (CVI) scores >0.79 were retained without revision. If an item's relevance and priority scores were >0.79, but the clarity score was <0.79, the item was retained, but revised by consensus of the investigative team. Most items which had a clarity rating >0.79, but relevance and priority ratings <0.79 were dropped. However, some items with relevance or priority <0.79 that were considered by the investigative team to be highly relevant for learning health systems research were retained. In these instances, the items were revised to help clarify relevance for LHS rehabilitation research. In addition to dropping the seven red herring items, 5 items were dropped after Round 1 review and 44 items were revised or combined with other items leaving 61 items for Round 2 content validation.

Content validation—Round 2

The revised item set was then circulated to the expert panel and the content rating process was repeated. One item was dropped after CVI review of Round 2, 5 items were combined, and 14 were revised resulting in a 55‐item survey organized in the seven domains. For some items, we also provided definitions of terminology based on the feedback of the expert panel (Appendix S2). Definitions were written by the investigative team after consulting the literature and, in some cases, simplifying or abbreviating for conciseness. The survey that was pilot tested addressed three areas for each item: level of knowledge, ability to apply this knowledge, and interest in learning more. The survey also included demographic and occupation questions as well as a screening question to ascertain interest in learning more about LHS research.

Pre‐pilot testing

The 55‐item survey was pre‐piloted with six rehabilitation researchers identified by the study investigators. The purpose of pre‐piloting was to test the programmed survey and collect data on time to completion. Participants in the pre‐pilot test sample were asked to complete the survey via the internet using the survey software system Qualtrics. For each item they were asked to: indicate their level of knowledge (novice, intermediate, expert); their ability to apply this knowledge (novice, intermediate, expert); and their interest in learning more about the survey item, that is, the competency (not at all, somewhat, very). They were also asked to track the amount of time it took to complete the survey and to provide any additional comments.

The average time to complete the survey in the pre‐pilot test sample was 15 minutes (range 10‐23 minutes). The investigator team revised response categories for the domains of knowledge and application from “novice,” “intermediate” an “expert” to “none,” “some,” and “a lot” based on feedback about the use of the terms novice and expert. The feedback was that more experience often leads to acknowledging how little one knows about a topic. Additionally, the team revised the response domain name of “application” to “ability” to improve clarity.

Pilot testing

The study was approved by the University of Pittsburgh Institutional Review Board. Pilot testing participants were recruited by members of the expert panel. The refined survey was circulated to 88 colleagues by email. Survey respondents were not identifiable and the expert panelists had no information on whether or not those they had recruited had completed the survey. No incentives were offered for survey completion.

Results of the Pilot test

Fifty‐six individuals began the survey, and 45 completed the survey in full. Of those who did not complete the survey, nine participants only completed the screening questions and two participants only responded to questions in the first domain (system science). Characteristics of participants in the pilot study are shown in Table 3. Briefly, respondents were 71% female, 84% white, and 93% non‐Hispanic. Professions with the greatest representation included research (33%), physical therapy (29%), physical medicine and rehabilitation (20%), occupational therapy (18%), speech language pathology (13%), and prosthetic and orthotics (11%). No demographic information is available for the 11 participants who did not complete the full survey.

TABLE 3

Characteristics of participants in the pilot study

	N = 45
	N (%)
Gender
Male	13 (28.89)
Female	32 (71.11)
Race (all that apply)
American Indian or Alaska Native	1 (2.22)
Asian	5 (11.11)
Black or African American	0
Native Hawaiian or Other Pacific Islander	0
White	38 (84.44)
Other	0
Not reported	1 (2.22)
Ethnicity
Not Hispanic or Latino	42 (93.33)
Hispanic or Latino	1 (2.22)
No reported	2 (4.44)
Profession (all that apply)
Audiology	0
Case Management/Administration	0
Occupational Therapy	8 (17.78)
Physical Medicine and Rehabilitation	9 (20.00)
Physical Therapy	13 (28.89)
Prosthetics and Orthotics	5 (11.11)
Psychology	3 (6.67)
Rehabilitation Counseling	0
Rehabilitation Nursing	0
Rehabilitation Technology/Engineering	1 (2.22)
Research	15 (33.33)
Social Work	0
Speech Language Pathology	6 (13.33)
Other	3 (6.67)
Work setting (all that apply)
Academic institution	34 (75.56)
Acute care hospital	8 (17.78)
Government—for example: VA, DOD	7 (15.56)
Health and wellness facility	1 (2.22)
Hospital‐based outpatient facility or clinic	7 (15.56)
Industry	1 (2.22)
Inpatient rehab facility	7 (15.56)
Integrated system	4 (8.89)
Home care	1 (2.22)
Outpatient office or group practice	5 (11.11)
Research center	8 (17.78)
School system	0
Skilled nursing facility (SNF)/Long‐term care	2 (4.44)
Other	3 (6.67)
Highest earned degree (all that apply)
Bachelor's degree (eg, BA, BBA, BFA, BS)	0
Master's degree (eg, MA, MBA, MFA, MS, MSW)	6 (13.33)
Applied or professional doctorate degree (eg, MD/DO, PharmD, DPT, OTD	13 (28.89)
Doctorate degree (eg, EdD, PhD, ScD)	31 (68.89)
Other	0
Not reported	1 (2.22)
Research experience: I conduct research: (all that apply)
In collaboration with a health system/health care practice	17 (37.78)
In collaboration with researchers as a health system stakeholder (provider, administrator)	8 (17.78)
As employee of a health system/health care practice	24 (53.33)
No experience conducting research within a health system or health care practice	12 (26.67)
Other	1 (2.22)
How familiar are you with the concept of a learning health system
Not familiar at all	22 (48.89)
Somewhat familiar	16 (35.56)
Very familiar	7 (15.56)
Missing	n/a

Time to complete the survey was extracted from Qualtrics. Mean time of all participants that completed the survey in full was 26 minutes (SD 33 minutes). Because Qualtrics captures the entire time to complete a survey, it also counts time for those who stop in the middle, close their browser and return again. As a result, we identified and removed five outliers whose range of completion time was 47‐192 minutes. Once outliers were removed, mean time was 16 minutes (SD 8 minutes). Item completion rates were calculated and items that were skipped by ≥5% of respondents were evaluated and considered for elimination. Spearman correlations between ratings of Knowledge and Ability were examined for each item to evaluate redundancy and potential to reduce response burden by eliminating one of the rating categories.

Only one item, “Composing research questions that address meaningful clinical and policy issues,” was skipped by more than 5% of pilot respondents. Because of the relevance and importance of the question, the investigator team revised the item, simplifying it to read, “Composing research questions that address meaningful issues to health systems.”

Correlation results are shown in Table 4. The strength of association between ratings of Knowledge and Ability was strong to very strong for all items (all rho >0.57). For 84% (46/55) of the items, the Spearman's Correlation Coefficient was >0.7. Given the high correlations between Knowledge and Ability ratings and the concerns about survey response burden, the team decided to remove the Ability ratings from the final survey instrument. A copy of the final survey is provided in Appendix S3.

TABLE 4

Correlations between knowledge and ability for each survey item

Item number	Domains
	Systems Science	Research Questions	Research Methods	Informatics	Ethics	Improvement	Engagement
1	0.77	0.57	0.85	0.64	0.89	0.84	0.70
2	0.77	0.65	0.81	0.66	0.70	0.75	0.74
3	0.80	0.88	0.74	0.80	0.72	0.78	0.66
4	0.71	0.81	0.83	0.74	0.84	0.81	0.75
5	0.86	0.82	0.69	0.80	0.88	0.77	0.81
6	0.76	0.83	0.66	0.83	0.89	0.63	0.88
7	0.81		0.74		0.92		0.86
8	0.81				0.95		0.79
9	0.91						0.67
10	0.93						0.83
11							0.83
12							0.85

DISCUSSION

We developed a survey for rehabilitation researchers to assess their knowledge of LHS research competencies and their interest in learning more about these competencies. The content of this survey was guided by the LHS research domains and core competencies identified by an AHRQ expert panel. To our knowledge this is the first survey to assess rehabilitation researchers' knowledge of LHS research competencies as well as interest in learning more about these areas. This survey may also be useful for rehabilitation educators and program directors in developing LHS research content and articulating more specific LHS research competencies, ultimately supporting efforts to promote the growth of rehabilitation‐focused LHS researchers.

Our survey is distinctly different from prior work that developed an appraisal inventory to help direct LHS scholar's individual development plans and which utilized the core competency statements verbatim. We revised some language in the original core competencies and wrote separate survey items for those core competencies that contained multiple components. Feedback from a multidisciplinary expert panel in rehabilitation was used to enhance clarity of language, and relevance for the rehabilitation research community. Items that were not deemed relevant for rehabilitation research were dropped from the survey, while the word “rehabilitation” was added to six items. Thus, this survey provides a novel and innovative active strategy for understanding the needs of the rehabilitation research community and provides LeaRRn with the necessary information to target training and resources to those needs.

We believe that our survey approach is transferable to other disciplines with minor revisions. In adapting the AHRQ core competencies for our survey, we included little discipline‐specific language, to make items applicable to multiple disciplines. Our process of item revision and tailoring is carefully detailed. Items dropped because they were not deemed relevant for rehabilitation, can be easily identified and considered for relevance by other disciplines (Table 1). Similar processes can be used to customize surveys for researchers from other fields.

Limitations

We used our expert panel members to provide feedback on clarity, importance and priority of each item; however, we did not conduct formal cognitive testing of items. Feedback from panelists was used to iteratively refine items and to help in identifying items with lower importance and priority for rehabilitation. Although our expert panel was nominated by professional associations and all were interested in research within health systems, these members were not necessarily content experts in LHS research nor were they sampled from our target population, rehabilitation researchers developing new LHS knowledge and skills. It is possible that the expert panel members may have had better comprehension or different interpretations of survey items as compared to the target audience.

CONCLUSION

This manuscript reports on a unique, new survey of knowledge of and interest in learning more about LHS research core competencies specifically targeted to rehabilitation researchers. Survey content was derived from the AHRQ core‐competencies for LHS researchers, but modified to reduce items with multi‐barreled content and to eliminate those deemed not relevant for rehabilitation research. Content validity was evaluated by an expert panel of rehabilitation researchers, and refinements made after CVI analysis, pre‐pilot testing, and pilot testing. The resulting survey is appropriate for researchers across the rehabilitation community, and may be useful for educators, programs directors and other initiatives aimed at assessing and improving LHS research competencies. The survey may also be adapted for use in inter‐professional training of research teams.

CONFLICT OF INTEREST

Coauthors of this manuscript are investigators funded through NICHD 1 P2C HD101895‐01 and this work was conducted as part of this award.

Appendix S1. Supporting Information.

Click here for additional data file.

Appendix S2. Supporting Information.

Click here for additional data file.

Appendix S3. Supporting Information.

Click here for additional data file.