A delphi procedure on rehabilitation outcome for patients with moderate to severe traumatic brain injury; first phase of the Neurotraumatology Quality Registry (NET-QURE).

OBJECTIVE: To select a set of rehabilitation outcome instruments for a national Neurotrauma Quality Registry (Net-QuRe) among professionals involved in the care of patients with traumatic brain injury.
DESIGN: A 3-round online Delphi procedure.
SUBJECTS: Eighty professionals from multiple disciplines working in 1 of the 8 participating rehabilitation centres were invited to participate. The response rate varied from 70% to 76% per round.
METHODS: For the Delphi procedure, multiple outcome categories were defined based on the International Classification of Functioning, Disability and Health (ICF) with concomitant measurement instruments. For each category we strived for consensus on one instrument of at least 75%.
RESULTS: After the first round, consensus was reached for the category subjective cognitive functioning. After the second round for quality of life, pain, general functioning, anxiety and depression, general psychological functioning, communication (impairment), and personal factors. Finally, after the third round, consensus was reached for activities of daily living, participation, self-awareness, and aphasia. No consensus was reached for the categories motor function, cognitive function, comorbidity, fatigue, and employment status.
CONCLUSION: Consensus was reached in 12 out of 17 outcome categories. A Delphi procedure seems to be a feasible method to collectively select measurement instruments for a multicentre study.

Worldwide, 13 million people sustain moderate to severe traumatic brain injury (TBI) annually. Most patients will have life-long disabilities after surviving moderate to severe traumatic brain injury (1). These disabilities vary in nature and severity. Domains that are often affected are cognition, self-awareness, motor function, language and communication. Moreover, patients may encounter problems in activities of daily life, may have fatigue, depression, anxiety or pain, which in turn may influence quality of life, social participation, or general and psychological functioning (2–4). Due to the variety of disabilities, care pathways for patients with TBI differ from one patient to another. To gain insight into the chain of care and outcomes on several domains after moderate to severe TBI, the Neurotraumatology Quality Registry (Net-QuRe) was initiated, in which patients with moderate to severe TBI are followed from hospital admission through the healthcare chain up to 2 years post-injury as an add-on study of Center-TBI (5). The aim of Net-QuRe is to evaluate treatment strategies and to optimize the quality of the total healthcare chain for patients with moderate to severe TBI. In contrast to the USA, in the Netherlands a national initiative like The Traumatic Brain Injury Model Systems (TBIMS) (6), whereby patients with TBI are followed in a standardized way to examine outcome and the course of recovery does not yet exist. Centre-specific protocols targeting outcome measurement are scarce and differ among Dutch rehabilitation centres.

To measure outcome in Net-QuRe, suitable measurement instruments needed to be identified from the numerous instruments available from the literature (7–9). For example, in systematic reviews, Tate et al. (2013) identified 728 behavioural assessment instruments for patients with TBI (7). Van Heugten et al. (2019) identified 381 neuropsychological instruments (8), and Laxe et al. (2012) identified 283 unique instruments in 193 papers on TBI (9). These instruments can be categorized according to the International Classification of Functioning, Disability and Health (ICF) (9, 10). The ICF provides a standard framework for the description of functioning and disability for health conditions. Its components are classified in body structures and functions, activities, participation, personal, and external factors.

Several initiatives have focused on defining core sets for TBI research. In 2010, the TBI Outcomes Working group of the National Institute of Neurological Disorders and Stroke (NINDS) selected a common set of outcome measures, consisting of a core, a supplemental, and an emerging part for TBI (11, 12). However, these core sets include instruments, such as the Functional Independence Measure, that are not common practice outside the USA. More recently, Honan et al. (2019) recommended 56 instruments out of 115 instruments in psychosocial research targeting moderate to severe TBI (13). Together these articles provided a complete overview of existing outcome measures. Selecting the best set from this abundance of outcome measures is a challenge.

To identify a set of outcome measures on ICF categories for patients with moderate to severe TBI to be used in Net-QuRe a Delphi procedure was performed, an iterative process based on expert opinion suitable to reach consensus (14). It is a flexible procedure concerning the number of rounds, consensus level and the number of participants (15, 16). By using expert opinion, experience of professionals, and existing knowledge for selecting a customized set of outcome measures we intended to create support for using the set of measurement instruments in both clinical care and research, and to minimize the chance of missing data in Net-QuRe. Choices made concerning the Delphi procedure are explained and questioned in this article.

METHODS

The ethics procedure for this study was waived by the medical ethics committee of Erasmus MC, University Medical Center Rotterdam (MEC-2015-395).

Design

Between 10 July and 30 September 2015 an online Delphi procedure was performed among experts from 8 rehabilitation centres in the Netherlands to select outcome measures for Net-QuRe (Netherlands Trial Register NL5761). In Net-QuRe detailed information concerning treatment and clinical data from the acute phase are collected according to Center-TBI. This Delphi procedure will focus solely on follow-up and rehabilitation outcome. Guidelines from the literature were used to make choices concerning the number of rounds, consensus level, and the number of participants (14, 16, 17). In this Delphi procedure questionnaires are completed anonymously, answers from each round are presented in the next round, and participants review their initial response based on the group response of the previous round (14). The procedure consisted of 3 Delphi rounds among professionals with expertise in treatment of patients with TBI. The procedure ended after 3 rounds or when a consensus level of 75% was reached. First-choice instrument references are given in Appendix I. The results from the Delphi procedure were used by the research committee to decide on the final outcome set for Net-QuRe.

Participants

Eight members of the Dutch Special Interest Group on TBI, which represents all Dutch rehabilitation centres, were asked to provide a list of 2 physiatrists, 2 psychologists, 2 physiotherapists, 2 occupational therapists, 1 or 2 speech and language therapists, 1 or 2 social workers, 1 or 2 nurse/nurse practitioners and, if available, an expert with specific knowledge of 1 of the domains or knowledge of TBI. Two elderly care physicians working in skilled nursing homes and not being a member of the Special Interest Group were invited to participate because of their knowledge of acquired brain injury and outcome measures. All experts were instructed about the Delphi procedure via an information letter.

Development of the online Delphi questionnaires

Based on the ICF model, interviews with physiatrists, and expert opinion, the following outcome categories were selected for the Delphi procedure: objective cognitive functioning, subjective cognitive functioning, aphasia, (cognitive) communication, self-awareness, general psychological functioning, anxiety and depression, pain, fatigue, physical functioning, activities and participation, activities of daily living, employment status, personal factors, general functioning, quality of life, and comorbidity.

Subsequently, a literature search was performed for each category to identify potential measurement instruments for patients with moderate to severe TBI. For this study, afore-mentioned review studies and core sets were used as a starting point for our overview of available instruments (7, 9, 11, 12). From there, extra instruments were added by screening references lists of relevant articles, and by checking at a website containing over 400 measurement instruments used in healthcare (18). All measurement instruments for which a translation into the Dutch language was available were included in the first round of the Delphi procedure. In the first round, instruments that were not listed could be added to the list by the experts.

Questionnaires were developed using an online survey tool named Limesurvey (15). With this tool it is possible to keep track of responses without showing the answers of a participant. Thus anonymity was guaranteed. Participants were invited by email and could enter the online questionnaire by using a personal token received by email. This token was sent out automatically by the system. For an optimal response, reminders were sent by the system during the procedure. In Appendix II examples of questions asked and feedback given to participants is provided.

Delphi rounds

Each round, participants answered questions about their profession, institution, and the years of experience they had with patients with TBI. Experts could participate in each round irrespective of their contribution to the previous round.

Round 1

For each Delphi category a list with potential measurement instruments was presented. Participants were asked if they were familiar with the instruments. In addition, extra instruments could be added by participants if they were missing in their opinion. By doing so, the list of outcome measures became complete. The main task for each participant for each Delphi category was to select a first- and second-choice instrument for each Delphi category, with the instruction to take into account psychometric quality, sensitivity to measure change, time efficiency, and suitability for patients with moderate to severe TBI. Participants could select instruments from the list or from the new instruments they added to the list. One of the standard answering options was “I am not the right person to answer questions regarding this category”. In this case, further questions regarding the concerning Delphi category were skipped automatically. Finally, there was the option to post a comment.

The procedure for cognitive and physical functioning was different from the other Delphi categories. In addition to a screening test, it is almost impossible to select 1 instrument that measures cognitive functioning extensively. The same holds for motor functioning. Therefore, instead of selecting a first- and second-choice cognitive- and physical functioning instrument, participants were asked to select a combination of multiple instruments. The physical functioning combination had to contain 1–4 physical functioning instruments. For the cognitive functioning category, participants were asked which test battery for people with TBI, with a maximum duration of 30 min, they would select in addition to the Montreal Cognitive Assessment (MoCA) (19), which is a general screening instrument.

Round 2

A detailed table with the results of the first round was given at the start of the second round for each Delphi category. If consensus was reached after the first round, this particular category was excluded from the second round. Similar to the first round, participants were asked to select a first- and secondchoice instrument. This time the list with instruments was updated based on the results from the first round. In the second round it was not possible to add instruments anymore (14). For the Delphi categories of cognitive- and physical functioning, all combinations given in the first round, formed the list of combinations from which 2 preferred combinations could be chosen in the second round. Again, it was not possible to add new combinations at this stage.

Round 3

In this round participants were asked to select only their first-choice instrument. Furthermore, the third round was comparable to the second round. Categories upon which consensus was reached after the second round were excluded. Instruments that were not chosen by at least 1 expert as first- or second-choice instrument in the first or second round were excluded from the list for the third round.

Statistical analysis

To summarize responses and to characterize the group of experts, descriptive statistics were performed using the Statistical Package for the Social Sciences (SPSS) statistics, version 23.

RESULTS

One representative of each of the 8 rehabilitation centres, provided a list of 12–15 professionals from different disciplines to be invited for the Delphi procedure. Eighty experts were invited, of whom 72 responded in at least 1 round (overall response rate 90%) and 40 experts (50%) participated in all 3 rounds. In the first round, 76% (n = 61) of the experts participated, 70% (n = 56) in the second round, and 74% (n = 9) in the third round. The mean number of years of experience in the field of acquired brain injury was 14.4 (SD 7.4) years. Details about the professional background of participants are shown in Table I.

Table I

Number of participants per discipline and per round

Discipline	Invited	Round 1	Round 2	Round 3
	n (%)	n (%)	n (%)	n (%)
Physiatrist	15	11 (18.0)	10 (17.9)	7 (11.9)
Psychologist	13	7 (11.5)	8 (14.3)	8 (13.6)
Physiotherapist	12	10 (16.4)	9 (16.1)	11 (18.6)
Occupational therapist	12	11 (18.0)	10 (17.9)	11 (18.6)
Speech and language therapist	9	7 (11.5)	7 (12.5)	8 (13.6)
Nurse/nurse practitioner	8	5 (8.2)	3 (5.4)	5 (8.5)
Social worker	6	5 (8.2)	4 (7.1)	5 (8.5)
Cognitive trainer	1	1 (1.6)	1 (1.8)	1 (1.7)
Researcher	1	1 (1.6)	1 (1.8)	1 (1.7)
Reintegration coordinator	1	1 (1.6)	1 (1.8)	0 (0.0)
Elderly care physician	2	2 (3.3)	2 (3.6)	2 (3.4)
Total	80 (100)	61 (100)	56 (100)	59 (100)

First round

In total, 99 instruments were presented to the experts in the first round of the Delphi procedure, divided into 17 categories. Consensus was reached by 80% of participants on using the Cognitive Failure Questionnaire (CFQ) in the category subjective cognitive functioning in the first round. Three out of the 99 (3%) instruments were not familiar to any participant. Another 6 instruments were not selected as first- or second-choice and were also removed. Ninety of 99 (91%) original instruments were selected for the second round. In addition, 30 new instruments were added to the list and 13 unique combinations of tests were created for the category cognitive functioning and 21 combinations for physical functioning. Seventeen physical functioning combinations contained the Berg Balance Scale. Further details about the instrument selection procedure are shown in Table II.

Table II

Instrument selection procedure in first, second and third round

Instruments
Category	R1, n	R2, n old/new/total	R3, n
General functioning	9	9/1/10	n.a.
General psychological functioning	7	5/2/7	n.a.
Fatigue	5	5/3/8	7
Anxiety & depression	6	5/0/5	n.a.
Physical functioning	16	16/21c/21c	12c
Aphasia	6	6/4/10	7
Communication (impairment)	1	1/4/5	n.a.
Cognitive functioning	0	0/13c/13c	7
Subjective cognitive functioning	2	n.a.	n.a.
Disease awareness	2	2/3/5	5
Pain	7	7/2b/9	n.a.
Comorbidity	2	2/1/3	2
Activities of daily living	7	7/4/11	11
Participation	11	11/2/13	9
Employment status	2	2/3/5	4
Personal factors	7	5/0/5	n.a.
Quality of life	9	7/1a/8	n.a.
Total	99	90/64/138	64

Targeting quality of life in persons with dementia in nursing home.

Targeting patients with mental disorders.

Combinations; n.a.: not applicable, consensus was reached in previous round; R: round.

Second round

In the second round, consensus was reached for 7 categories: for quality of life the Quality of Life after Brain Injury scale (QOLIBRI) was selected as first-choice instrument, for pain the visual analogue scale (VAS), for general functioning the Utrecht Scale for the Evaluation of clinical Rehabilitation (USER), for anxiety and depression the Hospital Anxiety and Depression Scale (HADS), for general psychological functioning the Symptom Checklist-90 (SCL-90), for communication (impairment) the Screening Test for Cognition and Communication (STCC), and for personal factors the Utrecht Coping List (UCL). Consensus levels reached are shown in Table III. From the 89 instruments/combinations that were presented in the remaining categories, 25 were not selected for the third round and were removed from the list. From the 21 test combinations for physical functioning 12 test combinations were selected for the third round. In addition, 7 out of 13 cognitive functioning test batteries were chosen for the next round.

Table III

Number of respondents per Delphi round and consensus level reached

Category	1st Round, (n)	2nd Round, (n)	3rd Round, (n)	Round consensus reached	Consensus level % (n)	1st choice instrument
Subjective cognitive functioning	15	n.a.	n.a.	1	80.0 (12)	CFQ
Quality of life	21	22	n.a.	2	86.4 (19)	QOLIBRI
Pain	29	30	n.a.	2	80.0 (24)	VAS
General functioning	29	29	n.a.	2	79.3 (23)	USER
Anxiety & depression	21	18	n.a.	2	94.4 (17)	HADS
General psychological functioning	11	13	n.a.	2	76.9 (10)	SCL-90
Communication (impairment)	8	13	n.a.	2	76.9 (10)	STCC
Personal factors	12	17	n.a.	2	76.5 (13)	UCL
ADL	28	33	36	3	77.8 (28)	BI
Participation	28	31	34	3	76.5 (26)	USER-P
Disease-awareness	13	18	19	3	84.2 (16)	AQ
Aphasia	15	12	10	3	90.0 (9)	ScreeLing
Physical functioning	22	30	27	n.a.	44.4 (12)	BBS, FAC, TCT,BFM
Cognitive functioning	17	17	27	n.a.	40.7 (11)/40.7 (11)	ACLS/Combi A
Comorbidity	4	9	11	n.a.	72.7 (8)	CIRS
Fatigue	37	33	35	n.a.	57.1 (20)	VAS
Employment status	5	9	11	n.a.	45.5 (5)/45.5 (5)	ERS/FML

CFQ: Cognitive Failures Questionnaire; QOLIBRI: Quality of Life after Brain Injury; VAS: visual analogue scale; USER: Utrecht Scale for the Evaluation of Clinical Rehabilitation; HADS: Hospital Anxiety and Depression Scale; SCL-90: Symptom Checklist-90; STCC: Screening Test for Cognition and Communication; UCL: Utrecht Coping List; BI: Barthel Index; USER-P: Utrecht Scale for the Evaluation of Clinical Rehabilitation – Participation; AQ: Awareness Questionnaire; BBS: Berg Balance Scale; FAC: Functional Ambulation Categories; TCT: Trunk Control Test; BFM: Brunnstrom Fugl-Meyer assessment; Combi A: Stroop, TMT, 15WT, visuoconstruction, zoo map BADS; ACLS: Allen Cognitive Level Screening; TMT: Trail Making Test; 15WT: 15 Word Test; CIRS: Cumulative Illness Rating Scale; ERS: Employability Rating Scale; FML: Functionele mogelijkheden Lijst. First-choice instrument references are available in Appendix II.

Third round

In the third and final round, consensus was reached for 4 categories: for activities of daily living the Barthel Index was selected as first-choice instrument, for activities and participation the USER-Participation, for self-awareness the Awareness Questionnaire (AQ), and for aphasia the ScreeLing (Table III). No consensus was reached for physical functioning, cognitive functioning, comorbidity, fatigue and employment status. In total, consensus was reached for 12 out of 17 categories. All selected measurement instruments, are validated and available in the English literature, except the STCC, which is a Dutch validated instrument (see Appendix I for references). Although no consensus was reached for cognitive functioning, 2 options were preferred: 1) the Allen Cognitive Level Screening (ACLS) and 2) the combination of Stroop, Trail Making Test (TMT), 15 Word Test (15WT), Visuoconstruction and ZOO map (BADS). Both options were preferred by 40.7% of the participants. Equivalent outcomes were found for the category employment status. Both the Functionele Mogelijkheden Lijst (FML) and Employability Rating Scale (ERS) were preferred by 45.5% of the participants. For physical functioning, the combination of the Berg Balance Scale (BBS), the Functional Ambulation Categories (FAC), the Trunk Control Test (TCT), and the Brunnstrom Fugl-Meyer Assessment (FMA) was preferred by 44% of the participants. Although 72.7% is below the threshold of 75% consensus, the Cumulative Illness Rating Scale (CIRS) was the most preferred instrument to measure comorbidity. With a consensus level of 57.1% for the VAS scale, agreement was low for fatigue. In 91% of the categories, the first-choice instrument was also the most familiar instrument.

DISCUSSION

An online Delphi procedure was performed aimed at defining outcome measures for the national Neurotraumatology Quality Registry (Net-QuRe), which was initiated to evaluate treatment strategies and to optimize the quality of the total healthcare chain for patients with moderate to severe TBI, as an add-on study of Center-TBI (5). This study identified multiple preferred outcome measures for TBI assessment within several pre-defined outcome categories, selected by experts in the field. In this study consensus on outcome measures was reached in 12 of 17 categories.

Multiple outcome measures were listed in the Delphi questionnaire and further divided into outcome categories based on the ICF model (10). Choosing ICF levels was challenging. For example, the decision to select both employment status and participation as separate categories. In the ICF model employment status is part of the broader category participation, but for the current study it was of interest to focus in more detail on employment status. The fact that participants suggested “new” instruments that were already listed in another category could be a consequence of this decision. Outcome categories form the backbone of a Delphi procedure targeting outcome measures, and should therefore be selected with care. Not surprisingly, selecting outcome categories/domains instead of instruments is the main goal of Delphi studies performed by other researchers, for example targeting outcome domains in patients with cancer (20), eczema (21), psoriatic arthritis (22), and ankylosing spondylitis (23). The categories selected in other studies, partly cover the 17 categories selected for Net-QuRe (20–23). For 10 out of 12 categories, the first-choice instrument was also the most familiar instrument. It could be debated whether this is desirable. In case the most familiar measure is the most used because of its psychometric quality, there is no need to change. In all other cases, being aware of new instruments is of high importance. The use of a measurement instrument is predicted by its psychometric quality, practicality, and having positive attitudes regarding the added value of an instrument over clinical judgement. Of these, attitudes and practicality seem to be the strongest predictors (24). This Delphi procedure focused mainly on the preferences of clinical experts, who were instructed to take into account psychometric quality, sensitivity to measure change, time efficiency, and suitability for patients with moderate to severe TBI. Because using validated measurement instruments is the norm among clinicians and researchers working in the Netherlands, detailed psychometric information about the listed outcome measures was not provided to the participants, which is a limitation of this study.

The strength of a Delphi procedure is the fact that it is based on expert opinion, with the ultimate goal of reaching consensus as a group. By making the procedure anonymous the method tackles problems that are related to decision-making in committees of groups, whereby 1 person or a group of persons dominate the process (25). This online Delphi procedure is also a cost- and time-efficient method, because there is no need to travel for face-to-face meetings, and respondents may choose their own moment for completing the online questionnaires.

A Delphi procedure is a flexible approach, for which methodological choices must be made by the researchers. In a systematic review, 15 other studies were examined that used a Delphi procedure to determine outcome measures. Methodological variability was found across these studies (16). Professionals from different disciplines from 8 rehabilitation centres were invited to create a multidisciplinary respondent group of 80 experts. Instead of inviting participants to respond to a specific category based on their profession, professionals were allowed to decide for each category whether their knowledge was sufficient to respond or to go on to the next section. This procedure minimized the risk of excluding participants who acquired their knowledge by experience. Secondly, experts could participate in each round regardless of involvement in the previous round. Whether this influenced the consensus reached remains unclear, but it certainly resulted in a high response rate per round compared with the response rate in other Delphi studies targeting outcome measures, focusing on pulmonary arterial hypertension (44%) (26), induction of labour (54%) (27), scleroderma (49%) (28), and primary healthcare (33%) (29). Another explanation for the high response rate in the current study could be the 2-step invitation procedure. We first selected contact persons in the participating centres, who subsequently asked their colleagues if they were willing to participate. This approach was also found in 2 other Delphi studies targeting outcome measures for patients with head and neck cancer (30), and balance outcome measures in cerebellar ataxia (31), in which the response rates were comparable with this study (respectively, 71% and 87%). A limitation of this approach is that it could have resulted in selection bias, because respondents were not randomly selected. In this study the focus was on the first-choice instrument of each category; there-fore we did not combine the first- and second-choice instruments. The second-choice instrument was used to provide researchers and participants with additional information about their preferences.

A strength of an online Delphi procedure is that participants could save the results and continue later. However, this procedure could lead to some duplication during data-export. This was identified during the second round, having only a marginal influence on the response feedback of the first round. Nevertheless, this is a limitation of this procedure.

In conclusion, consensus was reached by experts on 12 out of 17 categories of measurement instruments. The Delphi procedure is a feasible method to select measurement instruments for a research project. It appears to be a valuable method on the borderline of qualitative and quantitative research. Although this study focuses on outcome measures that are preferred in the Netherlands, the methods and challenges described will inform other researchers who are planning to perform a Delphi procedure targeting outcome measures.

Feedback given to respondents in round 2 concerning the category “Activities and Participation”

Instrument	Number of times 1st choice	Number of times 2nd choice
Utrecht Scale for the Evaluation of clinical rehabilitation – participation (USER-P)	15 (51, 72%)	3 (10, 34%)
Impact on Participation and Autonomy Questionnaire (IPA)	2 (6, 90%)	4 (13, 79%)
Sickness Impact Profile-68 (SIP-68)	2 (6, 90%)	2 (6, 90%)
Model of Human Occupation Screening Tool (MOHOST)	1 (3, 45%)	2 (6, 90%)
World Health Organization Disability Assessment Schedule (WHODAS)	1 (3, 45%)	0
Community Integration Questionnaire (CIQ)	1 (3, 45%)	3 (10, 34%)
Utrecht Activities List (UAL)	1 (3, 45%)	2 (6, 90%)
Frenchay Activities Index (FAI)	1 (3, 45%)	4 (13, 79%)
Life Habits (Life-H)	0	1 (3, 45%)
Participation Scale (P-Scale)	0	1 (3, 45%)
London Handicap Scale	0	1 (3, 45%)
Other, please specify:
Canadian Occupational Performance Measure (COPM)	3 (10, 34%)	2 (6, 90%)
Own questionnaire/inventory list	1 (3, 45%)	0
Occupational Performance History Interview (OPHI)	0	1 (3, 45%)
No answer given	1 (3, 45%)	3 (10, 34%)
Total	29 (100%)	29 (100%)