Interexaminer Agreement and Reliability of an Internationally Endorsed Screening Framework for Cervical Vascular Risks Following Manual Therapy and Exercise: The Go4Safe Project.

OBJECTIVE: Clinicians are recommended to use the clinical reasoning framework developed by the International Federation of Orthopaedic Manipulative Physical Therapists (IFOMPT) to provide guidance regarding assessment of the cervical spine and potential for cervical artery dysfunction prior to manual therapy and exercise. However, the interexaminer agreement and reliability of this framework is unknown. This study aimed to estimate the interexaminer agreement and reliability of the IFOMPT framework among physical therapists in primary care.
METHODS: Ninety-six patients who consulted a physical therapist for neck pain or headache were included in the study. Each patient was tested independently by 2 physical therapists, from a group of 17 physical therapists (10 pairs) across The Netherlands. Patients and examiners were blinded to the test results. The overall interexaminer agreement, specific agreement per risk category (high-, intermediate-, and low-risk), and interexaminer reliability (weighted κ) were calculated.
RESULTS: Overall agreement was 71% (specific agreement in high-risk category = 63%; specific agreement in intermediate-risk category = 38%; specific agreement in low-risk category = 84%). Overall reliability was moderate (weighted κ = 0.39; 95% CI = 0.21-0.57) and varied considerably between pairs of physical therapists (κ = 0.14-1.00).
CONCLUSION: The IFOMPT framework showed an insufficient interexaminer agreement and fair interexaminer reliability among physical therapists when screening the increased risks for vascular complications following manual therapy and exercise prior to treatment. IMPACT: The IFOMPT framework contributes to the safety of manual therapy and exercise. It is widely adopted in clinical practice and educational programs, but the measurement properties are unknown. This project describes the agreement and reliability of the IFOMPT framework.

Introduction

Neck pain and headache are 2 conditions commonly treated by physical therapists. Systematic reviews have shown small to moderate levels of evidence for the effectiveness of cervical manual therapy and exercise for patients with neck pain or headache., However, because of associated serious adverse events, such as cerebral ischemia and even death, there is discussion about the safety of manual therapy for the cervical spine. Potential hypotheses for serious adverse events are alterations of hemodynamics in the cervical arteries, arterial dissection, or atherosclerosis,,, and missed preexisting vascular pathologies that mimic musculoskeletal neck pain or headache,, including arterial dissection and atherosclerosis. The prevalence of vertebral artery dissection in the general population is 0.75–2.90 per 100,000 people, and the prevalence of vascular dissections associated with manual therapy is estimated between 0.40 per 100,000 to 5.00 per 100,000. There are no data on the prevalence of complications following exercise.,

The International Federation of Orthopaedic Manipulative Physical Therapists (IFOMPT) developed a framework based on the best available evidence to provide guidance regarding assessment of the cervical spine and potential of cervical artery dysfunction prior to manual therapy, with special attention to interventions at end-range positions of the cervical spine. The multivariable character of this clinical reasoning framework is in accordance with physical therapist practice in which diagnostic conclusions are based on multiple rather than single tests. The IFOMPT framework is intended to improve early identification of patients with neck pain or headache with an increased risk of developing serious adverse events following cervical manual therapy or exercise.

Although the framework is widely adopted in clinical practice and educational programs, the reproducibility and accuracy of the IFOMPT framework have not yet been assessed. Agreement, specific agreement, and reliability provide an insight into reproducibility. Agreement concerns how well outcomes from different examiners agree. It is expressed in terms of observed agreement. The proportion of specific agreement distinguishes agreement on positive or negative scores. Reliability concerns how well patients can be distinguished from each other despite measurement errors and is expressed as a coefficient varying from 0 to 1.

At present, there is lack of information about the measurement properties of the IFOMPT framework. Consequently, this study aimed to estimate the interexaminer agreement and reliability of the risk estimation of vascular complications following manual therapy and exercise prior to treatment among physical therapists.

Methods

Design

We conducted an interexaminer agreement and reliability study, reported here according to Guidelines for Reporting Reliability and Agreement Studies. The Medical Ethics Review Committee of the Amsterdam University Medical Centre (Location VUmc) approved the study protocol (METC-2017.086).

Participants

Consecutive patients with neck pain and/or headache were recruited from primary care physical therapist practices in The Netherlands between July 2017 and March 2019. Patients were eligible if they were at least 18 years old, consulted a physical therapist for neck pain or headache, and had sufficient knowledge of the Dutch language. All patients signed informed consent prior to inclusion in the Go4Safe project.

Examiners

Thirty-four physical therapists were recruited via social media, the Internet, newsletters of the Royal Dutch Society of Physiotherapy, and physical therapy networks. Thirty-two of these physical therapists had a master’s degree in manual therapy with at least 2 years of relevant clinical experience and were eligible to participate. Characteristics of the physical therapists are summarized in Table 1. All physical therapists attended a 3-hour refresher course on how to conduct the clinical reasoning framework and how to interpret the risk of vascular complications.

Table 1

Characteristics of the Physical Therapists (N = 34)

Characteristics	Active PT (n = 17)	Nonactive PT (n = 17)	P
Sex (F), n (%)	5 (29%)	3 (18%)	.69b
Age, y, median (IQR)	36 (31–47)	35 (31–44)	.55c
Experience in physical therapy	12 (8–23)	12 (8–19)	.55c
Experience in manual therapy	7 (4–16)	5 (2–12)	.21c

Active PT = physical therapists who included patients in the study; nonactive PT = physical therapists who did not include patients in the study. F = female; IQR = interquartile range.

Significant level of difference between groups using Fisher exact.

Significant level of difference between groups using Wilcoxon rank sum test.

During the refresher course, the early and late clinical manifestations of cervical artery dysfunction, such as the description of pain as “unlike any other,” and the risk factors according to the IFOMPT framework were highlighted.

Interactive case studies and clinical reasoning tasks focused on the estimation of the probability of serious vascular pathology or vascular complications following manual therapy and exercise. Participants practiced the clinical tests for the cranial nerves, palpation of the carotid artery, and blood pressure measurements on each other. Blood pressure was measured according to the guidelines from the National Institute for Health and Care Excellence. The training was delivered by 2 experienced instructors of the Masters of Science program in Manual Therapy at SOMT University of Physiotherapy. The instructors had 6 and 14 years of teaching experience in manual therapy, and 10 and 37 years of relevant clinical experience.

To ensure blinding of the patients and examiners, no information was given about the outcome of the test. Seventeen physical therapists (53%) actually included patients in the study. These formed 10 separate pairs from 10 different physical therapist practices. There were no significant differences in age, sex, or experience in physical therapy or manual therapy between physical therapists who provided patients compared with those who did not provide patients for the study (Tab. 1).

Test Procedures

Eligible patients were examined by their treating physical therapist using the IFOMPT framework. Subsequently, a patient interview and, if necessary, clinical testing of the cervical blood vessels (ie, evaluation of blood pressure and palpation of the carotid artery) and/or cranial nerves was performed. Afterwards, each physical therapist estimated the risk of a vascular complication following manipulation, mobilization, and exercise therapy as “low,” “intermediate,” or “high.” The Figure summarizes the flow of the IFOMPT framework.

Figure

Flow of the IFOMPT (International Federation of Orthopaedic Manipulative Physical Therapists) framework.

When the physical therapist expected a vascular origin (eg, dissection, atherosclerosis) for the symptoms, or when multiple risk factors for vascular complications (eg, stroke) following manual therapy were present, a high-risk score was given. The risk estimation (“high risk,” “intermediate risk,” or “low risk”) was based on a thorough clinical reasoning process in which risk factors for vascular complications, the results of patient interview, and physical examination were weighted by the physical therapist. No cutoff points for “low,” “intermediate,” or “high” risk were set. The benefits were not considered. Within 1 week, a second physical therapist performed the examination in the same physical therapist practice. Patients and examiners of the second test did not receive information about the outcomes of the first test. In addition to the patient interview and clinical examination by the physical therapists involved in the study, patients received an online questionnaire from the research team to collect demographic data and clinical characteristics (eg, potential risk factors for vascular pathology, neck pain intensity, and additional symptoms). The physical therapists recorded all adverse events during or after the examination.

Sample Size

A sample size calculation was performed in R-Studio (Version 1.1.453; R-Studio, Boston, MA, USA). Using the “KappaSize” package, 79 participants were needed for this study. This was based on 2 examiners, 3 risk categories, a lower limit of the 95% CI of weighted κ = 0.60, and an anticipated prevalence of 0.10 in the high-risk category, 0.20 in the intermediate category, and 0.70 in the low-risk category. The estimated prevalences were based on a pilot study in primary care physical therapist practices.

Statistical Analysis

“Patients” and “physical therapists” characteristics were summarized using descriptive statistics. Data were checked for normality using visual inspection of a histogram, a boxplot, and tested with the Shapiro-Wilk test. Means and SDs were described if numerical data were normally distributed, and median and interquartile range when not normally distributed. In case of categorical data, absolute and relative frequencies were presented.

Three-by-three tables were constructed in which the results of the paired examiners were plotted against each other. To express agreement, we calculated percentage overall agreement and specific agreement. Specific agreements were calculated for each risk category separately. To calculate reliability, we used linear weighted κ.

A combination of these coefficients provides a more detailed impression of the degree of agreement and reliability. Agreement and reliability values were calculated for the total group and per examiner pair (2 physical therapists from the same or adjacent physical therapist practice). A specific agreement of at least 75% was considered acceptable. For the interpretation of the linear weighted κ, the following criteria were used: almost perfect (0.81–1.00), substantial (0.61–0.80), moderate (0.41–0.60), fair (0.21–0.40), slight (0.00–0.20), and poor (<0.00). All data were analyzed with R software version 3.5.1 (CRAN.R-project).

Sensitivity Analysis

According to the IFOMPT framework, the consequences of a high-risk and intermediate-risk score assume no indication for manual therapy at that time. Specifically, the high-risk score implies “Avoid treatment,” and the intermediate risk implies “Avoid or delay treatment/monitor and reassess.” By merging these 2 categories together, the interobserver agreement and reliability of no indication versus indication for manual therapy is assessed, which is relevant for clinical practice.

Therefore, we performed a sensitivity analysis with 2 categories, “Indication for manual therapy” and “No indication for manual therapy,” in which the “high-risk” and “intermediate-risk” groups were combined. Two-by-two tables were constructed in which agreement, specific agreement, and reliability (Cohen κ) were calculated.

Role of the Funding Source

This study was funded by a research grant from the Dutch Association for Manual Therapy (Nederlandse Vereniging voor Manuele Therapie) and was supported by the MSG Science Network Physiotherapy (https://msg-sciencenetwerk.nl/). The funders played no role in the design, conduct, or reporting of this study.

Results

A total of 96 patients were included. The median number of patients included per examiner pair was 6 (interquartile range: 4–8). Median age of patients was 53 years (interquartile range: 40–62), and 62 (65%) were female. Sixty-two patients (72%) had neck pain or headache for more than 12 weeks. Characteristics of the patients are summarized in Table 2. Seventy-five percent of the patients received the retest within 7 days. Median time frame between test and retest was 0 days (interquartile range: 0–8), reflecting that more than half of the patients were tested twice on the same day.

Table 2

Self-Reported Characteristics of the Participants (N = 96)

Characteristics	Participants
Sex (F), n (%)	62 (65%)
Age, y, median (IQR)	53 (40–62)
BMI, kg/m2, mean (SD)	26.4 (4.9)
Distribution of symptoms, n (%)
Occipital headache	1 (1)
Headache around ear and jaw	2 (2)
Mid/upper cervical spine	13 (14)
Lower cervical spine	9 (9)
Mid/upper cervical spine and occipital headache	16 (17)
Mid/upper cervical spine and headache around ear and jaw	1 (1)
Other (eg, lower cervical spine and arm pain)	54 (56)
Pain intensity, NRS, median (IQR)
Headache	5 (3–7)
Neck pain	5 (3–6)
Cause, n (%)
Trauma within 30 db	15 (16)
Duration of symptoms,c n (%)
< 6 wk	12 (14)
6–12 wk	12 (14)
> 12 wk	62 (72)
Additional symptoms,d n (%)
Dizzinesse	34 (38)
Loss of sensibilityf	31 (39)
Muscle weaknessg	19 (24)
Possible cranial nerve dysfunction
Dysphagia	12 (15)
Aphasia	7 (9)
Diplopia	11 (14)
Drop attacks	17 (22)
Other risk factors present, n (%)
High blood pressureb	26 (28)
High cholesterolh	24 (27)
History of cardiac arresth	6 (7)
History of CVAe	6 (7)
Smokingi	16 (18)
Alcohol use (>1 unit/d)c	7 (8)
Recent infectionc	8 (9)
Migraine without auraj	9 (10)
Recent cervical manipulatione	50 (56)

BMI = body mass index; CVA = cerebrovascular accident; F = female; IQR = interquartile range; NRS = numeric rating scale in the past week; recent infection = recent infection in the past 30 days.

4 missing values.

8 missing values.

17 missing values.

7 missing values.

Loss of sensibility in arms or legs.

Muscle weakness in arms or legs.

6 missing values.

5 missing values.

9 missing values.

The overall distribution of the results of the first examiner and the second examiner is presented in Supplementary Appendix A. The prevalence of a high-risk category classification for the first examiner was 17% (16/96) and 26% (25/96) for the second examiner.

The overall agreement of the IFOMPT framework was 71% and ranged from 46% to 100% for the different examiner pairs. Overall, specific agreement in the high-risk category was 63%, ranging from 0% to 100%; specific agreement in the intermediate-risk category was 38%, ranging from 17% to 100%; and specific agreement in the low-risk category was 84%, ranging from 57% to 100%. Overall reliability between the examiners was fair (weighted κ = 0.39; 95% CI = 0.21–0.57). The interexaminer agreement and reliability values are summarized in Table 3.

Table 3

Agreement and Reliability of IFOMPT Framework

Examiners	OA	SAH	SAI	SAL	Weighted κ (95% CI)
Overall (N = 96)	71%	63%	38%	84%	0.39 (0.21–0.57)
Pair A (n = 50)	78%	78%	17%	89%	0.47 (0.23–0.72)
Pair B (n = 11)	46%	ND	29%	80%	0.08 (0.00–0.30)
Pair C (n = 8)	50%	50%	40%	57%	0.26 (0.34–0.86)
Pair D (n = 6)	67%	67%	ND	75%	0.40 (0.31–1.00)
Pair E (n = 6)	67%	ND	50%	75%	0.25 (0.00–1.00)
Pair F (n = 5)	100%	0%	100%	100%	1.00 (1.00–1.00)
Pair G (n = 4)	75%	100%	ND	80%	0.60 (0.31–1.00)
Pair H (n = 4)	50%	50%	ND	67%	0.14 (0.00–0.73)
Pair I (n = 1)	100%	ND	100%	ND	ND
Pair J (n = 1)	100%	ND	ND	100%	ND

IFOMPT = International Federation of Orthopaedic Manipulative Physical Therapists; ND = no data; OA = overall agreement; SAH = specific agreement of high-risk category; SAI = specific agreement of intermediate-risk category; SAL = specific agreement of low-risk category.

To address the influence of patients with a time frame of more than 7 days between test and retest on the agreement and reliability results, we performed a post hoc sensitivity analysis without this group. The results changed to: overall agreement 68%, specific agreement in the high-risk category 52%, specific agreement in the intermediate-risk category 39%, and specific agreement in the low-risk category 84%. Overall reliability between the examiners was still fair (weighted κ = 0.31; 95% CI = 0.10–0.53).

By combining the intermediate-risk group with the high-risk group, the prevalence of a high-risk classification increased from 0.17 to 0.36 according to Examiner 1, and from 0.26 to 0.45 according to Examiner 2. The distribution of the sensitivity analysis is presented in Supplementary Appendix B. The overall agreement, specific agreement, and κ values also increased. The κ value increased from 0.39 (95% CI = 0.21–0.57) to 0.60 (95% CI = 0.45–0.77). The interexaminer agreement and reliability values of the sensitivity analysis are summarized in Table 4.

Table 4

Agreement and Reliability of IFOMPT Framework (2 Categories)

Examiners	OA	SPA	SNA	κ (95% CI)
Overall (N = 96)	81%	77%	84%	0.60 (0.45–0.77)
Pair A (n = 50)	86%	80%	89%	0.70 (0.49–0.90)
Pair B (n = 11)	82%	83%	80%	0.63 (0.17–1.00)
Pair C (n = 8)	63%	67%	40%	0.25 (0.42–0.92)
Pair D (n = 6)	67%	50%	75%	0.25 (0.6–1.00)
Pair E (n = 6)	67%	50%	75%	0.25 (0.6–1.00)
Pair F (n = 5)	100%	100%	100%	1.00 (ND)
Pair G (n = 4)	75%	80%	67%	0.50 (0.35–1.00)
Pair H (n = 4)	75%	67%	80%	0.50 (0.35–1.00)
Pair I (n = 1)	100%	100%	ND	ND
Pair J (n = 1)	100%	ND	100%	ND

IFOMPT = International Federation of Orthopaedic Manipulative Physical Therapists; ND = no data; OA = overall agreement; SNA = negative specific agreement; SPA = positive specific agreement.

Post hoc Sensitivity Analysis

The main results showed a conspicuously large number of participants in Pair A (n = 50). To estimate the influence of Pair A, we performed a post hoc sensitivity analysis without Pair A. The distribution of the results is presented in Supplementary Appendix C. The agreement and reliability values are presented Table 5. The results revealed agreement and reliability values comparable to the main results, with exception of the specific agreement in the high-risk category. The specific agreement in this category decreased from 63% to 44%.

Table 5

Agreement and Reliability of IFOMPT Framework (Without Pair A)

	OA	SAH	SAI	SAL	Weighted κ (95%CI)
Overall (N = 46)	63%	44%	48%	78%	0.33 (0.10–0.56)

IFOMPT = International Federation of Orthopaedic Manipulative Physical Therapists; OA = overall agreement; SAH = specific agreement of high-risk category; SAI = specific agreement of intermediate category; SAL = specific agreement of low risk category.

Adverse Events

No adverse events occurred during the first or second test.

Discussion

This study revealed insufficient agreement and fair reliability of the IFOMPT framework among physical therapists in patients with neck pain or headache who attended primary care. Weighted κ and its 95% CI imply that the IFOMPT framework has a fair to moderate reliability among physical therapists. However, it is important to note that 1 examination pair included 50 patients, which might have influenced the results. Post hoc sensitivity analyses revealed only a small decrease in agreement and reliability, but still insufficient agreement and fair reliability (with boundaries of the CI ranging from slight to moderate).

The sensitivity analyses with 2 categories, “No indication for MT [manual therapy]” and “Indication for MT,” showed an increase of agreement and reliability. Values of κ increased from fair (0.39; 95% CI = 0.21–0.57) to moderate (0.60; 95% CI = 0.45–0.77), and specific agreement improved to an acceptable level (63%–77%). This implies that dichotomization of the outcome categories improves the agreement and reliability of the IFOMPT framework substantially.

To the best of our knowledge, no other studies have assessed the interexaminer agreement or reliability of the IFOMPT framework or other cervical vascular tests. The only information available concerns agreement and is about separate tests in the IFOMPT framework, such as the Sharp-Purser test and premanipulative hold. The Sharp-Purser test is a test for cervical instability, which demonstrates an intraexaminer agreement ranging from 55% to 83%. The premanipulative hold is a sustained hold of the manipulation position for at least 10 seconds and released for 10 seconds. The reliability of this test is considered “fair” to “good.” However, this needs to be interpreted with caution because of the large variability and wide 95% CIs. None of these studies reported specific agreement values. Other studies of tests advised in the framework merely reported about validity. The physical therapists in this study established the risk of vascular complications following manual therapy and exercise based on the information derived from the patient interview and physical assessment according to the IFOMPT framework. Patients were considered as “high risk” when signs and symptoms suggestive of vascular pathology (eg, dissection or atherosclerosis) were present, or when risk factors for vascular complications were present. The estimated risk of complications following manual therapy and exercise in this study was much higher (~22%) than the reported prevalence of vascular pathology (ie, vascular pathology related to manual therapy) in the literature.,, Moreover, not all “high-risk” patients actually have a vascular pathology or will experience a vascular complication after manual therapy or exercise.

The sample size was calculated with prevalence data for “high,” “intermediate,” and “low” risks derived from a pilot study in primary care physical therapist practices. Post hoc, we recalculated the sample size with the actual parameters and needed a minimum number of 55 patients. Therefore, our sample size (N = 96) was larger than required.

Although we aimed to include consecutive patients, the examiners did not register all eligible patients and reasons why patients refused to participate. Because of the high prevalence in the high-risk category, we assumed that patients who were at risk of vascular complications following manual therapy or exercise were more willing to participate because they received an extra screening by a second independent physical therapist. This implies that our study could contain selection bias, affecting the generalizability of our findings to clinical care. This bias could have overestimated the overall agreement and specific positive agreement values and might have resulted in an increased reliability. The relatively high number of examiner pairs in our study could have negatively affected the agreement values, while also positively influencing the generalizability of our findings in clinical care.

The time interval between the first examination and second examination was set at a maximum of 1 week. However, this could not be met for 23%, and the possibility that these patients received treatment between the 2 examinations cannot be ruled out, nor that this influenced their risk profile. To address the influence of patients with a time frame of more than 7 days between test and retest on the agreement and reliability results, we performed a post hoc sensitivity analysis without this group. This analysis showed results comparable to the overall results. Therefore, we conclude that patients with a time frame of more than 7 days between tests did not bias our results and that our participants could be considered stable.

The IFOMPT framework proposes a risk-benefit analysis. In a risk-benefit analysis, the risk component is the first and dominant component for safety purposes. The physical therapists in our study were therefore asked only to consider the risk of serious adverse events developing following cervical manual therapy or exercise.

In conclusion, this study found that among physical therapists the IFOMPT framework has an insufficient agreement and only fair reliability in patients with neck pain and/or headache attending primary care when scored into the original 3 categories. However, agreement becomes acceptable, and reliability becomes moderate when “indication for manual therapy” versus “no indication for manual therapy” are considered. Further evidence about the diagnostic accuracy of the risk estimation is necessary to confirm usage in clinical practice. Future research should therefore focus on the diagnostic accuracy of the IFOMPT framework.

Click here for additional data file.

Click here for additional data file.

Click here for additional data file.