Japanese Orthopaedic Association Cervical Myelopathy Evaluation Questionnaire: part 3. Determination of reliability.

BACKGROUND: The manner of measuring the outcome of cervical myelopathy must be patient-oriented and have sufficient reliability and validity. The current Japanese Orthopaedic Association (JOA) scoring system for cervical myelopathy is widely used but has not met this requirement. The first- and second-round surveys established 24 items for inclusion on a new questionnaire for cervical myelopathy. The purpose of this study (the third-round survey A) was to confirm the reproducibility of patient responses to the selected questions.
METHODS: A total of 201 patients with cervical myelopathy and with no change of symptoms between the two interviews were included. Each patient was interviewed twice using the same questionnaire at an interval of 4 weeks. The reliability of the questionnaire was evaluated by determining the extension of the weighted kappa coefficients.
RESULTS: The weighted kappa coefficient for each item was >0.4, confirming that the test-retest reliability was acceptable.
CONCLUSIONS: The newly developed JOA Cervical Myelopathy Evaluation Questionnaire was proven to have sufficient reliability.

Introduction

As described in Part 11 and Part 2,2 the Japanese Orthopaedic Association (JOA) decided to revise the assessment tools for cervical myelopathy and develop a new JOA Cervical Myelopathy Evaluation Questionnaire (JOACMEQ). The point of the revision was to make it: (1) more scientific from the standpoint of medical statistics, and (2) more patient-oriented. For the first- and second-round surveys, the committee created a new questionnaire of 77 items including SF-36 (the Medical Outcome Study Short-Form 36-Item Health Survey)3 to assess health-related quality of life (HRQOL) and 41 items to assess functioning of the cervical spinal cord and cervical spine. In these surveys, 53 items were excluded.1,2 The purpose of this study (the third-round survey A) was to confirm the reproducibility of the selected 24 questions written in Japanese (see Appendix). The questionnaire was self-administered, so interobserver reliability did not need to be confirmed. To examine the reliability of the questionnaire, only test-retest reliability was evaluated.

Materials and methods

Inclusion criteria were as follows: (1) Patient had to have cervical myelopathy secondary to cervical disc herniation, cervical spondylosis, or ossification of the posterior longitudinal ligament (OPLL); and (2) there was no change of symptoms between the two interviews, which was judged by attending surgeons. Exclusion criteria included (1) patient had other musculoskeletal disease requiring medical treatment; (2) patient had a mental disease and so could potentially furnish inappropriate answers; (3) patient was postoperative; 4) patient had participated in the former surveys.

We asked 460 Japanese board-certified spine surgeons to participate in the survey between January 2004 and June 2004. Patients were recruited from the outpatient clinic of each surgeon during the planned inquiry period. Self-assessment was used in this study. Each patient was interviewed using the same questionnaire twice at an interval of 4 weeks (±3 days). The attending surgeon filled out the patient’s information regarding the diagnosis and the presence or absence of concomitant diseases and judged the severity using a three-step rating scale (mild, moderate, severe). This study was approved by the Ethics Committee of the Japanese Society for Spine Surgery and Related Research, and informed consent was obtained from each patient.

The reliability of the questionnaire was evaluated by determining the extension of the weighted kappa coefficients. According to the most widely accepted criteria proposed by Landis and Koch, a kappa coefficient of ≥0.4 was judged reliable.4 The 95% confidence intervals were calculated for all reliability coefficients.

All statistics were calculated using SPSS software (version 12; SPSS, Chicago, IL, USA).

Results

Of 304 patients who participated in this survey, 103 were excluded owing to a change of symptoms between the two interviews or an inappropriate patient administration interval, leaving 201 available for the study (74 women, 127 men). The mean ± SD patient age was 63.8 ±12.8 years (Table 1). The diagnosis was cervical disc herniation in 29, cervical spondylotic myelopathy in 113, and OPLL/other in 59. Table 2 lists the severity of patient myelopathy evaluated by the current JOA scoring system and shows that the characteristics of the recruited patients were not specific. The distribution of answers for each item during the second interview is shown in Table 3, and neither ceiling nor floor effects were observed. The results for test-retest reliability are shown in Table 4. The weighted kappa coefficient for each item was >0.4, showing that the test-retest reliability of JOACMEQ was acceptable. Among the three groups with severity judged as mild, moderate, or severe, there was some difference in the weighted kappa coefficient for some items, but the difference was statistically not significant. The difference among the three diseases (cervical disc herniation, cervical spondylosis, OPLL) was also statistically not significant (data not shown).

Table 1

Baseline participant demographics (n = 201)

Parameter	Male	Female	Total
No. of patients	127	74	201
Age (years)
30s	7	4	11
40s	12	8	20
50s	26	13	39
60s	41	13	54
70s	32	27	59
80s	9	9	18
Average	63	65.2	63.8
SD	12.4	13.5	12.8
Minimum	30	34	30
Maximum	89	84	89
Median	64	69	66
Diagnosis
Cervical disc herniation	17	12	29
Cervical spondylosis	68	45	113
OPLL	39	16	55
Other	3	1	4

OPLL, ossification of the posterior longitudinal ligament

Table 2

Distribution of the severity evaluated by the current JOA scoring system

Score	No.
Motor function
Upper extremity
0	1
1	10
2	37
3	90
4	63
Lower extremity
0	1
0.5	0
1	21
1.5	22
2	40
2.5	20
3	40
4	57
Sensory function
Upper extremity
0	2
0.5	22
1	88
1.5	61
2	28
Trunk
0	1
0.5	1
1	14
1.5	26
2	159
Lower extremity
0	1
0.5	15
1	55
1.5	44
2	86
Bladder functiona
0	4
1	17
2	52
3	127

JOA, Japanese Orthopaedic Association

a Data defect: n = 1

Table 3

Distribution of answers for each item in the second interview

Item	1	2	3	4	5	No answer
C01	108	80	13
	53.70%	39.80%	6.50%
C02	155	41	5
	77.10%	20.40%	2.50%
C03	124	53	17	6		1
	61.70%	26.40%	8.50%	3.00%		0.50%
C04	105	61	23	10	2
	52.20%	30.30%	11.40%	5.00%	1.00%
C05	86	74	37			4
	42.80%	36.80%	18.40%			2.00%
C06	132	43	15	10		1
	65.70%	21.40%	7.50%	5.00%		0.50%
C07	65	103	33
	32.30%	51.20%	16.40%
C08	114	70	16			1
	56.70%	34.80%	8.00%			0.50%
C09	129	57	14			1
	64.20%	28.40%	7.00%			0.50%
C10	111	60	27			3
	55.20%	29.90%	13.40%			1.50%
C11	121	68	12
	60.20%	33.80%	6.00%
C12	138	52	11
	68.70%	25.90%	5.50%
C13	68	88	44			1
	33.80%	43.80%	21.90%			0.50%
QOL01	6	16	77	90	11	1
	3.00%	8.00%	38.30%	44.80%	5.50%	0.50%
QOL02	39	91	69			2
	19.40%	45.30%	34.30%			1.00%
QOL03	36	97	66			2
	17.90%	48.30%	32.80%			1.00%
QOL04	33	74	93			1
	16.40%	36.80%	46.30%			0.50%
QOL05	16	29	91	44	20	1
	8.00%	14.40%	45.30%	21.90%	10.00%	0.50%
QOL06	15	49	67	47	22	1
	7.50%	24.40%	33.30%	23.40%	10.90%	0.50%
QOL07	11	26	95	43	25	1
	5.50%	12.90%	47.30%	21.40%	12.40%	0.50%
QOL08	12	35	92	45	15	2
	6.00%	17.40%	45.80%	22.40%	7.50%	1.00%
QOL09	17	46	102	30	5	1
	8.50%	22.90%	50.70%	14.90%	2.50%	0.50%
QOL10	8	56	65	55	17
	4.00%	27.90%	32.30%	27.40%	8.50%
QOL11	18	60	74	38	11
	9.00%	29.90%	36.80%	18.90%	5.50%

Table 4

Reproducibility of each item

Item	Item no.	κ	95% CI
C01	3	0.76	0.69–0.82
C02	3	0.59	0.50–0.67
C03	4	0.71	0.64–0.77
C04	5	0.74	0.69–0.79
C05	3	0.75	0.70–0.80
C06	5	0.69	0.63–0.76
C07	3	0.80	0.75–0.85
C08	3	0.65	0.58–0.71
C09	3	0.68	0.62–0.75
C10	3	0.63	0.57–0.69
C11	3	0.72	0.65–0.77
C12	3	0.55	0.47–0.62
C13	3	0.71	0.66–0.76
QOL01	3	0.57	0.51–0.63
QOL02	3	0.68	0.62–0.74
QOL03	3	0.64	0.58–0.70
QOL04	3	0.73	0.67–0.78
QOL05	5	0.58	0.52–0.63
QOL06	5	0.55	0.49–0.60
QOL07	5	0.62	0.56–0.67
QOL08	5	0.56	0.50–0.61
QOL09	5	0.56	0.50–0.61
QOL10	5	0.62	0.57–0.67
QOL11	5	0.49	0.42–0.54

Discussion

Measurements of the outcome of cervical myelopathy must have sufficient reliability and validity and should be proven by means of statistical evaluation. The current JOA scoring system for cervical myelopathy has been widely used but has not been shown to meet such a requirement.5,6 A manner of measuring patient-based outcome has also been regarded as essential for evaluating a patient’s health status. We sought to develop a new questionnaire using a psychometric method. The newly proposed document, which is self-administered and disease-specific, includes function of the cervical spinal cord and cervical spine as well as health-related quality of life. The first- and second-round surveys selected 24 of 77 items after repeated discussions among the subcommittee members.1,2

In this third survey, we succeeded in confirming the reliability of the 24 items. The next steps are to check the validity of the questionnaire, perform a factor analysis to determine the underlying cluster of the question naire items, complete the measurement scale, and confirm the responsiveness of the questionnaire.

Conclusion

The newly developed JOA Cervical Myelopathy Evaluation Questionnaire (JOACMEQ) was proven to be sufficiently reliable.