Reporting on the level of validity and reliability of questionnaires measuring Katakori severity: A systematic review.

OBJECTIVES: Katakori is a Japanese term for non-specific symptoms including discomfort or dull pain caused by muscle stiffness around the neck through the shoulders and is one of the most frequently reported symptoms in Japan. However, there has been no standardized patient-reported outcome measure to evaluate Katakori severity. This study aimed to investigate the reporting level on validity and reliability of patient-reported outcome measures of Katakori severity.
METHOD: A systematic search in ICHUSHI, MEDLINE, EMBASE and PubMed was undertaken from inception to April 2017 without language limitations. Two authors independently undertook screening by inspecting the title and abstract. Inclusion criteria were as follows: (1) participants with Katakori symptoms, (2) reporting reliability or validity of questionnaire evaluating Katakori severity and (3) published journal articles. Studies that either of the authors retained through the screening process were inspected with full text by the two authors independently to examine eligibility of the study. Any disagreement on eligibility after full-text inspection was resolved by discussion between them. Methodological quality was rated with the COnsensus-based Standards for the selection of health Measurement INstruments checklist. Subsequently, the evidence level of each measurement property was assessed for each questionnaire. The two authors extracted data independently. Any disagreement was resolved by discussion between them.
RESULTS: Five questionnaires were identified in five studies. The Shoulder Pain and Disability Index and Scale for Measuring Felt Shoulder Stiffness had the highest level of methodological quality. However, excellent measurement properties were found in only two out of nine criteria. Furthermore, in particular, content validity was not investigated in any measure.
CONCLUSION: There is preliminary evidence for the reliability and validity of the Shoulder Pain and Disability Index and Scale for Measuring Felt Shoulder Stiffness; however, much further research is required. Identifying or developing a patient-reported outcome measure with content validity would be a future research agenda.

Introduction

‘Katakori’ is a Japanese term indicating a non-specific symptom including discomfort or dull pain caused by muscle stiffness around the occiput through the cervical spine to the acromion and scapular area.[1] Katakori significantly impacts on Japanese society, being the most frequent symptom (12.5%) in Japanese men and the second most frequent symptom (6.0%) in Japanese women.[2]

More than 550 papers have been published on Katakori in Japanese-language journals from 2011 to 2016.[3] However, there has been no standardized patient-reported outcome measure (PROM) developed to evaluate Katakori severity. Thus, it is necessary to comprehensively investigate the validity and reliability of PROM used to evaluate symptoms of Katakori. However, no systematic review has been undertaken.

Furthermore, developing a patient-individualized measure is one of the priorities in musculoskeletal research.[4] For a PROM, the patient-individualized measure includes a fully individualized questionnaire and a semi-individualized questionnaire, but not a fully structured questionnaire. A merit of the fully structured questionnaire is the ability to compare scores between patients as all patients answer all items. However, the importance of each item is different between patients. It is known that fully individualized questionnaires, where items were originally generated by each patient, have greater responsiveness than fully structured questionnaires.[5,6] Nevertheless, it is not possible to compare scores between respondents using fully individualized questionnaires. As a consequence, there has been research on the development of semi-individualized questionnaires, which allow the comparison of scores between patients by reflecting patients’ differences.[7,8] In the semi-individualized questionnaire, all patients answer all the same items and provide weight for each item. Therefore, it is also important to understand how items are described and whether the questionnaire is a structured questionnaire, semi-individualized questionnaire or a fully individualized questionnaire.

The primary purpose of this review was to evaluate the reporting level on validity and reliability of PROMs of Katakori severity in published studies. The secondary purpose was to understand the structure of the questionnaire.

Methods

Design

This systematic review was undertaken in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines[9] and updated method guidelines for Cochrane Musculoskeletal Group Systematic Reviews and Metaanalyses.[10] For comprehensive evaluation of validity and reliability of PROMs, we used the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) checklist.[11] This study was registered in the International prospective register of systematic reviews (CRD42018081104).

Identification and selection of studies

A systematic search was performed in MEDLINE, EMBASE and PubMed (Supplemental Appendix 1) from inception to April 2017 without language limitations. Katakori has been well studied in Japan and therefore a Japanese database, ICHUSHI, was also systematically searched in Japanese (Supplemental Appendix 2) from inception to April 2017. The database search was undertaken by one author (H.T.).

Two authors (K.A. and H.T.) independently undertook screening by inspecting the title and abstract of potential papers. Inclusion criteria were as follows: (1) participants with Katakori symptoms including discomfort or dull pain caused by muscle stiffness around the back of the head through the shoulders and/or shoulder blades and (2) published journal articles reporting reliability or validity of a questionnaire to understand Katakori severity. The full text of studies that either of the authors retained through the screening process was inspected by the two authors (K.A. and H.T.) independently to examine eligibility of the study. Studies for which design was not suitable to investigate reliability or validity of a questionnaire for the population with Katakori symptoms (e.g. mixing participants with other symptoms/diagnoses) were not considered eligible. Conference proceedings were also not considered eligible. Any disagreement for eligibility after full-text inspection was resolved by discussion between the two authors (K.A. and H.T.). During full-text inspection, a hand search of the reference list from the identified studies was undertaken to identify relevant work.

Quality assessment and data synthesis

Methodological quality of each study and level of evidence for a measurement property for each questionnaire were evaluated similarly to a previous systematic review.[12] The evaluations were undertaken by the two authors (K.A. and H.T.) with assessor blinding. Any disagreement was resolved by discussion between the two authors (K.A. and H.T.).

The COSMIN checklist[11] was used as the risk of bias tool for methodological quality in nine measurement properties. The COSMIN checklist has nine measurement properties (internal consistency, reliability, measurement error, content validity, structural validity, hypothesis testing, cross-cultural validity, criterion validity and responsiveness). In each checklist item, methodological quality was rated on a four-point scale (excellent, good, fair, poor). Each measurement quality was determined by the lowest score of the four-point scale in the checklist item. Unreported items were rated as not reported (NR). The evaluation was undertaken following a training session with five articles irrelevant to this study[13-17] in order to standardize skills for assessment of methodological quality.

Evidence levels of the nine measurement properties were assessed with the criteria modified from that developed by Elbers et al.[12] (Table 1). We added ‘no study’ to the unknown level.

Table 1.

Brief summary of the level of evidence for nine measurement properties, which was modified from that developed by Elbers et al.[12]

Evidence level	Criteria
Strong	Consistent findings in multiple studies of ‘good’ methodological quality OR in one study of ‘excellent’ methodological quality
Moderate	Consistent findings in multiple studies of ‘fair’ methodological quality OR in one study of ‘good’ methodological quality
Limited	One study of ‘fair’ methodological quality
Conflicting	Conflicting findings
Unknown	Only studies of ‘poor’ methodological quality OR no study

Data analysis

Agreement in the assessment of the methodological quality with five categories (excellent, good, fair, poor, NR) between the two authors (K.A. and H.T.) was evaluated with percent agreement as NR did not allow the use of an ordinal scale.

The following data were extracted from each study and summarized in a table format: (1) measure, (2) participants, (3) domains, (4) response option, (5) score range, (6) item descriptions possibly relevant to Katakori and (7) structure of the questionnaire (fully individualized, semi-individualized or structured). Two authors (K.A. and H.T.) extracted data independently. Any disagreement was resolved by discussion with the two authors (K.A. and H.T.).

Results

Flow of studies through the review

Figure 1 presents a flow of the study. Five studies were finally included in this review[18-22] and the following five questionnaires were identified: (1) Questionnaire on Musculoskeletal Disorders of the Japanese Committee on Occupational Cervico-branchial Disorders (QMD-JCOCD), (2) Shoulder Pain and Disability Index (SPDI), (3) Scale for Measuring Felt Shoulder Stiffness (SMFS), (4) Questionnaire on Stiff Shoulders with Activities of Daily Living (QSSADL) and (5) 12-item QSSADL.

Figure 1.

Flow of the literature inclusion.

Table 2 summarizes the methodological quality and evidence level of validity and reliability of the five studies. Percent agreement in the assessment of the methodological quality was 84.44%. There was no study investigating measurement error, content validity, cross-cultural validity and responsiveness.

Table 2.

Summary for methodological quality and evidence level of measurement properties in five studies and questionnaires.

Authors, year	Measure	Internal consistency	Reliability	Measurement error	Content validity	Structural validity	Hypothesis testing	Cross-cultural validity	Criterion validity	Responsiveness
Laubli et al.[19], 1991	QMD-JCOCD	NR/unknown	NR/unknown	NR/unknown	NR/unknown	Fair/limited	Fair/limited	NR/unknown	NR/unknown	NR/unknown
Hill et al.,[18] 2011	SPDI	Excellent/strong	NR/unknown	NR/unknown	NR/unknown	Excellent/strong	NR/unknown	NR/unknown	Good/moderate	NR/unknown
Nara,[22] 2011	SMFS	Excellent/strong	Fair/limited	NR/unknown	NR/unknown	Excellent/strong	NR/unknown	NR/unknown	Good/moderate	NR/unknown
Kuge et al.,[21] 2014	QSSADL	Poor/unknown	NR/unknown	NR/unknown	NR/unknown	Fair/limited	Fair/limited	NR/unknown	Fair/limited	NR/unknown
Miyazaki et al.,[20] 2014	12-item QSSADL	NR/unknown	Fair/limited	NR/unknown	NR/unknown	NR/unknown	Fair/limited	NR/unknown	Poor/unknown	NR/unknown

QMD-JCOCD: Questionnaire on Musculoskeletal Disorders of the Japanese Committee on Occupational Cervico-branchial Disorders; SPDI: Shoulder Pain and Disability Index; SMFS: Scale for Measuring Felt Shoulder Stiffness; QSSADL: Questionnaire on Stiff Shoulders with Activities of Daily Living; NR: not reported.

Methodological quality (excellent, good, fair, poor, NR)/evidence level of measurement properties (strong, moderate, limited, conflicting, unknown).

The SPDI and the SMFS had the greatest number of strong evidence for measurement properties, but for only two out of a maximum nine. Table 3 summaries the five studies. All questionnaires were identified as structured in design.

Table 3.

Summary of five studies.

Authors, year, country	Measure	Participants	Domains	Response option	Recall time	Number of items, Score range	Item descriptions	Structure (structured, semi-individualized, fully individualized)	Evaluated properties in reliability and validity
Laubli et al.,[19] 1991, Germany	QMD-JCOCD	644 individuals in 12 occupational categories in services-providing sector. Individuals with severe abnormality or serious and long-term illnesses were not considered eligible	Six domains: neck–shoulder region; right upper limb, correlated with fatigue, pain and cramp in the right arm and hand; left upper limb, correlated with fatigue, pain and cramps in the left arm and the hand; sensitization disorders, symptoms of falling asleep and tingling in both arms and sleeping/sleep disorders; spine and cross-section including pain and stiffness; and hand tremor	0: Never/seldom1: Occasionally2: Almost daily	Unknown	32 items, 0–64	• Neck shoulder: sum of the variables (stiffness, pain) for the neck and shoulders (0–24)• Back pain: sum of variables (stiffness, pain) (0–8)• The right arm: sum of variables (fatigue, pain, cramps) for the right arm and hand (0–12)• The left arm: sum of variables (fatigue, pain, cramps) for the left arm and hand (0–12)• Sensitization: sum of the four variables for tingling (0–8)	Structured	• Structural validity using factor analysis• Hypothesis testing using comparisons in the six factor scores between those taking and not taking pain-killers, and those with and without physician consultation
Hill et al.,[18] 2011, Australia	SPDI	588 individuals who had ever felt pain or aching in their shoulder at rest or when moving, on most days for at least a month or who had ever had stiffness in their shoulder when getting out of bed in the morning on most days for at least a month	Two domains: pain; disability	11-point numerical rating scale	1 week	13 items, 0–100	• How severe is your pain at its worst?• How severe is your pain when lying on the involved side?• How severe is your pain reaching for something on a high shelf?• How severe is your pain touching the back of your neck?• How severe is your pain pushing with the involved arm?• How much difficulty do you have washing your hair?• How much difficulty do you have washing your back?• How much difficulty do you have putting on an undershirt or pullover sweater?• How much difficulty do you have putting on a shirt that buttons down the front?• How much difficulty do you have putting on your pants?• How much difficulty do you have placing an object on a high shelf?• How much difficulty do you have carrying a heavy object of 10 pounds?• How much difficulty do you have removing something from your back pocket?	Structured	• Internal consistency using Cronbach’s alpha for the two domains• Structural validity using principal components factor analyses with varimax rotation
Nara,[22] 2011, Japan	SMFS	506 students with subjective symptoms of Katakori	Three domains: numbness and a prickle; dull pain of a deep part; sense of congestion	Seven-point Likert-type scale (very untrue of me–very true of me)	Unknown	28 items, 0–168	• Stretching• Swelling• Pricking• Throbbing• Tired• Heavy• Cold• Burning• Squeezing• Pressing• Like there is something in subcutaneous • Continuous pulling • Empty • Punched • Numb • Shooting • Stiff • Dull • Aching • Stuck • Stringy and hard • Insensitive • Cramping • Tender • Tense • Hard to move	Structured	• Internal consistency using Cronbach’s alpha for each domain• Structural validity using principal components factor analyses with varimax rotation• Test–retest reliability using Pearson correlation in a subset of the participants (n = 61) with a 5-week interval• Hypothesis testing using Pearson correlation to the frequency of Katakori (1: not at all, 4: frequent), scores of the State-Trait Anxiety Inventory, Beck Depression Inventory and the Coronary-Prone Scale for Japanese
Kuge et al.,[21] 2014, Japan	QSSADL	121 students	Two domains for Type C: movement of the pectoral girdle; vitality	Type A: Yes/No for the RDQ with the change of term ‘low back pain’ to ‘shoulder pain (Katakori)’Type B: Yes/No for the modified Type AType C: four-point Likert-type scale (never, seldom, often, very often) for Type B	Today	Type A: 24 items, 0–24Type B: 21 items, 0–21Type C: 21 items, 0–63	• Type A• I stay at home most of the time because of my shoulder pain (Katakori).• I change position frequently to try to get my shoulder pain (Katakori) comfortable.• I walk more slowly than usual because of my shoulder pain (Katakori).• Because of my shoulder pain (Katakori), I am not doing any jobs that I usually do around the house.	Structured	• Internal consistency using Cronbach’s alpha for all items in Type C• Structural validity using principal components factor analyses
							• Because of my shoulder pain (Katakori), I use a handrail to get upstairs.• Because of my shoulder pain (Katakori), I lie down to rest more often.• Because of my shoulder pain (Katakori), I have to hold on to something to get out of an easy chair.• Because of my shoulder pain (Katakori), I try to get other people to do things for me.• I get dressed more slowly than usual because of my shoulder pain (Katakori).• I only stand up for short periods of time because of my shoulder pain (Katakori).• Because of my shoulder pain (Katakori), I try not to bend or kneel down.• I find it difficult to get out of a chair because of my shoulder pain (Katakori).• My back is painful almost all of the time.• I find it difficult to turn over in bed because of my shoulder pain (Katakori).My appetite is not very good because of my shoulder pain (Katakori).• I have trouble putting on my sock (or stockings) because of my shoulder pain (Katakori).• I can only walk short distances because of my shoulder pain (Katakori).• I sleep less well because of my shoulder pain (Katakori).• Because of my shoulder pain (Katakori), I get dressed with the help of someone else.		• Hypothesis testing using Sidak multiple comparisons with the scores in Type C and Katakori severity in a four-point Likert-type scale (0: never feel, 1: sometimes feel, 3: feel, 4: always feel)
							• I sit down for most of the day because of my shoulder pain (Katakori).• I avoid heavy jobs around the house because of my shoulder pain (Katakori).• Because of shoulder pain (Katakori), I am more irritable and bad tempered with people than usual.• Because of my shoulder pain (Katakori), I go upstairs more slowly than usual.• I stay in bed most of the time because of my shoulder pain (Katakori).• Type B and C• I stay at home most of the time because of my shoulder pain (Katakori).• I change position frequently to try to get my shoulder pain (Katakori) comfortable.• Because of my shoulder pain (Katakori), I find it difficult to do manual operation (office work).• Because of my shoulder pain (Katakori), I am not doing any jobs that I usually do around the house.• Because of my shoulder pain (Katakori), I find it difficult to walk with a hand bag on the shoulder.• Because of my shoulder pain (Katakori), I lie down to rest more often.• Because of my shoulder pain (Katakori), I try to get other people to do things for me.• I get dressed more slowly than usual because of my shoulder pain (Katakori).
							• I only hold baggage for short periods of time because of my shoulder pain (Katakori).• Because of my shoulder pain (Katakori), I try not to reach the arm and get something at a high place.• I find it difficult to raise the arm because of my shoulder pain (Katakori).• I have shoulder pain (Katakori) almost all of the time.• • I find it difficult to turn because of my shoulder pain (Katakori).• My appetite is not very good because of my shoulder pain (Katakori).• I have trouble putting on my jacket because of my shoulder pain (Katakori).• I sleep less well because of my shoulder pain (Katakori).• Because of my shoulder pain (Katakori), I get dressed with the help of someone else.• I sit down for most of the day because of my shoulder pain (Katakori).•  I avoid heavy jobs around the house because of my shoulder pain (Katakori).• Because of shoulder pain (Katakori), I am more irritable and bad tempered with people than usual.• I stay in bed most of the time because of my shoulder pain (Katakori)
Miyazaki et al.,[20] 2014, Japan	QSSADL	137 individuals	Two domains: movement of upper quarter; vitality	Type A and B: four-point Likert-type scale	Today	Type A and B: 12 items, 1–4	• Type A• I stay at home most of the time because of my shoulder pain (Katakori).• Because of my shoulder pain (Katakori), I find it difficult to do manual operation (office work).• Because of my shoulder pain (Katakori), I am not doing any jobs that I usually do around the house.• Because of my shoulder pain (Katakori), I lie down to rest more often.• I get dressed more slowly than usual because of my shoulder pain (Katakori).• I only hold baggage for short periods of time because of my shoulder pain (Katakori).• Because of my shoulder pain (Katakori), I try not to reach the arm and get something at a high place.• I find it difficult to raise the arm because of my shoulder pain (Katakori).	Structured	• Test–retest reliability of Type B using ICC with a 2-week interval• Hypothesis testing using ICC between Type A and Type B• Hypothesis testing using comparisons of the mean score in each domain between individuals with and without Katakori
							• I have trouble putting on my jacket because of my shoulder pain (Katakori).• I sit down for most of the day because of my shoulder pain (Katakori).• Because of shoulder pain (Katakori), I am more irritable and bad tempered with people than usual.• I stay in bed most of the time because of my shoulder pain (Katakori)• Type B• I stay at home most of the time because of my Katakori.• Because of my Katakori, I find it difficult to do manual operation (office work).• Because of my Katakori, I am not doing any jobs that I usually do around the house.• Because of my Katakori, I lie down to rest more often.• I get dressed more slowly than usual because of my Katakori.• I only hold baggage for short periods of time because of my Katakori.• Because of my Katakori, I try not to reach the arm and get something at a high place.• I find it difficult to raise the arm because of my Katakori.• I have trouble putting on my jacket because of my Katakori.• I sit down for most of the day because of my Katakori.• Because of Katakori, I am more irritable and bad tempered with people than usual.• I stay in bed most of the time because of my Katakori

QMD-JCOCD: Questionnaire on Musculoskeletal Disorders of the Japanese Committee on Occupational Cervico-branchial Disorders; SPDI: Shoulder Pain and Disability Index; SMFS: Scale for Measuring Felt Shoulder Stiffness; QSSADL: Questionnaire on Stiff Shoulders with Activities of Daily Living; ICC: intra-class correlation coefficient; RDQ: Roland–Morris Disability Questionnaire.

Discussion and conclusion

This study systematically searched the literature for studies reporting on the reliability or validity of Katakori severity. This study provides research agenda in relation to Katakori evaluation.

Generally, the overall quality of evidence for measurement properties in five questionnaires was not strong. The SPDI and SMFS had the largest number of properties with strong evidence but for only two out of nine properties. Furthermore, there was no study investigating measurement error, content validity, cross-cultural validity and responsiveness. It is important to note that content validity, which is considered a principal measurement property,[23] was not established in the current literature. These findings indicate a need to investigate content validity of the existing measures in the population who report Katakori, and a need to develop a new questionnaire fully satisfying the nine measurement properties when there is no content validity in the existing measures.

In the current literature, all questionnaires were structured measures. Developing a patient-individualized measure is one of the priorities in musculoskeletal research.[4] Comparing scores between individuals is not possible with fully individualized measure, but it is possible with semi-individualized measures, allowing reflection on individual differences between patients. Therefore, a semi-individualized measure would be ideal when a new tool is investigated.

Potential future research direction

Content validity is conventionally assessed with the Content Validity Index and multirater kappa coefficient of agreement among an expert panel.[24] However, there is uncertainty whether the expert panel fully understands symptoms and disabilities associated with Katakori. Therefore, it would be prudent to first fully understand symptoms and disabilities due to Katakori from a patient’s point of view. Problem elicitation technique (PET) which identifies problems that are most important to the individual patient has been used for investigation of content validity in previous studies[25,26] and would appear a useful tool to investigate Katakori symptoms from the patient’s perspective.

Study limitations

This study was limited to investigations on patients with definite Katakori symptoms. However, there is no clear English translation for Katakori. Therefore, there may be questionnaires that were actually used for patients with Katakori symptoms, but patient’s symptoms were not specific enough to be included in this study (e.g. patients with neck pain). A further limitation is that full investigation of the gray literature was not undertaken in this review.

Conclusion

The SPDI and SMFS were the most fully investigated measures with respect to reliability and validity based on the literature review. However, content validity was not investigated in measures for Katakori severity and should be investigated in the future. Furthermore, it was found that all questionnaires regarding Katakori severity in the current literature were structured PROMs, not a patient-individualized PROM.

Click here for additional data file.

Supplemental material, supplemental for Reporting on the level of validity and reliability of questionnaires measuring Katakori severity: A systematic review by Kaori Aoki, Toby Hall and Hiroshi Takasaki in SAGE Open Medicine

Click here for additional data file.

Supplemental material, supplemental_2 for Reporting on the level of validity and reliability of questionnaires measuring Katakori severity: A systematic review by Kaori Aoki, Toby Hall and Hiroshi Takasaki in SAGE Open Medicine