The Development of Instrument to Assess Physician's Practice in the Management of Patients With Terminal Diseases.

BACKGROUND: The need for improving knowledge and practice of palliative care delivered by health workers become an agenda in several countries. In order to measure the practice, an instrument is needed. The study analyzed the validity and reliability of the instrument to assess the physician's practice in the management of patients with terminal diseases.
METHODS: This was a cross-sectional study involving 89 physicians practicing in primary health care. The instrument of practice has been developed and resulted 5 domains consist of 20 items. An overview of reliability, construct validity, uni-dimensionality, and hierarchy of the person-items of the instrument were analyzed using Rasch Model.
RESULTS: The reliability of the instrument is excellent with a person measure reliability of 0.85 and the item measure reliability of 0.96. Construct validity is confirmed with the MNSQ outfit values in the range of 0.54 to 1.59 and Pt Measure Corr. values in the range of 0.31 to 0.8. This instrument has a value of more than 20% unidimensionality which indicates the level of independence for items is good.
CONCLUSION: The instrument has good validity and reliability to assess physician's practice in the management of patients with terminal disease.

Background

Access to the health services at the end of life including access to palliative care is patient’s right.[1] Primary health care as a closest health facility in the community has a great opportunity to provide palliative care.[2] Health workers in primary health care have a significant role in providing palliative care, because most families of the patients prefer to take care for family members at home rather than in hospitals.[3,4] However, several studies show that some health workers in primary health care are not equipped with appropriate knowledge and skills for palliative care.[1,5-8] In some countries, palliative care has been not implemented yet due to limited number of trained health workers.[1] One study highlights that only physicians who were interested in the end of life care have good knowledge regarding the care.[6]

Health workers who demonstrate optimal practice in palliative care had a stronger effect to patient’s outcome.[9] The quality of palliative care influences the patient’s quality of life.[10]

The need for knowledge and quality of good palliative care by health workers become an agenda in several countries, including Indonesia.[11,12] The knowledge and skills for good palliative care include pain and symptom management, psychological, and spiritual support,[13] holistic communication,[4,14-16] decision making and timely referral.[17] In order to assess knowledge and practice, an instrument is needed. To our knowledge, there is no instrument measures the practice of palliative care or care for patients with terminal diseases in Indonesia. The study was aimed to develop the instrument to assess the physician’s practice in the management of patients with terminal diseases and analyzed its validity and reliability using Rasch Model.

Rasch model is statistical model to examine the observed data against a standard model.[18] The model measures the data at the response-level that indicate to what extent these ideals are realized within any particular data set.[19] Rasch analysis measures person ability and item difficulty to provide information regarding performance of the instrument.[18] Therefore, Rasch model analysis could evaluate the strengths and weakness of the instrument.[20]

Methods

Design

The overall study was designed through 2 phases. Phase 1 was qualitative study using in-depth interview to explore the experience of health workers. Phase 2 was cross-sectional study to analyze instrument’s validity and reliability. This paper will focus to report the result of phase 2. The study was conducted in Bandung District between September and November 2018. The study has been approved by The Medical Research Ethics Commission Faculty of Medicine, Universitas Padjadjaran with registration number 1198/UN6.C10/PN/2017.

Participants

The in-depth interviews were conducted to health workers exploring a reflection of their experience and skills needed to care patients with terminal diseases. Primary care physicians, specialists, psychologist, and nurses were selected purposively. The reflection of participants was used to develop domain of practice for the instrument.

The participants for validity test were physicians practicing in primary health care, at least 1-year experience in practice and willing to participate. Eighty-nine physicians were selected from 5 selected districts. Sample size of 89 was considered adequate to be analyzed by Rasch Model.[19] Sociodemographic characteristics of respondents is described in Table 1.

Table 1.

Characteristics of Participants.

Characteristics	Frequency (n)	Percentage (%)
Sex
Male	30	34
Female	59	66
Age (mean; SD)
39.6; 13.44
Education
Undergraduate	78	88
Postgraduate	11	12
Length of practice
<5 years	30	34
6-10 years	18	20
>10 years	41	46

The Instrument

The instrument was developed through a qualitative research. Practice is described by 5 domains and 20 questions.[1] The domains are case notification (4 questions), comprehensive and integrated care (10 questions), follow up (2 questions), and interdisciplinary collaboration (4 questions). Response categories for all items included yes, sometimes and never. Yes answer is defined as health personnel almost always performed an activity when dealing with terminal diseases. Sometimes answer is defined as health personnel almost never performed an activity when dealing with terminal diseases. Never answer refers to health personnel never performed an activity when dealing with terminal diseases.

Data Collection

Sampling frame were obtained from the district health office. The sampling method used was total sampling by selecting all doctors practicing in the selected area. The data was collected by trained enumerators. The enumerator made an appointment by telephone to meet and asked permission to distribute the questionnaire. After permission was granted, all participants were provided with written informed-consents, signed them and filled out a questionnaire. The participants completed the instrument in average of 10 to 15 min.

Data Analysis

Rasch analysis was performed to determine the validity and reliability using the Winstep software 3.73. Rasch analysis can produce good and accurate instruments to assess a person’s confidence in doing practice and determine person and items validity simultaneously.[20] The Rasch analysis provides an overview of reliability, construct validity, uni-dimensionality and hierarchy of the person-items.[20-22] The following fit criteria were used in the present analysis.

Reliability of the instruments is needed to assess the degree of consistency. A good measurement will show a high degree of reliability when producing a consistent score. Reliability is measured by (1) Cronbach’s alpha, to measure the interaction between person and items as a whole; (2) item reliability index and person reliability.[22-24] Item and person reliability are estimated with values more than 0.7.[21]

Item validity refers to items that are fit and misfit using values INFIT MNSQ (information-weighted mean square). Validity for each item refers to OUTFIT MNSQ (outlier-sensitive mean square), Z score standard (ZSTD) and Point Measure Correlation (Pt Mean Corr).[19,20] The values for OUFIT MNSQ between 0.5 and 1.5; OUTFIT ZSTD between −2.0 and 2.0; and Pt Mean Corr between 0.32 and 0.85 are acceptable.[19,25] Infit is used to indicate unexpected patterns of response for person ability and outfit indicates the unexpected rater responses.[19,26]

The Person-Item Map describes the respondent’s ability to answer the question correctly. This map illustrates the ability of respondents to be categorized from lowest to highest and the quality of items to be categorized from easy items to difficult items visually.[19,20,26]

One of the most important considerations in designing a measurement is to ensure unidimensionality based on PCA (Principal Component Analysis).[19,27] The questionnaire have to meet criteria for unidimensionality with minimum diversity requirements of 20%.[19]

Results

Reliability

Based on analysis, the reliability of the instrument is excellent with a person measure reliability of 0.85 (Table 2) and the item measure reliability of 0.96. The item separation index is 4.63 and the respondents’ separation index is 2.42. Cronbach’s alpha value is 0.91.

Table 2.

Summary of Person and Items Reliability and Index Separation.

	Person					Items
	Logit	Infit		Outfit		Logit	Infit		Outfit
		MNSQ	ZSTD	MNSQ	ZSTD		MNSQ	ZSTD	MNSQ	ZSTD
Mean	1.67	0.98	0.0	0.96	0.1	0.00	1.02	0.0	0.96	−0.1
Separation	2.42					4.63
Reliability	0.85					0.96

Validity

Based on the results of the Rasch analysis, MNSQ outfit values (0.5-1.5), and Pt Measure Corr (0.32-0.85) of the instrument are acceptable (Table 3). The item difficulty ranges from −3.87 to 2.25 logits. Item number 1 has a logits value of −3.87 which means this is a very easy question. This item also has Pt mean corr 0.19 and categorized as misfit item. Therefore, this item is removed from the final instrument.

Table 3.

Descriptive Statistics of Items.

No.	Items	Logit	Infit		Outfit		PT-measure
			MNSQ	ZSTD	MNSQ	ZSTD	CORR.	EXP.
1	Identify physical problems	−3.87	0.99	0.2	0.54	−0.1	0.19	0.19
2	Identify psychological problems	−1.09	1	0.1	0.77	−0.4	0.48	0.49
3	Identify the psychological, spiritual, socio-economic need of patients and family members	0.30	1.43	2.5	1.42	1.7	0.5	0.61
4	Provide information and discuss the patient’s problems	−1.69	1.38	1.4	0.97	0.2	0.32	0.42
5	Provide palliative care in the early stage of disease	1.49	0.91	−0.6	1.04	0.3	0.64	0.67
6	Give pain drugs based on guideline and standard	−0.88	1.57	2.4	1.59	1.4	0.38	0.51
7	Discuss the patient’s problems with psychologist	1.77	1.11	0.8	1.06	0.4	0.68	0.68
8	Provide psychological support to patients and family members	−0.05	0.87	−0.7	1.40	1.5	0.54	0.58
9	Involving family members in case management	−1.02	0.89	−0.5	0.65	−0.8	0.56	0.49
10	Consider patient’s preference in management plan	−0.05	1.02	0.2	1.24	0.9	0.53	0.58
11	Assess financial source of patients and family	1.22	1.37	2.4	1.31	1.7	0.53	0.66
12	Discuss patient’s problems from spiritual aspect	2.25	0.72	−2.1	0.73	−1.6	0.73	0.69
13	Assess patient’s perception of the diseases, death and how to deal with them	0.47	0.57	−3.3	0.67	−1.7	0.73	0.62
14	Prepare process of dying and death with comfort	0.82	1.05	0.4	0.86	−0.7	0.71	0.64
15	Regular follow up to assess patient’s and family’s problems	0.39	0.78	−1.5	0.67	−1.7	0.72	0.62
16	Provide care at the clinic, home, and community setting	0.30	1.06	0.4	1.11	0.6	0.61	0.61
17	Collaborate with other professionals for case management	0.26	0.71	−2.0	0.59	−2.1	0.72	0.61
18	Consult or refer patients to specialist when needed	−1.32	1.07	0.4	0.84	−0.2	0.45	0.46
19	Be care coordinator when referring patient to other professionals	0.66	1.11	0.8	0.93	−0.3	0.64	0.63
20	Share responsibility in the case management	0.04	0.8	−1.2	0.73	−1.1	0.66	0.59

The Person-Items Maps

The left side of the person-items map describes the respondent’s ability and the right side is the quality of the item (Figure 1). There are 5 people who have a highest person logit. The 20 items of the instrument are presented from the easiest (item number 1, bottom) to most difficult (item number 12, top).

Figure 1.

Person-items map.

Uni-dimensionality

Based on Rasch analysis, each item is analyzed to identify whether item could measure the intended construct (uni-dimensionality). The measurement results 49% diversity of data and meet the criteria of unidimensionality (more than 20%).

Discussion

The purpose of palliative care for patients will be achieved if the physician provides appropriate care. To our knowledge, there are no instruments assessing practice for palliative care in Indonesia. This study aimed to develop and analyze the validity and reliability of instruments to assess physician’s practice in palliative care using the Rasch analysis. Items in this instrument have been selected and correspond to the construct model of the Rasch.

Rasch analysis can predict source of measurements’ error from respondents and items. The more information about the person ability and items difficulty will suggest the less the measurement error.[18] Overall, all items (20 questions) show good person and items reliability. The value of the person reliability test is 0.85, indicates that the instrument has strong reliability to distinguish practice. The value of the item’s reliability item is 0.96, indicates the items in this instrument could be distinguished based on the level of difficulty. An alpha Cronbach indicates that data are in accordance with the Rasch analysis requirements and the instrument can be used in different conditions. Cronbach’s alpha value of 0.91 indicates that the interaction between respondents and items shows a good reliability. The high separation value indicates that the instrument is sensitive enough to distinguish participant with good and poor practice.[19]

Based on Rasch analysis, validity can be assessed from the person validity and items validity. The person validity can be seen from the average value of logits as shown in Table 2. The logits of +1.67 (a value of more than 1.0) indicates that the respondents have a tendency to approve the item in the instrument. The infit mean-square value of 0.98 (Table 2) indicates the observed data has 2% less variation than predicted by the Rasch model. Index separation of respondents of 2.21 means that respondents can be divided into 2 groups based on the level of practice. The greater the separation index, the better the quality of the instrument, because it can identify the group of person and items.[21]

The items validity is seen from the average logits value of 0.0 (Table 2). It shows the instrument can assess practice. Items separation of 4.83 means that the item can be used with 4 categories or scale of response. Items fit can be seen from the MNSQ value of each item (Table 3). The items in this study have MNSQ outfit value in the range of 0.54 to 1.59. In addition, the accuracy of items is evaluated from the polarity analysis of the items. It is assessed from Pt Measure Corr value. Most items in this instrument have values in the range of 0.31 to 0.8. Based on the result this instrument is appropriate for assessing practice.

The person-items map provide information regarding how well the items are distributed with regard to the ability of respondents.[20,23] Figure 1 indicates the items match to the practice of physicians. The analysis shows that the mean of items measure is close to the mean of person measure. This arrangement indicates good test-item targeting.

Unidimensionality is used to determine the extent to which the diversity of instrument measures what should be measured.[21] Unidimensionality show that the scale of the instrument measure only one construct. This instrument has a value of more than 20% unidimensionality which indicates the level of independence for items is good.

Conclusion

Overall, analysis with Rasch model revealed that the instrument has good validity and reliability. Therefore, this instrument could be used to assess physician’s practice in the management of patients with terminal diseases.