Developing and validating an instrument to measure: the medical professionalism climate in clinical settings.

This study was conducted to develop and validate an instrument to measure the medical professionalism climate in clinical settings. The item pool was developed based on the Tehran University of Medical Sciences Guideline for Professional Conduct. The items were distributed between two questionnaires, one for health-care providers and the other for patients. To assess the construct validity of the questionnaires, 350 health-care providers and 88 patients were enrolled in the study. The reliability of the questionnaires was evaluated by calculating Cronbach's alpha and ICC. At first a 74-item pool was generated. After assessing and confirming face and content validity, 41 items remained in the final version of the scale. Exploratory factor analysis revealed the three factors of "personal behavior", "collegiality" and "respect for patient autonomy" in a 25-item questionnaire for service providers and a single factor of "professional behavior" in a 6-item questionnaire for patients. The three factors explained 51.775% of the variance for service providers' questionnaire and the single factor explained 63.9% of the variance for patients' questionnaire. The findings demonstrated that from the viewpoints of patients and service providers, this instrument could be applied to assess the medical professionalism climate in hospital clinical settings.

Introduction

Professionalism is fundamental in maintaining the public trust in medical professions (1). Medical professionalism regulates health-care providers’ behavior in their interactions with patients, other health-care providers, and the health-care organization. In the past two decades, professionalism has been recognized as an essential medical competency and is now included in the undergraduate and postgraduate curricula (2-4). Professionalism has also received increasing attention in Iran over the past decade. Professionalism is currently defined as a core competency for graduate medical education, and its position is being recognized as an integral part of the formal curricula in Iran. In 2013, Tehran University of Medical Sciences (TUMS) developed a framework for professionalism based on a project. The framework consisted of six domains (altruism, honor and integrity, respect, responsibility, justice and excellence), and was used to formulate a guideline for professional conduct in medical practice (5). This guideline serves to direct training and the assessment activities related to professionalism for students and faculty members.

The hospital setting has an important role in shaping students’ professional behavior. The students’ professional identity is developed more effectively through socialization in clinical settings than formal education (6). Relationships, processes and policies of the hospital setting are important parts of the informal and hidden curriculum of medical education that can facilitate or hinder the formation of professional identity in students. For example, the unprofessional behavior of attending physicians could alter the attitude of trainees and staff. Professional practice can hardly take place in an unprofessional environment. There are reports of the deterrent role of the hidden curriculum in developing professional identity and enhancing professionalism, and the importance of corrective interventions (7 - 9).

Kalet et al. assessed the formation of professional identity in medical school graduates through different steps of professional formation: independent operator, team-oriented idealist, self-defining professional and self-transforming professional. They found that medical students can end up with various results during medical school training with regard to professional identity (10).

According to Goldie, professional identity is more efficiently influenced by hidden curricula, professional role models, feedback from others, and integration into social networks in the medical environment than formal teaching measures (11).

For evaluation of the effectiveness of any corrective interventions, it is very important to assess the professionalism climate in clinical settings. In addition, the development and application of questionnaires at the beginning of professionalism enhancement programs could encourage the involved people (residents, faculty members and staff) to adhere to professional codes and improve the current situation (12,13). Professionalism is a culture-sensitive concept and any tool for measuring its constructs in the clinical setting should reflect the cultural context (14).

As the first step, we need to provide a definition for professionalism climate. This can be described as the employees’ perceptions of organizational practices in the health-care facility that reinforce professional behavior and discourage unprofessional behavior. On the other hand, the organization’s ethical climate refers to the employee beliefs about the organizational atmosphere and shows (a) whether and how ethical decisions are made, or (b) whether conditions facilitating ethical decision-making and ethical reflection exist (15).

Although different instruments have been developed to evaluate some aspects of professionalism in hospital settings, no single tool does all at once. Some instruments have focused on the organizational climate or cultural differences, and have evaluated the social capital among health-care providers and leadership in the hospital setting (15, 16). Some other tools have measured a number of professional behavior outcomes (for example patient safety or patient-centeredness of services) in clinical settings (17-19). Some instruments have only investigated the professionalism climate from the viewpoints of certain groups (16-20). Quaintance et al. developed an instrument for the assessment of the professionalism climate in clinical settings from the viewpoint of students (20), but did not include the viewpoints of faculty members and other hospital staff. Olson also proposed a tool for the assessment of ethical climate in the clinical setting from the nurses’ perspective (15).

In this study we aimed to develop a comprehensive tool for the assessment of different aspects of professionalism climate from different viewpoints.

Methods

The current instrument development study aimed to develop and validate a questionnaire to measure the climate of professionalism in clinical settings in Iran. The study was conducted in two phases:

Phase I: Item Generation

The researchers generated an item pool through developing a detailed list of examples of professional conduct based on the “Guideline for Professional Conduct in Medical Practice” developed by TUMS (5) and review of the literature. The items were generated deductively. In this phase, a comprehensive literature review was performed on published literature dating from 2010 to 2017 indexed in Medlib, Iran Medex, Magiran, SID, Irandoc, MEDLINE, EBSCO and ProQuest. This phase was designed to complete the items based on the existing literature. Articles could enter the study if: 1) they had been written in the English language, and 2) they contained the words “professional behavior”, “professional attributes” or “professional climate”, and “assessment”, “instrument” or “tool” in the title or abstract.

Considering the main attributes of professionalism, the appropriate phrases were extracted from literature. The items were reviewed, and duplicates and overlaps were merged. Then, five experts were asked to review the items and add any other relevant or necessary items. It should be added that the experts were attending physicians and had knowledge of professionalism or published papers on the subject.

The primary item pool included 91 items, which were reduced to 74 after deleting duplicates and overlaps.

As patients are a rich resource for evaluation of professionalism in the clinical setting, the experts allocated some of the items to patients, and the structure and wording of the items were changed so that they could be easier for the patients to read and understand.

Phase II: Validation

In this phase, the psychometric properties of the scale, such as its face, content and construct validity (exploratory approach), as well as its reliability were evaluated. The process of evaluating the psychometric properties of the scale was as follows: a) face and content validity, b) factorial (construct) validity, and c) reliability.

The draft of the instrument included two separate questionnaires, one for receiving the viewpoints of the patients and one for obtaining the viewpoints of service providers, including physicians, nurses, Other care providers, and residents across the university hospitals.

A) Face and Content Validity

The content validity of the questionnaire was assessed three times. Each time, a content validity assessment questionnaire was sent to 20 experts who had knowledge, information and published papers in professionalism through Google Forms. The experts were medical ethicists or attending physicians of various clinical departments of TUMS. Also, all experts were familiar with medical professionalism in the Iranian context.

In the first round, the experts were requested to rate the items as necessary, useful but unnecessary, or unnecessary. Then, the content validity ratio (CVR) was calculated based on the critical values for Lawshe’s content validity ratio (21). Out of 11 experts, at least 9 had to vote an item as necessary. The experts also were asked to provide their comments for adding new items and completing the item pool. In the second round, all items, including the new and modified ones, were returned to the experts to determine the CVR.

In the third round, the content validity index (CVI) was calculated through evaluation of each item’s relevance to the concept of professionalism using a 4-point Likert scale (1 = irrelevant, 2 = relatively relevant, 3 = acceptably relevant, and 4 = totally relevant). Moreover, they were asked to modify the items that were not relevant to the concept of professionalism. CVI was calculated for each item and items with CVIs below 0.79 were deleted. CVI was calculated by dividing the number of experts who gave each item a score of 3 or 4 by the total number of experts participating in the panel.

The face validity of the questionnaire was assessed by receiving the viewpoints of 20 experts, and also through conducting interviews with a sample of the final scale responders (including 14 faculty members, 10 residents and 15 nurses for the service providers’ questionnaire, and 10 patients for the patients’ questionnaire). The responders were requested to read the items aloud and share their comprehension with the researcher in order to assess the difficulty and ambiguity of the items. Finally, their comments for simplification and clarification of the items were collected.

For scoring, we formed an expert panel consisting of 4 experts in instrument designing and 4 experts in professionalism to determine the proper Likert scale. The items were divided into two groups, one with a five-point Likert scale of “never” to “almost always”, and the other with a five-point Likert scale of “very little” to “very much”. The Likert options for each item were given scores between 1 and 5 based on the item’s accordance with expected professional behaviors (reverse items were scored reversely). Accordingly, the total score of each questionnaire was calculated by adding up the scores of all items.

B) Factorial Validity

For evaluation of the construct validity of the instrument, we used exploratory factor analysis (22, 23). The questionnaires were completed by the staff and patients of a hospital affiliated with TUMS.

Service Providers’ Questionnaire: Subjects were selected through convenient proportional to size sampling according to the different levels of service providers. The inclusion criteria were at least one year’s work experience and at least 6 months’ experience in the ward of interest. Therefore, medical students did not meet the inclusion criteria due to short-term rotations in each ward. Three hundred and 95 service providers were enrolled, considering 5 to 10 subjects for each item (24), and there were a total of 41 items in the service providers’ questionnaire. Overall, 365 questionnaires were completed: 210 out of 220 nurses (response rate = 96%), 70 out of 85 residents (response rate = 82.3%), and 70 out of 90 faculty members (response rate = 78%). Fifteen questionnaires were excluded due to the responders’ inattention in answering the questions, so the final analysis was done on 350 subjects.

Patients’ Questionnaire: The inclusion criteria were hospitalization for at least three days in the ward and complete alertness. Eighty-eight patients were chosen to participate in the study using systematic random sampling. Six questionnaires were excluded due to missing data. The KMO index of sampling adequacy and the Bartlett’s test of sphericity for confirming the fitness of the factor analysis model were applied. The Kaiser criteria and a scree plot with a minimum factor load of 0.4 were employed to maintain the items in the extracted factors. Since some items were loaded in more than one factor, we changed the rotation to non-orthogonal rotation (direct oblimin) (25).

C) Reliability

Cronbach's alpha coefficient of internal consistency was calculated to assess the scale reliability. In order to detect at least 80.0% power of the test, minimum sample size was set at 207 (26). Cronbach's alpha coefficient was calculated on the factorial validity sample (n = 365).

Also, the relative reliability of the instrument was measured by test-retest at an interval of two weeks, and the intraclass correlation coefficient was calculated. The test-retest was done on 30 nurses for the service providers’ questionnaire and 40 patients for the patients’ questionnaire.

Results

The primary item pool included 91 items, which were reduced to 74 after deleting duplicates and overlaps. Of these items, nine pertained to the patients’ and 65 to the service providers’ questionnaires.

A) Validity

Content and Face Validity: The changes in the questionnaire item numbers in different stages of content and face validity assessment are presented in Table 1.

Table 1

The changes in the patients’ and service providers ’questionnaires in the validation process

Validation Process		Service Providers’ Questionnaire					Patients’ Questionnaire
		Initial Number of Items	Added	Removed	Modified	Total	Initial Number of Items	Added	Removed	Modified	Total
Content validity	First round	65	2	21	9	46	11	0	1	0	10
	Second round	46	0	0	6	46	10	0	1	0	9
Face validity		46	1	0	17	47	9	2		5	7
Content validity	Third round	47	0	6	0	41	7	0	1	0	6
Construct validity (Exploratory Factor Analysis)		41	-	11	-	30	6	-	0	-	6

The mean CVI was 0.945 and 0.921 for service providers’ and patients’ questionnaire, respectively.

Factorial validity/ Service providers’ questionnaire: Homogeneity of the questions was confirmed before conducting exploratory factor analysis. The KMO value was 0.925, which confirmed sampling adequacy. The high significance of the Bartlett’s test (P < 001) showed an acceptable correlation among variables for factor analysis. The scree plot showed a three-factor structure of the scale. Considering a minimum factor load of 0.4, fifteen items were eliminated. Factor analysis revealed three factors: “personal behavior” (15 items), “collegiality” (7 items), and “respect for patient autonomy” (3 items). These three factors explained 51.775% of the variance: 36.642% for “personal behavior”, 8.395% for “collegiality” and 6.738% for “patient autonomy”. All explained variances are presented in Table 2.

Table 2

The Three-factor structure of the service providers’ questionnaire

Items	Component Factor Loading
My Colleagues in this ward:	Factor 1	Factor 2	Factor 3
1 .   Make sure the patients are provided with complete information and care continuity is maintained during patient transfer or shift change.	0.837
2 .   Do their duties in a timely manner.	0.793
3 .   Observe the university dress code.	0.761
4 .   Wear their identification badges in plain sight.	0.701
5 .   Provide patients with sufficient medical information and seek their consent.	0.680
6 .   Share patients’ information only with the treatment team, the patient, or his/her companion(s), and observe data confidentiality.	0.673
7 .   Give the necessary instructions to patients and their companions.	0.673
8 .   Take the necessary steps to correct any medical errors that may have occurred.	0.614
9 .   Have a respectful behavior towards one another.	0.607
10 .   Pay attention to updating their knowledge and skills.	0.602
11 .   Are careful not to waste hospital resources and facilities.	0.598
12 .   Seek help if they lack sufficient knowledge or skills in a matter.	0.576
13 .   Resolve disputes with colleagues in a friendly manner with respect for dignity of the involved people.	0.547
14 .   Pay attention to the system’s shortcomings and give feedback to superiors.	0.515
15 .   Are careful not to waste the time of colleagues, patients and their companions.	0.474
16 .   Welcome constructive feedback.	0.837
17 .   Professional behavior is encouraged in this ward.		-0.805
18 .   Make certain that there is a good balance between educational and treatment activities.		-0.778
19 .   Make sure that the educational and administrative rules are observed.		-0.738
20 .   Take all the necessary measures to promote effective team work among members of the treatment team.		-0.738
21 .   Ensure that if there is an error, respectful feedback is given to the person at fault.		-0.736
22 .   Do everything they can so that patients’ complaints are used for correcting processes and improving services.		-0.701
23 .   For research purpose do not impose a risk or a cost on the patient..			0.847
24 .   Respect patient's beliefs.			0.704
25 .   For education do not impose a risk or a cost on the patient.			0.657
Initial Eigenvalues	9.161	2.099	1.684
Explained variance (%)	36.642	8.395	6.738
Cumulative variance (%)	36.642	45.038	51.775

Patients’ Questionnaire: Homogeneity was confirmed in the related questions. The KMO value of 0.853 showed data adequacy for statistical analysis. The high significance of the Bartlett’s test (P < 001) confirmed a sufficient correlation between variables for factor analysis. Finally, the scree plot showed a one-factor structure for this scale.

In factor analysis of patients’ data, all items were loaded under a factor with an eigenvalue of 3.78 that explained 63.9% of the variance. All items had a factor loading above 0.4 and none was eliminated in factor analysis. The highest and lowest factor loading was related to “They took care of me with sympathy and compassion” (0.786) and “They paid attention to my comfort when I was resting” (0.429), respectively (Table 3).

Table 3

The one-factor structure of the patients’ questionnaire

Item	Factor Loading
1.  They treated me respectfully and politely	0.614
2.  They answered my questions with patience	0.643
3.  They gave adequate explanations before any procedure	0.633
4.  They took care of me with sympathy and compassion	0.786
5.  They took care of my tasks in a timely manner	0.681
6.  They paid attention to my comfort when I was resting	0.429
Explained variance (%)	63.9

B) Reliability

Service Providers’ Questionnaire: In the final questionnaire, the Cronbach’s alpha coefficients for factor 1, factor 2 and factor 3 were 0.787, 0.815 and 0.888, respectively. The intraclass correlation coefficient (ICC) of the final scale was 0.816 at a significance level of 0.001.

Patients’ Questionnaire: The Cronbach’s alpha coefficient and ICC were 0.616 and 0.60 at a significance level of 0.001.

C) Further Evidence for Validity

We assessed between group differences (nurses’ vs. physicians and residents) using ANOVA. There was a significant difference [F(2,362) =9.683, P=000] between the nurses, residents, and faculty members. Post hoc comparisons using the Tukey HSD test indicated that the mean score given by residents (M=66.5, SD=14.5) was significantly lower than that given by nurses (M = 74.4, SD = 13.5) and faculty members (M = 73.8, SD = 11.5). However, the scores given by nurses did not significantly differ from those given by faculty members.

Discussion

In this study, we designed an instrument for measuring the professionalism climate in hospital settings. This instrument can be used by hospital managers and help caregivers and patients express their viewpoints on health-care workers’ adherence to each item.

The instrument consists of two parts. The first part is the service providers’ questionnaire, which contains 25 items including three factors of “personal behavior” (15 items), “collegiality” (7 items), and “respect for patient autonomy” (3 items) (Table 2). The second part is a 9-item patients’ questionnaire containing only one factor. In another scale 7 dimensions were considered: self-interest, company profit, friendship/the interest of the team, social responsibility, personal morality, rules, and laws (professional codes) (27). Quaintance et al. assessed medical professionalism based on dimensions such as respect/caring/compassion, duty/service, altruism, honesty / integrity, accountability/ responsibility, and excellence. They evaluated the extent of teaching and also acting in accordance with principles of professionalism in their environment. The instrument they used helped them assess and simultaneously upgrade the level of teaching professionalism in their setting (20). Wangsaturak and McAleer tried to develop an instrument to evaluate the different aspects of the medical education climates. In their instrument, some items about the institutional environment and practices of colleagues and teachers were related to the topics of medical professionalism including altruism, honor, integrity and excellence (28).

In this study, we assessed professionalism climate, and not ethical climate. Ethical climate has been described as maximized self-interest and collective interests and adherence to universally accepted ethical principles. In this description patients seem to be ignored. Respect for patient autonomy is highlighted in our scale in the Iranian context. Collegiality and flexible inter-professional collaborative relationships (29) are other important attributes of professionalism. In order to foster proper collaboration between nurses and physicians, it is essential to improve workplace civility to provide high-quality services (30)

Factor analysis showed that the items in the service providers’ questionnaire explained an acceptable percentage of the instrument variance for evaluation of the professionalism climate in clinical setting. Personal behavior and environment are two major factors in levels of change in the Onion model (31). Evaluation of the professionalism climate is of major significance, because in a non-professional climate, the behavior of individuals who lack adequate knowledge and the right attitude will deviate from professional norms (13). It could be stated that the climate of professionalism represents hidden curriculum management. The students’ professional identities are influenced by the context in which they are formed, so they may be affected by collective professional values (32); therefore, the climate and context of professional interactions are very important.

This instrument can measure the prevailing practice of professionalism in interaction with patients, self (excellence), colleagues, and hospital management. This questionnaire can help identify clinical settings that foster professionalism and those that require improvement, as well as enable us to follow the changes in different settings over time. Furthermore, it can detect the differences among settings or improvements over time.

The length of the instrument seems suitable for busy hospital staff to fill out. The items of the questionnaire are not related to a certain person’s behavior, and the respondents should consider the behavior and interactions of all the staff in the ward to answer the questions; therefore, people can complete the questionnaire honestly without being worried about stigmatizing a person. This questionnaire is self-administered and honesty in completing it depends on the personnel’s trust in the confidentiality of the responses.

In comparison with Olson’s questionnaire that only measures the climate of professionalism from the nurses’ perspective (15), our questionnaire considers the viewpoints of all service providers as well as patients. We believe the viewpoints of the patients could be used as a complementary resource in the assessment of professionalism, and therefore some items of the questionnaire are related to patients, which is one advantage of our instrument. This helps researchers employ the viewpoints of both health-care providers and recipients to assess the climate of professionalism in clinical settings.

The instrument designed by Thrush et al. assesses medical professionalism in the learning environment in 11 items, but it only measures the personal behavior of faculty members and residents, and medical students are the only assessors (12). In our instrument, however, professionalism is assessed from the perspectives of all parties whose professional behavior forms the climate of professionalism.

Analysis of variance indicated that the residents gave lower scores to the professionalism climate of the hospital than nurses and faculty members, but this does not mean that residents are stricter at evaluating the professionalism climate compared with nurses and professors. The difference between residents’ estimation of the professionalism climate and faculty members’ or nurses’ cannot support further validity of our tools. This difference might be due to the lower exposure of residents to the ward atmosphere and increased sensitivity to its flaws. We recommend further research in this regard to evaluate the factors truly correlated with participants’ perspective of the professionalism climate. Our instrument provides a direct observation in clinical settings. Direct observation creates a sense of safety while learning, and, since observation is bidirectional, it allows trainees to observe supervisors’ illustrating corrections (33).

In spite of the mentioned advantages, our study had some limitations. For instance, many respondents may not have the patience to complete the questionnaire, although it can provide additional information for feedback. The current questionnaire is an interim step between the original raw material and the creation of a "short-form" test.

We did not assess the correlation between the results of the service providers’ and patients’ questionnaires, which warrants other studies with appropriate sample size. Also, we did not assess the validity and applicability of the instrument in outpatient wards. Other studies are required to show whether this questionnaire can be applied in outpatient settings.

Moreover, the present study did not offer any cut-off points for acceptability of the climate of professionalism in clinical settings. Among the inclusion criteria was at least 6 months of experience in the ward; since medical students spend less than 6 months in each ward, only residents were considered eligible in the measurement of the professionalism climate. More studies are needed to determine the minimum required length of stay in the ward for assessment of professionalism climate.

Reliability of the patient questionnaire was also not quite satisfactory. This might be due to the patients’ change of mind about the professional practice in the ward after two weeks. Professionalism is understood to be a complex concept, as it is likely to mean different things in different contexts, locations and cultures (34). Since the questionnaire items are based on an Iranian medical professional code of conduct (5), the instrument may not be completely applicable in other countries and could therefore benefit from some modifications.

Conclusion

We designed a valid, and reliable instrument for the assessment of the professionalism climate in clinical settings. This questionnaire can be used to evaluate hospital clinical settings in terms of adherence to professionalism, determine areas requiring improvement, and assess the effectiveness of interventions aimed at enhanced collegiality and personal behavior.

Ethical Considerations

This study was a research project approved by TUMS, aimed at designing and validating a questionnaire to assess professional behavior in clinical settings. The protocol of the study was approved by the Ethics Committee of TUMS under code number 95-01-74-318.