The reliability and validity of a new professionalism assessment scale for young health care workers.

With the development of medical technique, many medical challenges have been solved. This cross-sectional study was aimed to assess the reliability and validity of a new developed scale for assessing the professionalism of young health care workers.This cross-sectional population-based study was conducted based on cluster sampling method in 2015. The participants were from 2 medical centers of Nantong city of Jiangsu province in China. The internal consistency reliability was assessed using Cronbach's alpha and split-half reliability coefficients. Construct validity was analyzed using exploratory factor analysis (EFA) and confirmatory factor analysis (CFA). Content, convergent, and discriminant validities were also assessed in this study.Total 749 individuals aged 29.09 ± 3.91 years old participated in this investigation. There were 745 valid questionnaires and 730 were complete. Cronbach's alpha value (0.944) and the split-half reliability coefficient (0.873) reflected satisfactory internal consistency reliability of this new professionalism assessment scale. EFA extracted a 7-factor model. About 63.4% of the total variance was explained by these factors. However, CFA showed a good model fit after excluding the items with factor loading lower than 0.5. Good discriminant validity of this new developed professionalism assessment scale was also shown (P < .05). However, the evidences for content and convergent validity were not enough in this study.The results showed the satisfactory reliability of this new developed professionalism assessment scale. However, this scale should be modified to improve the validity in further studies.

Introduction

With the development of medical technique, many medical challenges have been solved. However, the low professionalism is always a key factor influencing the practice outcomes of these medical techniques and development of medical and health services.[ Currently, the academic study of medical professionalism is becoming very common. The fostering or training of professionalism in medical education is encountered and shows a positive role in development of medical professionalism.[ Thus, it is necessary to develop a standard scale to evaluate the medical professionalism at present.

In 1988, Arnold et al[ developed a questionnaire and supported the development of a reliable scale that measures professionalism within the environment of medical education and residency training. However, the reliability of this questionnaire was not high enough (Cronbach's alpha  =  0.71). Afterward, many related scales or inventory have been developed. Among them, some scales are used to measure the professionalism of medical students.[ Besides, there is also an inventory to measure the medical students’ critical reflections on professionalism in gross anatomy.[ However, there is still no standard scale for assessing the professionalism of health care workers.

Generally, professionalism was associated with the occupational values, job stress, organizational commitment, and so on. So, we developed a new designed professionalism assessment scale based on these related scales.[ As most health care workers were young adults in China, so we evaluated the professionalism in young health care workers aged less than 40 years old in this study and the reliability and validity of this new designed scale were assessed.

Methods

Questionnaires

The questionnaire used in this study was composed of 3 sections (attachment 1). The first section was an ad-hoc questionnaire for recording the demographics of participants including sex, age, marriage, education background, annual income, time for learning, frequency of participating training for continuing education in past 1 year, professional title, and type of staff. The second section was a questionnaire for investigating the hot issues about professionalism. The third section was a professionalism assessment scale which was designed based on the Occupational Values Scale developed by Wu et al in 1995,[ Job Stress Scale developed by Liu et al in 2005,[ and Organizational Commitment Scale developed Mowday et al in 1979.[ Give no theoretical construct of health care professionalism has been reported in detail in previous studies, we designed this new professionalism assessment scale with 8 subscales based on the above 3 previous scales, including professional ideals, professional attribute, professional responsibility, professional credibility, professional conscience, professional discipline, professional style, and professional skills. There were 5 items in each subscale. The score of each item was ranged from 1 to 5 based on the answers: very agree (5 scores), agree (4 scores), neutral (3 scores), disagree (2 scores), and very disagree (1 scores).

Participants and study design

We conducted this cross-sectional population-based study based on the cluster sampling method in 2015. The cluster sampling method was performed by including all the health care workers (aged <40 years old) in Nantong First People's Hospital and Nantong Traditional Chinese Medicine Hospital. This study has been approved by local ethics committees. Participants were informed the nature and purpose of the study before providing written consent.

Specially trained investigators visited participants to anonymously fill in the questionnaires. A total of 749 health care workers have participated this investigation with a response rate of 92.8%. Among them, there were 745 valid questionnaires (99.5%) and 730 questionnaires (97.5%) were complete. According to the criteria in the book of Advanced Medical Statistics,[ the sample size was recommended with at least 10 fold of number of items in questionnaire. Thus, there was enough sample size in the present study.

Statistical analysis

Data were entered using EpiData 3.1 and analyzed using SPSS19.0 and Amos 17.0. Data of continuous variables were shown as mean ± standard deviation (SD). The frequencies and percentages were used to describe the classification variables. The differences among groups were evaluated using Student's t test, one-way analysis of variance (ANOVA), or chi-square test. A P-value < .05 was considered statistically significant.

For the reliability test, Cronbach's alpha coefficient was used to assess the internal consistency with alpha ≥0.70 as satisfactory internal consistency reliability.[ Meanwhile, split-half reliability was assessed based on the Spearman-Brown Formula, where the Spearman–Brown coefficient ≥0.7 was considered satisfactory.[

Content validity was tested based on the correlations between the subscale and total scale scores using the Pearson correlation coefficient. Convergent validity was assessed by correlational analyses (Pearson correlation coefficient) of total scores of this new designed professionalism assessment scale and the previous scale (Occupational Values Scale developed by Wu et al in 1995,[ Job Stress Scale developed by Liu et al in 2005[ and Organizational Commitment Scale developed Mowday et al in 1979.[ For all the analysis, the absolute value of correlation coefficient (r) ≥ 0.40 was considered satisfactory (0–0.2: poor; 0.21–0.40: fair; 0.41–0.60: good; 0.61–0.80: very good; 0.81–1.0: excellent).[

Discriminant validity was evaluated by comparing results between participants grouped by characteristics using Student's t test or ANOVA. Good discriminant validity was confirmed by significantly different results (P < .05) in participants grouped by characteristics.

Construct validity was analyzed using exploratory factor analysis (EFA) and confirmatory factor analysis (CFA). EFA was performed using the principal component analysis (PCA) with varimax rotation. Bartlett's test of sphericity and Kaiser–Meyer–Olkin (KMO) statistic were used for testing the possibility of performing factor analysis. Factors with eigenvalues greater than 1 were extracted. Factor loadings of more than 0.50 were considered satisfactory. CFA was performed to determine the goodness-of-fit of the extracted factor model. The root mean square error of approximation (RMSEA) and comparative fit index (CFI) were recorded for testing the fit of the model to the covariance matrix in CFA. The ACFI value of >.90 and a RMSEA of < .08 represents a satisfactory and acceptable model fit.[

Results

Demographics of the subjects

The demographics of these participants were shown in Table 1. Most participants were doctors (36.3%) and nurses (56.1%). The other participants were medical technicians or administrators. There were significant differences between doctors and nurses in age, annual income, frequency of participating training for continuing education in past 1 year, sex, education background, marital status, professional title, and type of staff (P < .001). The doctors were markedly older than nurses and have more annual income. Although doctors had higher frequency of participating training for continuing education in past 1 year than nurses, the time for learning in usual was similar between them (P  =  .701). Most nurses were females, but the sex radio was almost equal to 1 in doctors. There were 55.4% of doctors with an education background at Master's or higher degree, whereas 59.8% of nurses have an education background at associate's degree. Most doctors (75.7%) were married, but only half of nurses (49.2%) were married. There were significantly more individuals with intermediate professional title in doctors compared with that in nurses (32.6% vs 8.2%). Nearly half of doctors (44.4%) were authorized staff, but most nurses (84.7%) were nonauthorized staff. Based on the investigation results in the second section, most participants (78.3%) believed that the professional responsibility was reflected in helping patients to cure diseases. Results also indicated that the development of professionalism could be affected by the health care system as well as social and working environments (data not shown).

Table 1

The characteristic of participants and differences between doctors and nurses.

Professionalism assessment

The results of this investigation were shown in Table 2. The total score was 31.61 ± 3.40. The doctors have significantly higher total score than nurses (32.12 ± 4.13 vs 31.17 ± 2.90, P  =  .005). For all the subscales scores, results also showed significantly higher scores in doctors than in nurses (P < .05), except for the subscale of professional conscience.

Table 2

The total and subscale scores in this investigation.

Reliability

Cronbach's alpha value of this scale was 0.944, which exceeded the 0.70 level indicating high internal consistency reliability of this new developed professionalism assessment scale. Meanwhile, the alpha value was reduced when each subscale was deleted. Therefore, no subscale needs to be omitted from this scale. In addition, the split-half reliability was performed by averagely dividing questions based on the parity of item numbers. The calculation of a Spearman–Brown coefficient resulted in a split-half reliability of 0.873, which was enough to confirm the internal consistency of this new designed professionalism assessment scale.

Convergent and discriminant validity

There was a significant positive correction between the total scores of this new designed professionalism assessment scale and Occupational Values Scale developed by Wu et al in 1995 (r  =  0.560, P < .001)/Organizational Commitment Scale developed Mowday et al in 1979 (r  =  0496, P < .001), indicating good convergent validity of this new designed professionalism assessment scale. However, the total scores of Job Stress Scale developed by Liu et al in 2005 showed a fair negative relation with this new designed professionalism assessment scale in this study (r  =  0.214, P < .001).

Table 3 showed that the males had a higher total score than females (P < .001), but no significant difference was found between female and male doctors. Results also showed that the individuals with associate's or lower degree had lower total scores than the individuals with bachelor's or higher degree (P < .05). In addition, the significantly different total scores between nurses and doctors were disappeared in individuals aged less than 30 (P  =  .454), individuals with bachelor's degree (P  =  .070), unmarried individuals (P  =  .549), individuals with intermediate or higher professional title (P  =  .996), as well as authorized staff (P  =  .091). Moreover, the doctors with primary professional title showed higher total scores than the doctors with immediate or higher professional title (P < .001). These results indicated that the age, sex, education background, marriage, professional title, and type of staff may be the factors affecting the professionalism of health care workers, and this new designed professionalism assessment scale had good discriminant validity to discriminate the different professionalism among these participants.

Table 3

The results of subgroup analyses.

Content validity

As shown in Table 4, there were significant corrections between total and subscale scores as well as between subscales scores (P < .001, r >0.4). Moreover, the Pearson correlation coefficient between total and subscale scores was higher than that between subscales scores. These results indicated these subscales could explain the professionalism well, suggesting that there was a satisfactory content validity of this scale.

Table 4

The correction between the total score and subscale scores.

Construct validity

The EFA showed that the data were appropriate for factoring (KMO  =  0.943; Bartlett's test  =  17,411, P < .001). The PCA revealed 7 factors with eigenvalues greater than 1 (Fig. 1). About 63.5% of the total variance was explained by these factors. The factor loadings of items were shown in Table 5. Subsequently, the CFA was conducted to determine the goodness-of-fit of this 7 factors model. Results showed that this seven factors model did not show a good fit to the data (P < .001, CFI  =  0.794, RMSEA  =  0.082). After excluding the items with factor loading lower than 0.5 (c5: You suppose that hard working bring satisfied salary and decent life; d5: You are able to calmly deal with issues which are not being understood by others; e2: You suppose that it is the professional responsibility for every medical staff to make great efforts to relieve disease suffer of patients; e3: You can persist in hard working and take initiative to improve the quality of work; g3: You can sacrifice your interests to help patients; h5: When confronted with work pressure, you are able to face up to it with a positive attitude) and modifying the model based on the modification indices, a good model fit was obtained (P < .001, CFI  =  0.903, RMSEA  =  0.061, Fig. 2).

Figure 1

Scree plot for the eigenvalues of factors.

Table 5

The factor loading of items in this 7 factors model.

Figure 2

The 7-factor model based on confirmatory factor analysis.

Discussion

In the present study, a new scale for assessing the professionalism of young health care workers was developed. Results showed this scale had enough internal consistency reliability and split-half reliability. However, the validity of this scale may be not enough based on the following evidences: (i) although satisfactory content validity was shown based on the correlations between subscale and total scale scores, the content validity still need to be further assessed by a panel of experts in future considering the method used in this study was weak to assess the content validity. This is a limitation of this study. (ii) Only 7 factors were identified in the EFA, which was inconsistent with the actual number of subscales. Besides, this 7 factors model was not well fit to the actual data based on CFA. However, a good model fit was obtained after excluding the items with factor loading lower than 0.5 and modifying the model based on the modification indices. Thus, the construct validity of this scale was not satisfactory and this scale should be revised (some items with factor loading lower than 0.5 should be eliminated or modified) based on the results of EFA and CFA in this study. (iii) Although the good discriminant validity of this scale was shown, the evidences for good convergent validity were not enough because fair but not good negative correlation between the Job Stress Scale and this new scale was found. Moreover, Occupational Values Scale, Job Stress Scale, and Organizational Commitment Scale only can reflect partly content of this new scale, so they are not enough to be used for assessing the convergent validity. Thus, the convergent validity of this new scale should be further assessed based on a professionalism assessment scale commonly used in global.

The definitions for professionalism were different in the previous studies. In some previous studies, only behavior was considered as medical professionalism.[ With the development of medicine, the definitions for medical professionalism become more and more extensive and complicated. Some studies have reported professionalism in medicine includes a set of values, attitudes, and behaviors that results in serving the interests of patients and society before one's own.[ Besides, medical professionalism is also defined as the ability to meet the ethical expectations required to practice medicine competently.[ In the present study, the content in the new designed professionalism assessment scale included items assessing values, attitudes, behaviors, and ethics, and could comprehensively evaluate the professionalism of health care workers. However, the Learners’ Attitude of Medical Professionalism Scale the scale, which is developed for Arabian, only assessed the medical professionalism based on attitudes, and only the good reliability of this scale was reported in this published study.[ Similarly, the Penn State College of Medicine Professionalism Questionnaire is just developed to assess the attitudes of medical students toward professionalism, and the good reliability and unsatisfied construct validity of this scale were shown.[ For the scale developed by Klemenc-Ketis and Vrecko,[ it not only considers the attitudes towards professionalism but also the self’ perceptions of participants (medicine students), and good reliability and validity were reported. However, these previously developed scales could not be used to evaluate the professionalism comprehensively based on above definitions for medical professionalism. Thus, although the dissatisfied construct validity, the new developed scale in the present study was still supported to be used for assessing the professionalism of young health care workers. For improving the construct validity, the exclusion or revision of these items with factor loading lower than 0.5 should be considered in the further studies.

Based on the results of investigation, we found that the doctors had higher professionalism than nurses, except in professional conscience. It was reported that conscience was associated with medical ethics.[ Thus, the doctors and nurses may have similar knowledge in ethics. In addition, results also showed that the professionalism assessment results could be affected by age, sex, education background, marriage, professional title, and type of staff. The differences between doctors and nurses in professionalism may be caused by these confounding factors. Some previous study reported that the social responsibility and individual responsibility could be affected by age.[ In addition, the highly educated individuals would have more and better professional skills than individuals with lower education. In this study, the differences in professionalism between individuals aged <30 and ≥30 years old as well as between individuals with associate's or lower degree and bachelor's or higher degree were identified, indicating this new developed scale have a good content validity.

A new professionalism assessment scale was developed. The satisfactory reliability and discriminant validity of this scale were obtained. Further studies should be performed to modify this scale to improve the construct validity. More evidences for good content and convergent validity should be explored in further studies.