The development of an evaluation tool to assess professional behavior and clinical competencies from the graduates' perspective.

PURPOSE: This study was designed to develop an evaluation tool for assessing professional behavior and clinical competencies from the graduates' perspective.
METHODS: This study employed mixed method in a sequential exploratory design. The semi-structured interviews were conducted with three graduates from different cohorts. The qualitative analysis of the interviews found six emerging themes for professional behavior and clinical competencies development. These themes were then developed into a 55-item questionnaire. The questionnaire was then distributed to 84 medical graduates for exploratory factor analysis (EFA) from February to April 2019. The quantitative data were analyzed using IBM SPSS ver. 21.0 (IBM Corp., Armonk, USA) for principal axis factoring. After conducting EFA, we proceeded with confirmatory factor analysis (CFA) with another 120 graduates to validate the tool.
RESULTS: Eighty-four graduates completed the questionnaire for EFA. Upon completion of EFA, 35 out of 55 items of the questionnaire were found to be valid and reliable. The most appropriate fit was seven factors, which explained 58.18% of variance between them after 15 iterations with Cronbach's α of 0.916. The personal satisfaction factor was noted to be weak. It was therefore added to patient management factor due to its similar intention. The final EFA factor after the modification was six. The CFA found that 34 out of 35 items was valid and reliable that representation of the latent variables.
CONCLUSION: The questionnaire has achieved the desired construct validity score and can be used as an evaluation tool to assess professional behavior and clinical competencies from the graduates' perspective.

Introduction

Medical professionalism is a normative system that organizes and delivers health care. It calls upon its group members to jointly declare (“profess”) what the public and individual patients expected, based on shared competency standards and ethical values. These standards and values ensure that all medical professional will deliver quality care to their patients [1]. Professionalism is understood to exist or develop in varying degrees as a characteristic or attribute that is identifiable within individuals. In other words, professionalism can be conceptualized and assessed at different levels: individual, interpersonal and institutional/societal [2].

The professional behavior and clinical competency of a doctor reflect a range of personal and interpersonal qualities, attributes, behaviors, commitments, and values [3-6]. Doctors are expected to be accountable, altruistic, committed to excellence, compassionate, respectful, responsive, sensitive to diversity, and ethically sound [3,6].

Many evaluation tools have been developed to assess a doctor’s professional behavior. For example, Professional Mini Evaluation Exercise assesses the medical residency’s professional behavior from the clinical supervisor’s perspective [7,8]. Our search of the literature, however, could not find a tool that was developed to assess professional behavior and clinical competencies development from a graduate perspective, although they are one of the stakeholders in medical education [9]. As self-assessment is an essential factor in enhancing and improving professional ability [10,11], graduates can provide valuable input to the faculty for improving a medical education program because they have undergone the curriculum and have the means to assess its effectiveness for their medical practice. Therefore, there is a need to develop a tool to assess professional behavior and clinical competencies development from the graduates’ perspectives.

We have published our work on graduates’ performance evaluation from the perspectives of the preceptors and patients [11,12]. In this article, we will elaborate on the graduates’ perspective on their professional behavior and clinical competencies development. The graduates’ perspective of professional behavior and clinical competencies is important in helping Universitas Islam Bandung to evaluate its educational outcome achievement as the graduates can further elaborate on the behaviors and competencies that were needed from their workplace-based experience.

This study aimed to develop a tool for assessing professional behavior and clinical competencies development from the graduates’ perspective.

Methods

The development of the questionnaire involved three aspects: (1) the study participants; (2) the factor analysis, with 20% of target population as the sample size [13] or between 10:1 to 20:1 subject to factor ratio [14,15]; and (3) the unstandardized parameter and their standard error of mean [16].

1. Study design

Using a mixed-method sequential exploratory design, a qualitative study preceded the tool (questionnaire) development [17]. It was followed by exploratory factor analysis (EFA). The results of the EFA guided questionnaire revision. The revised questionnaire was then distributed for confirmatory factor analysis (CFA). Fig. 1 summarizes the steps involved in the study.

Fig. 1.

Summary of the Steps Involved in the Study

2. Sampling strategy and study participants

A total of 207 graduates were involved in the study. The qualitative study used purposive sampling while EFA and CFA used random sampling strategies. The qualitative study involved three graduates for semistructured interviews. The EFA involved 84 graduates while the CFA involved another 120 graduates.

3. Study procedures

The qualitative study involved three medical graduates from batches 2006, 2007, and 2008. Upon receiving written consent, they were interviewed between 45 to 60 minutes per person. Examples of the questions asked to the graduates were as shown in Fig. 2. The interview questions were formulated based on the literature review of the topic and the research objectives. It was reviewed many times by the research team to ensure its clarity. The qualitative study preceded tool development because it was necessary to explore the graduates’ perception of their professional behavior and clinical competencies development first. All interview transcripts were transcribed verbatim. Thematic analysis of the transcripts found seven emerging themes. The themes were (1) professional behavior, (2) humanity, (3) patient management, (4) clinical skills competence, (5) personal satisfaction, (6) cognitive competence, and (7) interpersonal skill. The description of the theme is as listed in Table 1 [18,19].

Fig. 2.

Examples of the Questions for Semi-Structured Interview with the Graduates

UNISBA: Universitas Islam Bandung.

Table 1.

The Description of the Themes

Theme	Description
Professional behavior	A set of quality and attributes of a person behavior that is observable and relates to medical practice [18,19].
Humanity	The qualities of being a human that relates to a physician role [19].
Patient management	The way physician act in managing a patient.
Clinical skills competence	Clinical skills deemed necessary by the graduates for medical practice.
Interpersonal skill	Skills thatcontribute to successful dealing in maintaining appropriate relationship with patient and other colleagues [18].
Cognitive competence	Knowledge required by the physician for medical practice.
Personal satisfaction	Any means that bringgratification, pleasure, or contentment to the graduates during medical practice.

The themes that emerged from the interviews were then developed into a 55-item questionnaire based on the seven steps recommended by Stenfors-Hayes et al. [16]. The seven steps were familiarization, condensation, comparison, grouping, articulating, labelling, and contrasting. The 55-items questionnaire consists of eight items for professionalism, 11 items for humanity, five items for patient management, 12 items for clinical skill competence, four items for personal satisfaction, six items for cognitive competence, and nine items for interpersonal skill. Table 2 illustrated the items for each theme and the source of references [1-3,5,20-23].

Table 2.

Items Supporting Each Theme and the Source of References

Theme	Items	Source of references
Professional behavior	Adequate experience, doctor-patient communication, problem-solver, adequate knowledge, time management, analytical, adequate skill, adequate competence	Arnold [1] (2002)
		Epstein and Hundert [20] (2002)
Humanity	Comforting, humble, religious values, respect, religious, character, continually improving, lifelong learner, counselor, discipline, empathy, hardworking	Epstein and Hundert [20] (2002)
		Mueller [5] (2009)
Patient management	Wholeheartedness, psychological support, confidence, proper management, patient education	Arnold [1] (2002)
		Campbell et al. [21] (2007)
		Mueller [5] (2009)
Clinical skills competence	Active listener, subject expert, nasogastric tube insertion, intravenous cannulation, circumcision, performing minor surgery, handling trauma patient, injections, burns, poisoning & emergency, hospital admission, clinical decision	Epstein and Hundert [20] (2002)
		Hodges et al. [2] (2011)
Interpersonal skill	Subordinate communication, cooperation, peer communication, community perspective, personal instinct, opportunity for improvement, self-confidence, lifelong learning, humble	Hodges et al. [2] (2011)
		Haque et al. [22] (2016)
		Adam et al. [3] (2012)
Cognitive competency	Knowledge about disease, knowledge about treatment, referral, number of cases learnt, knowledge about management, cases variation	Epstein and Hundert [20] (2002)
		Mueller [5] (2009)
Personal satisfaction	Personal satisfaction, patient perspective, job satisfaction, medical knowledge satisfaction	Epstein and Hundert [20] (2002)
		Bahaziq and Crosby [23] (2011)

4. Ethical approval

The Health Research Ethics Committee, Faculty of Medicine, Bandung Islamic University has approved this observational study (approval letter no., 005/Ethic committee, FK/VI/2017). The informed consent was obtained from all individual participants included in the study.

Results

1. Exploratory factor analysis

The 55-item questionnaire was later distributed to 84 medical graduates using with a participant to factor ratio of 12:1 [15]. It was distributed for 3 months from February to April 2019. Prior to conducting EFA, Kaiser-Myer-Olkin (KMO) counting and Bartlett’s test of sphericity were performed to obtain sampling adequacy [24]. The KMO counting showed an adequate sample has been achieved if it has a value above 0.5 or 0.6. After obtaining an adequate sample size with a KMO value of 0.546 (>0.5) and Bartlett’s test of sphericity of p<0.001 (χ2=3,682.84), the extraction method was conducted to acquire and describe the data structure [13,14] using IBM SPSS ver. 21.0 (IBM Corp., Armonk, USA). Confirmation of the data extraction that was able to be loaded in the same factor or scale was done using principal axis factoring of varimax rotation with Kaiser normalization.

The varimax rotation method with Kaiser normalization on EFA extracted seven factors based on eigenvalue of more than one (Table 3, Fig. 3). These seven factors were retained initially in the questionnaire and accounted for 58.18% of the total variance. In Table 4, the matrix pattern of the varimax rotation method with Kaiser normalization showed the grouping of the items based on a loading value of >0.5. For example, question 11, 12, 13, 15, and 16 were categorized into factor 4 due to their loading values. Other items were classified in the same manner.

Table 3.

Total Variance Explained Based on the Initial Eigenvalue >1

Component	Initial eigenvalue			Rotation sums of square loadings
	Total	% of variance	Cumulative %	Total	% of variance	Cumulative %
1	14.040	25.527	25.527	5.740	10.437	10.437
2	4.497	8.176	33.703	5.417	9.848	20.285
3	3.970	7.219	40.921	4.673	8.496	28.780
4	2.734	4.971	45.892	4.400	8.001	36.781
5	2.644	4.807	50.699	3.353	6.097	42.878
6	2.146	3.902	54.601	3.081	5.602	48.480
7	1.968	3.579	58.180	2.202	4.003	52.483

Fig. 3.

Result of Confirmatory Factor Analysis of Six Components on the Graduate's Instrument

(A) Item C1–C16. (B) Item C17–C35. hum: Humanity, cog: Cognitive competence, skl: Clinical skill competence, pro: Professional behavior, Pxm: Patient management, ips: Interpersonal skill, df: Degrees of freedom, RMSEA: Root mean square error of approximation.

Table 4.

Results of Principle Axis Factoring with Orthogonal Varimax Rotation of 55 Items (n=84)

Item		Factor 1	Factor 2	Factor 3	Factor 4	Factor 5	Factor 6	Factor 7[a)]
Q1.	Knowledge about diseases that I gained during my education supported my work in the field.	0.276	0.220	0.145	-0.001	-0.051	0.480	0.049
Q2.	Knowledge about kind and dose of remedy that I learned helped me towards medical practice.	0.092	0.113	0.197	0.120	0.176	0.477	-0.080
Q3.	Physician should make a referral when it is beyond their capacity.	0.221	0.498	0.054	-0.048	0.171	0.141	0.027
Q4.	The cases learned during clerkship are useful and applicable at workplace of medical practice.	0.212	0.643	0.060	0.255	0.096	0.288	-0.115
Q5.	Management of trauma I learned during medical education helps me in the field.	0.033	0.074	0.201	0.115	-0.096	0.646	0.128
Q6.	The variation of cases during the clerkship was sufficient.	0.178	0.214	-0.117	0.194	0.223	0.550	0.017
Q7.	To be a good doctor we should be able to be an active listener.	-0.029	0.223	-0.054	-0.021	0.373	0.226	-0.089
Q8.	Learning during the medical doctor program makes me master medical subjects optimally.	0.465	0.361	-0.041	0.338	-0.005	0.262	-0.146
Q9.	I have adequate skill in inserting nasogastric tube.	0.094	0.357	-0.018	0.363	0.067	0.200	0.080
Q10.	I am quite skilled in performing the insertion of the iv catheter.	0.048	-0.014	0.003	0.397	0.115	0.240	-0.015
Q11.	I am capable to perform circumcision.	0.059	-0.068	0.221	0.681	0.192	0.342	0.247
Q12.	I am skilled in performing minor surgery at workplace of medical practice.	0.168	0.234	0.267	0.634	-0.090	-0.165	0.222
Q13.	I am skilled in handling patient with trauma.	0.237	0.164	0.070	0.711	0.046	0.191	0.093
Q14.	I am skilled in injecting intra-muscular, intravenous, or sub-cutan.	0.194	0.422	0.030	0.465	0.124	0.208	0.013
Q15.	I can handle cases of burn injury.	0.279	0.045	0.036	0.656	0.186	-0.150	-0.028
Q16.	I can handle cases of poisoning and emergency.	0.521	-0.079	0.068	0.528	-0.017	-0.297	0.025
Q17.	I know well when patient needs to be hospitalized or not.	0.397	0.310	0.025	0.208	0.322	0.023	0.068
Q18.	I can make clinical decisions well.	0.544	0.225	0.182	0.305	0.305	-0.061	0.143
Q19.	I am pleased going through my medical profession.	0.627	0.285	0.177	0.055	0.389	-0.034	-0.045
Q20.	I feel my patients are quite satisfied with my performance.	0.092	-0.075	-0.036	0.591	0.564	0.008	0.008
Q21.	I am satisfied to be a medical doctor.	0.660	0.023	0.226	0.049	-0.007	0.126	0.053
Q22.	I am satisfied with my medical education.	0.350	-0.200	-0.060	0.189	0.117	0.128	-0.020
Q23.	Clerkship experience makes me competent as a medical doctor.	0.484	0.247	-0.028	0.095	0.043	0.066	0.141
Q24.	I can communicate well with patient and her/his family.	0.539	0.421	0.046	0.086	0.432	0.020	0.057
Q25.	My medical education makes me become a good problem-solver.	0.501	0.108	0.224	0.263	0.218	0.246	-0.069
Q26.	My medical knowledge is appropriate and applicable in medical practice.	0.579	0.109	0.225	0.257	0.047	0.203	-0.228
Q27.	My medical education made me manage my time properly.	0.661	0.048	0.249	0.166	0.044	0.102	0.311
Q28.	My medical education enables me to analyze every medical problem that I find.	0.621	0.020	0.001	0.025	-0.026	0.116	0.323
Q29.	Clinical experience during the clerkship has supported me to be a skilled doctor.	0.453	0.230	0.369	0.149	-0.147	0.169	-0.026
Q30.	My education is sufficient to achieve competence as a physician.	0.496	-0.029	0.374	0.087	-0.135	0.426	-0.012
Q31.	Making the patient feel comfortable is a compulsory attitude that must be had by a Muslim doctor.	-0.042	0.785	0.134	0.066	-0.031	-0.067	0.014
Q32.	In serving patients I prioritize the attitude of being humble & not conceited.	-0.064	0.567	0.522	0.068	0.112	-0.158	0.020
Q33.	I work on my medical profession in accordance to the values of Islam.	0.076	0.543	0.553	-0.100	-0.004	0.061	0.028
Q34.	I have shown politeness and respect to the patient and his family.	0.185	0.802	0.193	-0.001	0.034	-0.018	0.065
Q35.	Understanding about Islam that I learned during my medical education has built my character as a Muslim doctor.	0.042	0.414	0.653	-0.037	-0.267	0.101	0.167
Q36.	My improvisation ability increased when I am running the medical practice.	0.092	0.389	0.074	0.210	0.229	-0.226	0.475
Q37.	I am always updated with my medical knowledge and skill.	0.155	0.223	0.153	0.145	-0.034	0.180	0.697[b)]
Q38.	I always give counseling, information, and education to the patients.	-0.096	0.474	0.494	0.002	0.084	0.166	0.232
Q39.	My education has established me to be a disciplined doctor.	0.216	-0.030	0.570	0.021	-0.096	0.275	0.475
Q40.	I always empathize toward my patients.	0.085	0.037	0.352	0.008	-0.032	-0.004	0.359
Q41.	As a Muslim doctor, I have to give my best effort for the optimal outcome of the patient and submit myself to God.	0.051	0.602	0.011	0.045	0.175	0.160	0.279
Q42.	I am considered as a good and skilled professional by other health workers at my workplace.	0.268	-0.161	0.206	0.370	0.455	0.143	0.357
Q43.	I am capable of cooperating in the team well.	0.049	0.144	0.247	0.261	0.760	0.022	-0.052
Q44.	I am capable of communicating with hospital staff and other health care professions.	0.388	0.213	0.079	0.074	0.551	0.070	0.039
Q45.	The community appraises me fairly well.	0.112	0.001	0.539	0.210	0.412	0.153	0.210
Q46.	My instinct as a doctor was trained during internship.	0.291	0.491	0.135	0.008	0.246	-0.005	0.313
Q47.	The examinations during medical education provide me opportunities for better performance.	0.217	-0.061	0.267	0.005	0.059	0.545	0.240
Q48.	I have a good self-confidence.	0.169	0.096	0.214	0.481	0.489	0.071	-0.001
Q49.	My medical education made me always self-aware about life-long learning.	0.244	0.133	0.501	0.230	0.121	0.258	0.013
Q50.	I do not appraise myself exclusively.	-0.053	0.429	0.144	0.290	0.152	0.179	-0.168
Q51.	I serve my patients wholeheartedly.	0.261	0.023	0.720	0.018	0.046	0.254	0.146
Q52.	I give support to the patient psychologically to encourage his recovery.	0.137	0.146	0.538	0.066	0.219	0.112	-0.034
Q53.	My confidence comes after handling 3–4 patients.	0.027	-0.144	0.177	-0.079	-0.329	0.250	-0.110
Q54.	I can manage my patients properly.	0.548	-0.166	0.336	0.219	0.229	0.157	0.263
Q55.	My explanation to the patient & his family makes them adhere to the treatment.	0.183	0.113	0.558	0.245	0.101	-0.017	-0.012

Factor 1: Professional behavior, Factor 2: Humanity, Factor 3: Patient management, Factor 4: Clinical skill competence, Factor 5: Interpersonal skill, Factor 6: Cognitive competence, and Factor 7: Personal satisfaction.

Item in factor 7 was inserted to factor 3 (due to only 1 item).

Thirty-five items were retained as latent/potential variables (loading value of >0.5 except item Q3 which had a value of 0.498). Q3 item was retained in the questionnaire because it provided the purpose to assess professional capacity. Twenty items were removed because their loading values were less than 0.5.

The calculation of internal consistency was conducted using Cronbach’s α coefficient [9,10] and interpreted according to the five categories recommended by Guilford and Fruchter [25] in 1978. The 35-item questionnaire had a very high internal consistency as evidenced by Cronbach’s α of 0.916 (Table 5) [14,26,27]. However, upon completion of EFA, one of the seven factors was noted to be weak and was added to another factor. Therefore, only 35 items within six factors were retained in the questionnaire, as demonstrated in Table 6. The list of questions for the revised questionnaire used in CFA is shown in Table 7 [28-31].

Table 5.

Result of Internal Consistency Cronbach’s α of Six Factors (n=84)

Factor name	Cronbach’s α
Professional behaviors	0.875
Humanity	0.849
Patient management	0.832
Clinical skills competence	0.814
Interpersonal skill	0.785
Cognitive competence	0.624

Table 6.

Subscales and Items of 35-Item Extracted Instrument following Factor Analysis of the Medical Graduate Results (n=84)

The item (Q)	Factor name	Amount of item retained
18, 19, 21, 24, 25, 26, 27, 28, 54	Professional behavior	9
3, 4, 31, 32, 33, 34, 41	Humanity	7
35, 39, 45, 49, 51, 52, 55, 37[a)]	Patient management	8
11, 12, 13, 15, 16, 20	Clinical skills competence	6
43, 44,	Interpersonal skill	2
5, 6, 47	Cognitive competence	3
Total item		35

Inserted to factor 3 (patient management); factor 7 (personal satisfaction) was deleted.

Table 7.

Comparison of Factors Extracted in This Study with CanMEDS and ACGME’s Competency Framework

Factors extracted	CanMEDS Physician Competency Framework [28,29]	ACGME’s competencies [30,31]
Cognitive competence	Medical expert, scholar	Medical knowledge
Interpersonal skill	Communicator, collaborator, manager	Inter-personal & communication skills
Professional behavior	Professional	Professionalism
Patient management	Health advocate	Patient care
Clinical skill competence	-	Practice-based learning & improvement, systems based practice
Humanity	-	-

ACGME: The Accreditation Council for Graduate Medical Education.

2. Confirmatory factor analysis

CFA is a part of structural equation modelling. Upon completion of EFA, CFA was then conducted on another 120 graduates to validate the construct of the evaluation tool. The sample size for CFA was based on participant to factor ratio of 20:1. The CFA was conducted using LISREL ver. 8.8 program for Windows (Scientific Software International Inc., Lincolnwood, USA). The CFA results were presented in Fig. 3. The t-loading value of all items in each group of factors was counted and was stated as “valid” if it has a value above 1.96 (at 95% confidence interval). The highest t-loading value was deemed as the most reliable indicator that represents its latent variable.

Fig. 3 showed that 34 out of 35 items (indicator variable) were valid and reliable in representing the six latent variables. Humanity variable was represented by item C2-C7. Item C7 was the strongest indicator with a t-value of 12.22. Cognitive competence variable was represented by item C8-C10. The strongest indicator was item C10 with t-value of 7.31. Clinical skills competence was represented by items C11 to C16 with item C11 being the strongest indicator with a t-value of 9.14. Professional behavior variable was represented by item C17 to C25. Item C20 was the strongest indicator with t-value of 8.62. Patient management variable was represented by item C26-C33. Item C29 was the strongest indicator with t-value of 9.88. Inter-personal skill variable was represented by item C34-C35. Item C34 was a stronger indicator with t-value of 8.98.

Discussion

David Kern stated that the process of curriculum development and evaluation requires an assessment of the stakeholders’ needs, in order to determine what aspects should be included in the curriculum [26]. This study was conducted to develop an evaluation tool for assessing the development of professional behavior and clinical competencies from the graduates’ perspective. The data collected from this tool will be used by UNISBA to evaluate its current medical curriculum.

The seven factors obtained from conducting EFA with 35-item questionnaire indicated high communality in which almost all items have the value of more than 0.7 [27]. The factor analysis yielded nine items for professional behavior, seven items for humanity, seven items for patient management, six items for clinical skill competence, one item for personal satisfaction, three items for cognitive competence, and two items for interpersonal skill.

Five of the seven factors identified in this study were strong factors because they have three or more items except for interpersonal skill factor and personal satisfaction factor. The one item for personal satisfaction was then inserted to patient management factor because it shared similar intention. The interpersonal skill factor remained as it is with two items. Therefore, the final extracted factors were six.

The factor which had the most items was factor 1 (professional behavior). All nine items in factor 1 were retained. The item with the highest loading value was item 34 (0.802) in factor 2 (humanity). Items included in patient management factor were aspects and attitudes that the graduates perceived they must have. These factors were extracted from the semi-structured interviews. As doctor-patient communication is one of the aspects known to increase patient’s adherence to medical treatment and enhanced clinical outcomes [11,32], its inclusion in the questionnaire was deemed necessary.

It was surprising that only two items were retained in the interpersonal skill factor [3], although this skill was important for medical practitioners [7]. A possible explanation could be due to the paternalistic way of managing patient in our country [33].

Most of the factors identified in this study were similar to the competency domains of the CanMEDS Physician Competency Framework [28,29] and the Accreditation Council for Graduate Medical Education [30] competencies except for humanity factor (Table 7) [28-31]. Humanity was chosen as one of the factors because it represented the vision and mission of UNISBA’s Medical School. Therefore, this factor might not be of interest to other institutions unless they have similar vision and mission.

1. Study limitation

Our study has several limitations. Firstly, the sample size of EFA was small due to a limited number of graduates available for the study. Second, the interview was conducted in Bahasa Indonesia. There could be a misinterpretation of meaning when the emerging themes were translated into English. Third, the original 55-item and 35-item questionnaires were distributed to the graduates in Bahasa Indonesia. There could also be a misinterpretation of meaning when the factors extracted from the literature was translated from English to Bahasa Indonesia for the questionnaire, and when it was translated again into English for publication.

2. Conclusion

Validity and reliability are essential psychometric properties of an evaluation tool. A high index of validity and reliability of an evaluation tool indicated that the tool was well constructed. The validity and reliability index obtained by this tool signified that this tool is valid and reliable in assessing the professional behavior and clinical competencies development from the graduates’ perspective. It also indicated that the evaluation tool can be valid and reliable to be used for collecting data as a source of information for curriculum evaluation exercise. Future work will investigate the similarities and differences of perspectives by all stakeholders in evaluating UNISBA’s medical program.