Development and validation of a questionnaire to evaluate medical students' and residents' responsibility in clinical settings.

There is a shortage of quantitative measures for assessing the concept of responsibility as a fundamental construct in medical education, ethics and professionalism in existing literature. This study aimed to develop an instrument for measuring responsibility in both undergraduate and graduate medical students during clinical training. Instrument content was based on literature review and mainly qualitative data obtained from a published grounded theory research. The draft questionnaire (Persian version) was then validated and revised with regard to face and content validity. The finalized 41-item questionnaire consists of four domains that were identified using factor analysis. Test-retest reliability and internal consistency were also assessed. Test-retest reliability was rather high, ranging between 0.70 and 0.75 for all domains. Cronbach's alpha coefficients were 0.75 - 0.76 for all domains and 0.90 for the composite scale of the whole questionnaire. Correlations between the four domains of the instrument were also satisfactory (r ≤ 0.47 for most domains). The correlation between each domain and the composite scale was higher than its correlation with other domains (r ≥ 0.79 for most domains). The instrument demonstrated good construct and internal validity, and can be suitable for measuring the concept of responsibility in practice in different groups of undergraduate and graduate medical trainees (MTs).

Introduction

The concept of “responsibility” is considered to be a core value in medical education, medical ethics and medical professionalism. It is one of the main characteristics highly expected of members of the health care profession (1). Nevertheless, the existing literature consists of a limited number of scattered studies on this concept in health care (2) and more specifically in the field of medical ethics (3). A considerable bulk of related studies have merely attempted to explain this concept in relation to other research subjects or introduce new scopes of responsibility into preexisting notions (4-6). Thus the concept of responsibility itself has literally been employed as an intelligible and basic concept with a completely obvious meaning for readers of scientific papers (7, 8), and has rarely been handled as an independently ascertainable subject for research.

To the best of our knowledge, studies are also rare in the case of measuring responsibility as a quantitative term. In this respect, Mergler’s study (9) is a remarkable instance. She has developed a questionnaire for assessing “personal responsibility” in adolescents using data from focus groups and a number of related measures in the literature. Additionally, a general aspect of this concept can be found in at least one item of many quantitative instruments developed for assessment of “professionalism” (10, 11). The current study aimed to develop a questionnaire to contribute to the existing literature on concrete measurement of the concept of responsibility in medical trainees (MTs). This questionnaire measures responsibility in both undergraduate and graduate MTs during clinical trainings.

Method

Conceptual framework

The items of the questionnaire were obtained by literature review and mainly from a PhD dissertation (12). The latter was a qualitative research that used “grounded theory methodology” to provide a theoretical explanation for the phenomenon of responsibility in MTs and the involved processes in the clinical settings of Shiraz University of Medical Sciences (SUMS). Consequently, three categories were extracted that could explain how MTs took on and met their educational responsibilities: 1) Try to find acceptance towards expectations; 2) Try to be committed to meeting expectations (including three subthemes of “commitment to others”, “commitment to self-improvement”, and “commitment to fulfill academic duties”); and 3) Try to cope with unacceptable expectations (including two subthemes of “adopting effective strategies” and “adopting non-effective strategies”).

Based on the above-mentioned qualitative research and considering the objective of this study, responsibility in MTs may be defined as their “ability to gain acceptance toward existing educational expectations, and their desire and attempt to meet those expectations, or their reaction to unacceptable ones using effective or non-effective strategies”. Eventually, the items of the instrument were generated based on the emergent themes and subthemes of the above-mentioned qualitative study.

Development of the questionnaire

A total of 55 initial questions were designed and collected as a draft questionnaire (Persian version). In order to ensure face and content validity, these items were reviewed for syntax, appropriateness (13), difficulty, relevancy and ambiguity by a number of experts including clinical attending physicians and professional nurses as well as MTs. Participating trainees were representatives of all groups of MTs including undergraduates and medical residents, and minor modifications were made to the layout and wording accordingly.

In order to test the content validity of this initial version, each item was rated by at least ten experts and MTs to calculate content validity ratio (CVR) and content validity index (CVI). Since the questionnaire had to be short and easy to complete, the purpose was to determine whether each item should be retained or rejected (14, 15). Finally, after a cautious rejection of 9 items, a 46-item questionnaire was developed for further validation. Lack of conformity between the items and their corresponding category and content overlap were two main causes for item rejection. Response to items was based on a 5-point Likert scale ranging from strongly agree to strongly disagree. Additionally, in the demographic section of the instrument, questions with regard to age, gender, educational level, monthly household income, marital status and total grade point average of undergraduate education were added.

Study population

The questionnaire was administered to undergraduate MTs in their 5th year (juniors or medical students), 6th year (medical externs) and 7th year (seniors or medical interns), as well as graduate MTs (residents) in various specialties.

MTs were recruited across a wide variety of clinical wards in three educational hospitals of SUMS using stratified sampling. In most wards, the general population of MTs consisted of a variety of undergraduate and graduate MTs with different educational levels. In all cases, the questionnaires were administered directly by one of the authors to participants and then collected back. In addition to the questionnaire, each participant received a set of instructions explaining the purpose of the study, their freedom to participate and a confirmation of confidentiality.

Statistical analysis

Exploratory factor analysis and then confirmatory factor analysis were performed using varimax rotation to identify distinct domains in the questionnaire. In all items, higher score (score 5 in the Likert scale) represents higher responsibility. The internal reliability was estimated overall and for each domain using Cronbach’s alpha coefficient. Pearson correlation coefficients were calculated between each domain of the questionnaire to assess redundancy and independence. The concurrent and divergent validity of the instrument were examined by calculating correlation coefficients between the domain and composite scores of the instrument. All analyses were performed using IBM SPSS Statistics version 21. All statistical testing was also two-sided, and P ≤ 0.05 was considered to be statistically significant.

Results

Participant characteristics

Table 1 demonstrates MTs’ demographic characteristics (n = 237) stratified by their educational level. Since all trainees were either single or married, we have demonstrated here only two options for marital status. In addition, monthly household income has been exchanged from Rials (the currency of Iran) to dollars for international comprehension. Of the 237 participants that completed the questionnaires, 195 (82 %) rated all 46 questions.

Table 1.

Demographic characteristics of participating MTs

Characteristic	Stratum	Educational Level
		Student (n = 72)	Extern (n = 64)	Intern (n = 32)	Resident (n = 69)
Gender (%)	Male	40	46.9	40.6	46.4
	Female	60	53.1	59.4	53.6
Mean age (years)		23.2	23.7	24.7	32.8
Total grade point average (SD)		16.2 (1.3)	16.5 (1.0)	16.4 (1.0)	16.6 (0.8)
Marital status (%)	Single	78.6	85.9	75	38.8
	Married	21.4	14.1	25	61.2
Monthly household income[b] (%)	< 340	33.9	32	21.8	25.8
	340 – 680	37.1	41.5	37.5	39.4
	680 – 1000	12.9	17	18.8	9.1
	1000 – 1340	3.2	5.7	3.1	3
	> 1340	12.9	3.8	18.8	22.7

SD= standard deviation;

in dollars.

Reliability

To assess reliability, the internal consistency of the four domains (see validity and the factor loading results) and the composite score for the 237 participants were determined (Table 2). As can be seen, the calculated internal consistency for the composite score and all domains were high with Cronbach’s alpha > 0.70. Test-retest reliability was conducted after a two-week interval using 35 MTs. The results were good with intra-class correlation coefficients all above 0.80. Moreover, internal consistency of the split-half coefficients was computed using a two-way fixed model. Cronbach’s alpha coefficient was 0.84 for part one and 0.79 for part two, and the Guttman split-half coefficient and intra-class correlation coefficient were 0.88 and 0.90 respectively.

Table 2.

Characteristics of the instrument composite and domain scores

Responsibility measure	No. of items	Cronbach’s alpha	Test-retest
CtFE domain	15	0.75	0.71
SCvs.DC domain	11	0.75	0.75
ItBEiME domain	10	0.74	0.75
AoE domain	5	0.76	0.72
Composite	41	0.90	0.90

CtFE= commitment to fulfill expectations; SCvs.DC= self-centeredness vs. duty-centeredness; ItBEiME= inclination to be engaged in meeting expectations; AoE= acceptance of expectations.

Validity

Face and content validity were assessed by our expert panel including clinical attending physicians and nurses as well as a number of undergraduates and medical residents. After the process of revision, a 46-item draft questionnaire was developed for factor analysis. In order to ensure construct validity, the questionnaire was administered to 237 MTs (Table 1). Responses were used to assess interpretability, internal consistency, and factor loading. After exploratory factor analysis and by using component matrices and a scree plot (Fig 1), it was decided that four or five components had to be retained. Following a number of successive confirmatory factor analyses, the 41-item instrument was finalized with four domains. In this way, 5 items with low (one item) or negative (two items) factor loadings, and items lacking consistency with the extracted domains were deleted.

Fig 1

The scree plot. The curve reaches a fairly stable plateau after five factors

The extracted domains were as follows:

“Commitment to fulfill expectations (CtFE)” with 15 items and extracted eigenvalue of 9.77

“Self-centeredness vs. duty-centeredness (SCvs.DC)” with 11 items and extracted eigenvalue of 3.76

“Inclination to be engaged in meeting expectations (ItBEiME)” with 10 items and extracted eigenvalue of 2.15

“Acceptance of expectations (AoE)” with 5 items and extracted eigenvalue of 1.84

These components could explain 42.75 % of the total variance of MTs’ responsibility in clinical settings. The results of the principal component analysis (PCA) are demonstrated in Table 3. In this table, all calculated factor loadings with values larger than 0.3 are shown in addition to rotated eigenvalues and the percentage of variance by each significant extracted factor. The instrument is available in the appendix at the end of the article.

Table 3

Results of factor loading using principal component analysis (PCA) as extraction method

	Factors
	CtFE		SCvs.DC		ItBEiME		AoE
	CIb	FLc	CI	FL	CI	FL	CI	FL
	Qd1	0.74	Q16	0.68	Q27	0.64	Q37	0.71
	Q2	0.68	Q17	0.65	Q28	0.60	Q38	0.60
	Q3	0.67	Q18	0.63	Q29	0.52	Q39	0.56
	Q4	0.65	Q19	0.63	Q30	0.51	Q40	0.47
	Q5	0.64	Q20	0.55	Q31	0.50	Q41	0.36
	Q6	0.59	Q21	0.55	Q32	0.48
	Q7	0.57	Q22	0.54	Q33	0.43
	Q8	0.55	Q23	0.54	Q34	0.41
	Q9	0.54	Q24	0.50	Q35	0.40
	Q10	0.52	Q25	0.44	Q36	0.30
	Q11	0.52	Q26	0.42
	Q12	0.51
	Q13	0.50
	Q14	0.50
	Q15	0.49
Rotated eigenvalue	6.236		5.119		3.760		2.412
% of variance	15.210		12.485		9.172		5.882
Cumulative %	15.210		27.696		36.867		42.749

CtFE= commitment to fulfill expectations; SCvs.DC= self-centeredness vs. duty-centeredness; ItBEiME= inclination to be engaged in meeting expectations; AoE=acceptance of expectations; C= Content of the Item (see the appendix); FL: factor loadings; Q: question.

Table 4 illustrates the intra-scale correlations between the instrument domains and the composite score. As can be seen, all inter-correlation measures between domains are lower than that of each correlation with the composite score. Almost all intra-scale correlations were low with r ≤ 0.48, suggesting that the domains are measuring unique constructs.

Table 4.

Intra-scale correlations between the instrument domains and the composite score

Responsibility measure	Responsibility measure
	Composite	AoE	CtFE	ItBEiME	SCvs.DC
Composite	1.00
AoE domain	0.53	1.00
CtFE domain	0.79	0.22	1.00
ItBEiME domain	0.82	0.32	0.48	1.00
SCvs.DC domain	0.82	0.43	0.42	0.63	1.00

CtFE= commitment to fulfill expectations; SCvs.DC= self-centeredness vs. duty-centeredness; ItBEiME= inclination to be engaged in meeting expectations; AoE= acceptance of expectations;

All correlations were significant at the 0.01 level

Discussion

In this study, we developed a questionnaire for practical measuring of MTs’ responsibility in clinical settings. The main source for generating the items of this instrument was a published grounded theory research (GTR) (12). As demonstrated by factor loading, the instrument assesses MTs’ responsibility with respect to four domains. Overall, there was a suitable correspondence between the items of this questionnaire and the emergent categories and sub-categories of the GTR. This correspondence will be discussed in more detail below.

There was not a close correspondence between items of the component “commitment to fulfill expectations (CtFE domain)” and its corresponding category “try to be committed to meeting expectations” in the GTR. As table 5 demonstrates, questions 1, 2, 4, 5, 10 and 14 in this domain have been generated based on the dissimilar category “try to find acceptance toward expectations” in the GTR. Originally, these items aimed to assess how “acceptance parameters” of motivation, hope, attitude, etc. might influence MTs’ responsibility in practice. Moreover, questions 3 and 6 were generated to assess how effectively MTs might deal with “unacceptable duties” in this domain. While this distribution of items seemed unusual at the beginning, a more thorough contemplation revealed that despite their original purpose, these items could also measure trainees’ responsibility. In fact, these items would measure trainees’ level of commitment, especially in situations where they might find their personal benefit or convenience in conflict with their duties. Therefore, contrary to our expectation, factor analysis revealed that MTs responded to these questions in the same manner they responded to the questions based upon the category “try to be committed to meeting expectations”.

Table 5.

Corresponding items of the “responsibility questionnaire” generated based upon emergent categories and sub-categories of a published PhD dissertation

	Emergent categories of the grounded theory research (GTR)
Responsibility measure/Corresponding items	“Try to find acceptance towards expectations”	“Try to be committed to meeting expectations”	“Try to cope with unacceptable expectations”
CtFE domain/Q1–Q15	Q1, Q2, Q4, Q5, Q10, Q14	Q7, Q8, Q9, Q11, Q12, Q13, Q15	Q3, Q6
SCvs.DC domain/Q16–Q26	Q17, Q19, Q21, Q22	---	Q16, Q18, Q20, Q23–26
ItBEiME domain/Q27–Q36	---	Q27–Q36	---
AoE domain/Q37–Q41	Q37–Q41	---	---

CtFE= commitment to fulfill expectations; SCvs.DC= self-centeredness vs. duty-centeredness; ItBEiME= inclination to be engaged in meeting expectations; AoE= acceptance of expectations.

As demonstrated by factor loading, the component of “self-centeredness vs. duty-centeredness” (SCvs.DC domain) includes items that measure MTs’ real approach towards fulfilling duties, especially when duties could potentially conflict with their convenience and benefit. Thus, lots of items generated based on the third category of the GTR “try to cope with unacceptable expectations” would fall within this domain. In order to facilitate understanding this component and its general meaning, the domain had to be named differently from its corresponding category. However, like the previous component, items 17, 19, 21 and 22, which had been developed according to the category “try to find acceptance towards expectations”, would be placed in this domain. This unlikely distribution revealed that although these components primarily aimed to assess the role of other “acceptance criteria” pertaining to responsibility in MTs, they were also indexes of self-centeredness vs. duty-centeredness. As the collection of questions indicates, this domain could assess MTs’ priorities regarding qualitative fulfillment of duties over their comfort and benefit.

In the case of the extracted component “inclination to be engaged in meeting expectations” (ItBEiME), all items were consistent with the second category of the GTR “try to be committed to meeting expectations”. According to factor loading, items of this domain would measure MTs’ inclination and desire for engaging in fulfillment of academic duties. On this basis, higher responsibility was usually followed by a higher inclination to be involved in meeting expectations. Finally, all five items of the fourth component “acceptance of expectations” (AoE) also corresponded to the category “try to find acceptance towards expectations” in the GTR. This domain measures which criteria of duties or expectations are personally evaluated by MTs to be accepted and then fulfilled.

As mentioned before, the literature on responsibility as a quantitative measure is rather limited (9), especially in the fields of medical education and medical ethics. Moreover, results of published works often cannot be applied to all countries alike due to social and cultural differences unless adaptability measures are implemented. There are also certain nonspecific instruments that have investigated the concept of responsibility mainly for other purposes (16, 17) or rather broadly and in order to evaluate concepts such as “professionalism” (10, 11). Additionally, these instruments are often employed by third parties. Therefore, we believe the present valid and reliable questionnaire could specifically contribute to the existing literature on the subject and be utilized as a self-reporting instrument for assessing Iranian MTs’ responsibility in practice.

The most important limitation of the present questionnaire may be that in general assessment of responsibility in a group of MTs, only the more responsible ones complete and return the questionnaires to researchers. This is especially true when forms are sent electronically or by mail, in which case the results would most likely be false or unreliable. It is therefore proposed that in these situations the instrument be delivered and then collected back in face-to-face appointments. The questionnaire can also be administered to MTs as a mandatory assignment in one of their formal courses such as medical ethics, or as a required twelve/six-monthly evaluation form for educational purposes.

Conclusion

In this study, we aimed to develop an instrument for practical evaluation of responsibility in medical trainees. The questionnaire was prospectively validated in a diverse population of both undergraduate and graduate MTs all recruited from educational settings of SUMS. The instrument was intended as a generic survey to supplement the literature on practical assessment of responsibility in MTs within clinical settings. This instrument is intended to measure trainees’ beliefs, attitudes and behaviors with regard to the concept of responsibility in practice. We believe that this questionnaire can be used to provide highly beneficial information for medical education and ethical development purposes.