Anesthesiologists' perception on their residency training.

BACKGROUND: Self-regulated professions such as medicine are characterized by professional commitment to the public they serve and require life-long development of expected skills. There is a paucity of data regarding quality of training during residency in anesthesiology. The objective of this study was to create an instrument to assess the anesthesiologists' perception regarding the quality of their training during medical residency.
METHODS: An electronic questionnaire was sent to 120 anesthesiologists, assuming 15% response rate for worst case scenario, considering a number of 613 potential respondents. The answers to the questionnaire were submitted to psychometric evaluation through internal consistency analysis measured by the Cronbach's alpha coefficient, and factorial analysis by the principal components' method with varimax rotation method with Kaiser normalization. The level of Concordance (C) and Disaccord (D) of each item were compared byz tests (consensus if p < 0.05). A question asking the respondents if they would recommend their training center to a relative or a friend was added to the questionnaire and considered as a secondary outcome.
RESULTS: One hundred and one responses were obtained. The Cronbach's alpha value was 0.86, suggesting good reliability of the questionnaire. The initial analysis including all the 14 items presented on the questionnaire demonstrated that four components obeyed the Kaiser criterion, depicting 68.20% of variance. Consensus was achieved among participants regarding all items of the questionnaire. The medical residency in anesthesiology was recommended by 93% of the participants. Preceptors were considered role-models by 83% of the participants. Acquisition of practical skills was better evaluated in comparison to other areas.
CONCLUSION: The questionnaire effectively characterized the perception of anesthesiologists regarding the quality of their training during medical residency. The information produced by this instrument could provide interesting clues on the quality of residency programs, pointing out areas of education that need more attention.

Introduction

Medical performance results from acquisition of practical skills and theoretical knowledge, combined with appropriate attitudes towards patients and colleagues. Self-regulated employments, including medical workers, are characterized by professional commitment towards the public they serve and requires continuous development of expected skills. A core component in creating an efficient and competent physician is professionalism. According to Irvine, professionalism is centered on three pillars which form the basis for professional independence (autonomy): expertise, ethics, and work. Epstein and Hundert define professional competence as the usual and judicious use of communication, knowledge, technical skills, emotions, values, and reflections on the daily practice for the benefit of individuals and communities assisted.

Learning in Anesthesiology entails the development of skills based on the grounds of theoretical knowledge (cognitive skills) and social interaction (non-technical skills). The main goal is the acquisition of necessary professional skills from three standpoints: theoretical knowledge, technical skills, and social skills. In Brazil, the anesthesiology training curriculum is defined by the Brazilian Society of Anesthesiology (SBA) and the Federal Government, represented by the Ministry of Education National and the Medical Residency Commission. This curriculum is implemented and followed by registered Training Centers in each State of the Federation. Training during medical residency is generally supervised by anesthesiologists with the Título Superior de Anestesiologia (TSA – Certificate of Anesthesiology) and hospital staff members (anesthesiologists), neither of which are (usually) formally trained to educate.

In our country, there are few training programs for preceptors. Also, there is a scarcity of data on the quality of teaching in residency programs. We believe that the anesthesiologists’ perception on the quality of teaching during medical residency could be compared to the scores obtained with the National Certification Examination. This comparison could provide interesting information on the quality of residency programs, pointing out areas of education that need more attention.

The objective of this study was to create an instrument in order to assess anesthesiologists’ perception regarding the quality of their training during medical residency. We built a questionnaire covering three distinct areas: practical, theoretical, and non-cognitive skills. It is hypothesized that a psychometric instrument in the form of a questionnaire will be able to measure the perception of anesthesiologists regarding the training quality during medical residency.

Methods

Ethical approval for this study was provided by the local Ethical Committee on 19 December 2017. The study design was cross sectional. Data gathering was conducted prospectively from October 2018 to December 2018. The invitation to participate in the study was sent through e-mail containing the link for the electronic questionnaire, to anesthesiologists duly registered in the Regional Anesthesiology Society (Sociedade de Anestesiologia do Rio Grande do Sul – SARGS). An electronic version of informed consent was added to the questionnaire. All those who agreed to provide informed consent were included in the final statistical analysis. The respondents’ anonymity was assured.

A list containing email addresses of all anesthesiologists duly registered in the Regional Anesthesiology Society was used. The list was granted by the Regional Society after ethical approval. The number of registered members was 613 at the time of study inception. An electronic random sequence of 613 numbers was created (MS Excel®, Microsoft Corp., Bellvue, WA), and each member of the list was assigned with a number in ascending order. We assumed a response rate of 15% (worst case scenario) out of 613 potential responders. Therefore, we have sent the invitation to participate to 120 members of the list. Unanswered emails were successively replaced, following the numerical order, until the pre-established number was reached. Two reminder messages were sent to all participants in a three-week interval. Data gathering was interrupted three weeks after the last reminding message.

Considering the minimum acceptable agreement or disagreement rate for defining consensus on each item of the questionnaire (P) is equal to 70% of respondents and assuming an error (e) equal to 5%, we have estimated the number of responses to the questionnaire equal to 84 (n = PQ/e2), with P – Q = 1.

Questionnaire was initially submitted to content and face validity.7, 8 The content validity was conducted via a two-step process: item development (reviewing the literature for comprehensiveness of construct), and item judgment accomplished by a panel of experts composed by three anesthesiologists.

Face validity was conducted as a pretest. Twenty anesthesiologists agreed to respond the questionnaire (face-to-face interview) and were allowed to detect problems, such as defective questions expressed as ambiguity and accuracy of wording and meaning, and to comment upon issues regarding length, question format, amount of space provided for replies and question sequence.

The questionnaire (Table 1) was built on Google-forms. It contained 14 items measured in a 5-point Likert scale (5 = strongly agree; 1 = strongly disagree). Places for demographic data were also included. The Cronbach alpha coefficient was used for reliability analysis of responses, followed by factor analysis with the principal components’ method. The consensus of the participants on each item was defined when value of p < 0.05 (z-test for proportions). Answers on Likert levels 4 and 5 were grouped as concordant and Grade 1 and 2 as discordant.

Table 1

Questionnaire on the Anesthesiologists’ Perception Regarding Their Training During Anthesiology Residency.

Please chose the descriptor that best expresses your perception on applicability of the medical residency learning in your daily practice:

1. Applicability basic sciences (anatomy, physiology, and pharmacology).

2. Applicability of ethical concepts.

3. Applicability of communication and interpersonal relationship skills.

Please chose the descriptor that best expresses your perception of your learning during medical residency, in the following areas of knowledge:

4. Learning of basic sciences

5. Theoretical teaching in anesthesiology

6. Learning of medicais procedures

7. Learning of non-cognitive skills (ethics, communication, and interpersonal relationship)

8. Learning through Evidence Based Medicine

9. Learning through case discussions and Problem-Based Learning (PBL)

Please chose the descriptor that best expresses your perception on the preceptors in your training center.

10. Were preceptors continuously supervising your training?

11. Did preceptors demonstrate theoretical knowledge and mastery of medical procedures?

12. Did preceptors demonstrate ethical behavior and care towards patients?

13. Did preceptors foster continuous progression of learning (clinical reasoning, critical thinking, and theoretical knowledge)?

Please chose the descriptor that best expresses your assessment on the quality of your residential training towards preparation for working life:

14. What is your assessment on the quality of your training during residency in preparation for professional life?

15. Would you recommend your anesthesiology training center to a relative or friend?

A last question asking the respondents if they would recommend their training center to a relative or a friend was added to the questionnaire and analyzed as a secondary outcome.

The answers to the questionnaire were submitted to psychometric evaluation through internal consistency analysis measured by the Cronbach alpha coefficient, and factorial analysis with the principal component’s method and varimax rotation method with Kaiser normalization. The responses to the questionnaire were exported using Microsoft Excel® and analyzed using SPSS® – version 23.

Results

Sixty-one responses (10%) were obtained with the first email. The answering rate increased to 101 (16%) after a second electronic reminder. Demographic data are depicted in Table 2. Results are presented as frequency or percentage.

Table 2

Questionnaire respondents’ (anesthesiologists) demographic data.

Gender (%, M/F)	56.4/43.6
Age in years – median (interquartile range)	42 (35–50)
Total time after completing anesthesiology residency (in years) – median (interquartile range)	14 (6–22)
Would you recommend your anesthesiology training center to a relative or friend (%, Y/N)	93/7
Supervisors considered as role models (%)	83

The Cronbach’s alpha presented a value of 0.86, suggesting good reliability of the questionnaire. Table 3 shows the 14 items included in the questionnaire, used to measure respondent’s perception on the quality of their medical residency training.

Table 3

Questionnaire corrected item-total correlation and internal consistency (Cronbach’s alpha) if the item is deleted.

	Corrected item – total correction	Cronbach’s alpha if item deleted
Applicability of residency learning in professional life
1. Knowledge of basic subjects (anatomy, physiology, and pharmacology)	0.25	0.87
2. Ethical aspects	0.29	0.87
3. Communication and interpersonal skills	0.24	0.87
Experience acquired during residency in anesthesiology
4. Knowledge of basic sciences and their application to anesthesiology	0.53	0.86
5. Theoretical teaching in anesthesiology	0.75	0.84
6. Learning of medical procedures	0.56	0.86
7. Learning of non-cognitive skills (ethical aspects, communication, and interpersonal relationship)	0.54	0.86
8. Learning of Evidence Based Medicine	0.71	0.84
9. Learning through case discussions and Problem-Based Learning (PBL)	0.52	0.86
Preceptor perception during your residency in anesthesiology
10. Were your preceptors continuously supervising your training?	0.57	0.86
11. Did the preceptors demonstrate adequate theoretical knowledge and mastery of medical procedures?	0.50	0.86
12. Did the preceptors demonstrate adequate ethical behavior and care towards patients?	0.67	0.85
13. Did preceptors foster continuous progression of learning (clinical reasoning, critical thinking, and theoretical knowledge)?	0.50	0.86
Overall assessment on the quality of your residential training towards preparation for working life
14. Overall assessment on the quality of your residential training towards preparation for professional life?	0.65	0.85

A principal components analysis was conducted on the 14 items of the questionnaire along with orthogonal rotation (varimax) in the sample of 101 participants. Measure of Kaiser-Meyer-Olkin verified the sample adequacy for the analysis (KMO = 0.77) and all KMO values for individual items was greater than 0.5. Bartlett sphericity test (Chi-Square [91] = 653.890, p < 0.001), indicated that the correlations between the items are sufficient for the analysis.

Considering the 14 items presented on the questionnaire, initial analysis showed that four components obeyed the eigenvalue Kaiser criterion greater than 1, depicting 68.20% of variance. Component 1 (preceptorship and residency) had an eigenvalue of 5.39, accounting for 38.53% of variance and comprised items 10 to 14. Component 2 (theoretical-practical experiences) had an eigenvalue of 1.85, being responsible for 13.19% of variance and comprised items 5, 6, 8 and 9. Component 3 (non-cognitive skills) had an eigenvalue of 1.22, accounting for 8.75% of variance and comprised items 3 and 7. Component 4 (ethics and basic areas) had an eigenvalue of 1.08, accounting for 7.76% of variance and comprised items 1 and 2. Consensus was reached among participants regarding all items of the questionnaire. Table 4 presents the questionnaire distributed according to factorial structure previously described.

Table 4

Responses to the Questionnaire formed according to the factorial structure.

Factor/Item	Agree	Disagree
	n (%)	n (%)
Factor 1 – Quality of supervision and anesthesiology residency
10 – Were preceptors continuously supervising your training?	76 (89.4)a	09 (10.6)
11 – Did preceptors demonstrate theoretical knowledge and mastery of medical procedures?	82 (95.3)a	04 (4.7)
12 – Did preceptors demonstrate ethical behavior and care towards patients?	78 (85.7)a	13 (14.3)
13 – Did preceptors foster continuous progression of learning (clinical reasoning, critical thinking, and theoretical knowledge)?	68 (90.7)a	07 (9.3)
Quality training evaluation during anesthesiology towards preparation for professional life
14 – What is your perception on the quality of your residency training towards preparation for professional life?	85 (100)a	0 (0)
Factor 2 – Theoretical and practical skills acquired during anesthesiology residency
4 – Knowledge of basic sciences and their application in anesthesiology	74 (96.1)a	03 (3.9)
5 – Theoretical teaching in anesthesiology	62 (79.5)a	16 (20.5)
6 – Learning of medical procedures	92 (96.8)a	03 (3.2)
8 – Learning of Evidence Based Medicine	73 (85.9)a	12 (14.1)
9 – Learning through case discussions and problem-based learning (PBL)	65 (86.7)a	10 (13.3)
Factor 3 – Applicability and learning of non-cognitive skills
3 – Communication and interpersonal skills	79 (98.8)a	01 (1.3)
7 – Non-cognitive skills learning (ethics, communication and interpersonal relationship)	54 (83.1)a	11 (16.9)
Factor 04 – Applicability of academic training in professional life
1 – Knowledge of basic subjects (anatomy, physiology, and pharmacology)	84 (93.3)a	06 (6,7)
2 – Ethics learning	88 (97.8)a	02 (2.2)

Data presented as n (%).

p-value < 0.05 (z test for proportions).

Discussion

When a questionnaire is built to measure a construct, the answers’ reliability degree must be taken into account. The Cronbach’s alpha score measures the degree to which the result obtained reflects a true result and not only by chance. Exploratory Factor Analysis (EFA) is generally used in initial stages of research, in order to explore the data set. In this phase, we seek to explore the relationship between a set of variables, identifying patterns of correlation. In addition, EFA can be used to create independent or dependent variables that can be used later in regression models. The principal component analysis aims to explain as much as possible the total variance in the variables. The purpose of factor analysis is to explain the covariance or correlations between the variables. It is used to understand which constructs are underlying the data.

The internal consistency analysis returned a Cronbach’s alpha of 0.86, attesting the instrument’s reliability to measure the perception of anesthesiologists about the quality of residency training. The structure of the questionnaire was divided into 4 areas (1 – Preceptorship and residence; 2 – Theoretical-practical experiences; 3 – Non-cognitive skills; 4 – Ethics and basic areas) required for the development of medical professionalism described by Irvine. This structure was confirmed by the factorial analysis of the principal components.

The acquisition of practical skills (Item 6) during residency in anesthesiology was better evaluated in comparison with other areas (clinical skills, basic science, and non-technical skills). A French study, using questionnaires in a population of anesthesiology residents, demonstrated similar results with better response rates for practical skills, compared to theoretical skills. During the training period, residents are exposed to a multitude of patients, several drugs, procedures, and clinical settings before they even complete their training period. In this context, other approaches to knowledge acquisition may have been used, for example, informal learning, which is primarily driven by the need to solve real problems and leads to the acquisition of superficial knowledge with immediate application. Residents must develop practical skills early in their training periods and may eventually divert the focus from theoretical learning. It has already been demonstrated that acquisition of practical skills proficiency is achieved earlier compared to theoretical skills.

The questionnaire used in the present study probably reflects a subjective measure of learning in different areas of knowledge. The gold standard for objective assessment of practical skills acquisition during medical residency consists in a combination of validated checklists and global assessment scales, applied prospectively by a trained observer, during procedures performed on real patients. Such an assessment is not generally used by medical residency programs in our country.

The pattern of questionnaire responses suggests that, in general, participants recognize the importance of learning basic sciences for professional life (Item 1). However, the perception about the learning of basic sciences during medical residency was not well rated (Item 4). In our country, the curricular content on basic sciences taught during anesthesiology residency defined by the Brazilian Society of Anesthesiology (SBA) and includes anatomy, physiology, biochemistry, pharmacology, and physics applied to anesthesiology. Residents are exposed to this content during their first training year. Daily use of physiological and pharmacological concepts is among the most cited reasons when choosing anesthesiology as a career. This may be one possible explanation for the high level of agreement on the importance of learning and the applicability of these concepts in professional life.

Items 4, 5, 8 and 9 are related to the quality of the theoretical education during anesthesiology residency. Those items were worse rated compared to others. Possible explanations to this phenomenon could include: absence of a training program and preceptorship qualification (Items 4, 5, and 9), and anesthesiologists’ preference for a more traditional approach of teaching (Item 9). This finding may reflect the formal curricular structure applied in our country. However, the same study suggests that anesthesiology residents could have a perception that is more favorable learning based on problems resolution (Problem-Based Learning – PBL) than with traditional approach. The implementation of different methodologies – such as PBL – poses a challenge due to difficulties, such as the lack of human resources (supervisors) and their training. Although PBL emphasizes the construction of operational concepts of basic sciences applied to clinical scenarios,15, 16, 17 evidence of learning and clinical outcomes are still lacking. The use of methods based on problem solving and evidence-based medicine may not have been widely adopted during the training process. This could explain, at least in part, the participants’ evaluation on theoretical teaching.

The applicability of particular kind of knowledge, such as ethical aspects and communication skills – whose importance is usually not consensual among professionals –13, 18, 19, 20 was highly rated in this study. However, the applicability of basic science and ethics learning (Items 1 and 2) was considered of greater importance compared to interpersonal relationship and communication (Item 3).

Item 7 (non-technical skills) was rated worse compared to theoretical teaching. The former are to be considered important to development of professionalism.21, 22 Although respondents considered the learning of non-technical skills relevant to clinical practice, it seems to be marginalized in some training centers. This attitude is contrary to the world trend, in which behavioral aspects, often associated with professionalism, are linked to high quality medical practice.

Supervisors’ theoretical knowledge and mastery of medical procedures (Item 11) was better rated compared to all other questions regarding the quality of supervision during anesthesiology residency.

Respondents demonstrated a lower level of agreement regarding items towards ethical issues and patient care (Item 12), supervision during activities (Item 10), and fostering learning progress (Item 13). In spite the results obtained regarding the quality of preceptorship, the participants classified 83% of their preceptors as role models. Part of this phenomenon could be explained by the “halo effect”,23, 24 in which the “raters” judge the “ratees” based on personal impressions about them, instead of analyzing their professional performance.

According to Nunes et al., the in-service training during medical residency should articulate teaching and learning, providing a gold standard for professional training. Medical residents should have the opportunity to deepen their knowledge, improve their attitudes and skills, and develop specific ones. Unfortunately, there are few studies focused on medical residency in our country, as well as training programs for supervisors or consensus on the proper evaluation of residents. A good professional is not necessarily a good supervisor. Developing skills necessary for good medical practice is a challenge, often occurring in a tense and dynamic environment. Learning must not to compromise patient safety, thus it must occur in an adequate manner by properly trained supervisors.

Ninety-three percent of participants would recommend the training center where they performed their residency in anesthesiology to a friend or relative. In general, the training centers were well evaluated by anesthesiologists (Item 14). The numbers exposed above contain an unquestionable subjective component, since objective data are scarce in the literature.

An electronic questionnaire was used to collect the information. However, this kind of research instrument may have some limitations, such as rate of return, representativeness of sampling and external validity of the data. In this study, we used sample size calculation in an attempt to minimize these potential errors. Some circumstances may contribute to lower the rate of return, including anti-pop up and email filters, lack of interest in participating, the size of questionnaire, distrust of confidentiality, among others. As suggested by other authors, limiting the study population to people with a common interest (registered anesthesiologists) may be used in an attempt to reduce the errors mentioned above.

Although the population studied included professionals trained in different units of the Federation, the questionnaire showed an adequate level of consensus in all items. Even so, it seems essential to conduct a larger study with national proportions.

Conclusion

Results presented by this study suggest that the questionnaire was effective in characterize the perception of anesthesiologists regarding the quality of their training during medical residency. The areas related to practical skills were better evaluated than those related to theoretical and non-cognitive skills, suggesting an important imbalance attributed to each of them. Investments in pedagogical training of preceptors and rethinking teaching methods can be an effective way to ascertain residency evaluation will reach excellency.

Authors’ contributions

We would like to thank Dr. Marcos André Lehnemmann Tannhauser (M.D., Anesthesiologist and President of the Sociedade de Anestesiologia do Rio Grande do Sul – SARGS – Brazil), Dr. Patricia Wajnberg Gammermann (M.D., Anesthesiologist of the Hospital de Clinicas de Porto Alegre - HCPA and Scientific Director of Sociedade de Anestesiologia do Rio Grande do Sul – SARGS – Brazil) and Dr. Neil Winawer (M.D., Department of Medicine Professor – Emory University School of Medicine) for their assistance with the study.

Funding statement

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Conflicts of interest

The authors declare no conflicts of interest.