Formative assessment in postgraduate medical education - Perceptions of students and teachers.

CONTEXT: One of the most important factors of medical education that can revolutionize the learning process in postgraduate students (PGs) is assessment for learning by means of formative assessment (FA). FA is directed at steering and fostering learning of the students by providing feedback to the learner. However, though theoretically well suited to postgraduate training, evidence are emerging that engaging stakeholders in FA in daily clinical practice is quite complex. AIMS: To explore perceptions of PGs and teachers (Ts) about factors that determines active engagement in FA. SUBJECTS AND METHODS: It was a descriptive qualitative study involving focus group discussions (FGDs) with PGs and Ts from Departments of Pediatrics and Orthopedics. FGDs data were processed through points/remarks, data reduction, data display, coding followed by theme generation for content analysis.
RESULTS: Four higher order themes emerged: Harsh reality of present summative assessment structure, individual perspectives on feedback, supportiveness of the learning environment, and the credibility of feedback and/or feedback giver.
CONCLUSIONS: Engaging in FA with a genuine impact on learning is complex and quite a challenge to both students and Ts. Increased acceptability along with the effective implementation of FA structure, individual perspectives on feedback, a supportive learning environment and credibility of feedback are all important in this process. Every one of these should be taken into account when the utility of FA in postgraduate medical training is evaluated.

INTRODUCTION

It promotes learning by providing feedback to the learner[1] resulting in an efficient postgraduate training along with transparent and credible assessment. Consequentially, the role of purely summative assessment (SA), at the end of the training period is waning and FA, is gaining ground, thereby facilitating its implementation in many postgraduate training curricula worldwide.[2] However, whereas in SA, validity and reliability are seen as dominant determinants of utility, in FA, utility, defined as learning that results from the assessment process, is much more dependent on how stakeholders that is, PGs and Ts employ the instrument in practice.[3]

So far, few studies have addressed the issue of the effect of FA on doctors learning and performance.[4] Moreover, it is becoming increasingly clear that, even though FA is theoretically well suited to postgraduate medical training, engaging both PGs and Ts in meaningful FA is quite complex.[5] We set out to qualitatively explore PGs and Ts perceptions on what factors determine active engagement in FA in postgraduate medical education in India. Both PGs and Ts views were sought because engaging in meaningful FA requires efforts from both feedback giver and feedback recipient.[6] Our objective has been to formulate the themes generated by focus group discussion (FGD) regarding perceptions of FA.

SUBJECTS AND METHODS

This was a descriptive qualitative study conducted over 6 months in a rural medical college in a hilly terrain, where postgraduate training comprises a full-time training program lasting 3 years. Training consists of both in- and out-patients services supervised by faculty. Although this study was exempted from ethical approval, considerable effort was taken to protect the interest of participants; participants were informed about the voluntary nature of participating in the study, about the aim of the study and that data would be analyzed anonymously. Twenty-three PGs of all 3 academic years and 14 Ts were enrolled from Departments of Pediatrics and Orthopedics of our institute. To create an optimally safe environment, FGDs with PGs and Ts were held separately. Ten focus group sessions were held, four with Ts and six with PGs. FGDs for PGs of both the departments were held separately. Each group of FGD consisted of PGs of the single academic session from the same department. The group sizes of the FGDs with PGs and Ts ranged from 3 to 5 participants.

A FGD approach rather than in-depth interviews or questionnaires was preferred, as this would provide more information, make interconnections visible and perhaps even trigger the formulation of new ideas or theories on the subject. At the start of each FGD session, all participants were informed on the purpose of the study and guaranteed full confidentiality. For FGD in Department of Pediatrics, the moderator was from the same department, and the same moderator moderated the FGD in Department of Orthopedics too. The moderator was not involved directly or indirectly in the assessment process. Moderator initiated the discussion using a predefined list of nine questions for guidance [Table 1]. The first questions were meant to elucidate the current assessment structure, followed by questions on expectations and needs regarding assessment for learning. FGDs data were analyzed using QDA Minor 4 Lite and was processed through points/remarks, data reduction, data display, coding followed by theme generation for content analysis.

Table 1

Questions-focus group PGs/focus group Ts

RESULTS

After content analysis, four higher order themes emerged, explaining the level of active engagement in FA by PGs and Ts-harsh reality of present SA structure, individual perspectives on feedback, and supportiveness of the learning environment and the credibility of feedback and/or feedback giver [Table 2]. We have presented the results, interlaced with distinctive quotes, organized according to these themes. Quotes have been coded as follows: PG denotes postgraduate student, T denotes teacher, 1–3 refer to training year, and P and O refer to Pediatric and Orthopedic Departments.

Table 2

Key themes of FGDs

Harsh reality of present summative assessment structure

In spite of guidelines, still there is thinking of old era that being in good books of someone is the only key to pass the examination. Fear of failure and untoward consequences instead of zeal for knowledge and constant improvement is the driving force for a graduate or PG. The ground reality is that actively engaging in seeking and obtaining feedback does not prevail in the system. The likely causes are lack of acceptability, poor implementation, and poor sensitization of faculty toward FA. Some of the comments read:

Negative impression is always fatal for students (PG2-O)

I do not think that my supervisor is well informed on how I′m progressing in my training (PG3-P).

Lack of acceptability

Lack of acceptability of FA is prevailing in the system due to the centralized decision-making structure in the departments.

Poor implementation

FA is difficult to implement due to time constrains, scarcity of faculty and inbuilt inertia to a possible change by coming out of their comfort zone of SA structure.

Lack of faculty sensitization

Lack of faculty sensitization is one of the reasons for inadequate assessment as there are shortcomings in providing hands-on training workshops on FA.

When Ts were asked regarding present assessment structure mostly referred to specific guidelines and little was revealed on their individual perspective on it.

Individual perspectives on feedback

During the FGDs, it became clear that the individual perspectives on feedback were largely determined by ownership and goal orientation.

Ownership

Ownership can best be described as the belief that making the most of one's training period is a personal responsibility.

I strongly believe that it is my training, and thus my responsibility to ask for feedback (PG3-O).

Typically, feelings of ownership grew with years of experience; PGs in the beginning of specialist training were initially more focused on learning how to cope with the new working environment and were less actively involved in their own training pathway.

In the beginning, you tend to have a more consuming attitude (PG1-P).

However, once training progressed, their competence grew, and transition to progressive independence was set in motion. At the same time, growing awareness of their personal responsibility to make the most of training resulted in more self-reflection and active search for learning opportunities and feedback.

I do feel that my ability to self-reflect has grown during my training. It has happened in my 2nd year and I′m more aware that it is important to select appropriate learning goals and that I′m actively engaged in getting there (PG3-P).

Ts were of the opinion that how important it is to them that PGs show ownership right from the start of training.

You would expect them to study spontaneously, just being involved in patient care should provide enough incentives to start looking things up (T-O).

Goal orientation

Both PGs and Ts expressed varied assessment and feedback preferences. These seemed to be determined by their achievement goal orientation (performance- or mastery-orientated). Performance orientated PGs and Ts preferred SA in which competence is assessed against a predefined standard, and in which failing has clear consequences.

I would prefer good old knowledge exams: Clear study materials, clear pass/fail standards and clear consequences. It helps me to start studying and in this way I know once I have mastered a subject (PG3-P).

I would prefer rigorous assessment system with clear consequences. In my experience PGs study most when they get targeted assignments followed by assessment. They like it when they notice that they have actually mastered a subject (T-O).

Mastery-orientated PGs and Ts are more predisposed toward self-assessment and/or FA. They believe that learning is stimulated by feedback, self-reflection, coming back on issues and personal coaching.

To me, all feedback is valuable; I think you can use all information one way or another on your way to medical expertise (PG2-P).

To do postgraduation, one has to be very well motivated. I sincerely wonder whether more SA will increase performance for this group of motivated people (T-P).

Learning environment

The perception of the learning environment was another important feature when active engagement in assessment for learning was discussed. The term “learning environment” in this study means encouraging Ts, clear assessment procedures and supportive learning and work culture.

Committed Ts

PGs frequently stressed the importance of committed supervisors, who are interested in teaching as well as in developing their teaching skills.

You can notice which supervisors are really teaching-minded: They tend to teach the T courses, prepare themselves and give structured feedback (PG2-O).

When Ts were involved in-training assessment, mentoring skills became another precious asset. It transpired that the ability to approach PGs with a genuine interest in their long-time progression, both in their career and private lives was sorely missed by the PGs.

I feel that T should be interested in his PGs and should be well informed on their progress and the competencies that they have achieved (PG2-P).

Clear standards and consequences

Furthermore, both PGs and Ts expressed the need for clear standards and clarity on the consequences of substandard performance. Several PGs mentioned devaluation and/or disregard of feedback as a result of the absence of these.

If you are being assessed with the purpose to stimulate learning and the result of the assessment is without consequences, the impact will be disappointing (PG2-O).

Acknowledgment and appreciation

Acknowledgment of the importance of clinical teaching, dedicated teaching time for PGs and Ts was stressed as an important enabler of FA.

And you need dedicated staff and time, because in present setup, you have to go and do something else before you even had the opportunity to give feedback (T-P).

Appreciation/reward gives you stimulus to excel (PG2-O).

Credibility of feedback

Not all feedback automatically translated into learning. For this, the credibility of both feedback content and feedback giver were of paramount importance. When these were not credible enough, feedback was often rejected and consequentially did not result in the intended learning. The credibility of feedback content depended on issues like authenticity (does the feedback relate to a representative, directly observed doctor–patient encounter), and whether feedback can be judged against a clear, well accepted standard (e.g. guideline, latest research).

There is no problem to get some advice of a T on a patient problem; however, usually I get one, without him seeing the patient. I would like to get some structured feedback after being observed with the patient (PG1-O).

Apart from feedback content, personality traits and feedback strategies were other important determinants of the credibility of Ts. Feedback from a T who is perceived as a role model, well respected, enthusiastic passionate about his subject, encouraging to PGs was valued most. Especially, if this person was also able to provide structure during feedback sessions and remembered when and how to come back on issues.

Especially someone whom I personally regard as an exemplary doctor. If I see he/she is a professional in a way that I would like to be in the future. That's the person from whom I prefer to get feedback (PG3-P).

DISCUSSION

After FGD of PGs and Ts on perception of FA the themes emerged were, harsh reality of present assessment structure, individual perspectives on feedback of trainees and supervisors, a supportive learning environment and credible feedback.

Assessment has been taken rather casually in the present postgraduate medical education program. Ignoring educational principles while assessing students, merely because it results in more work, seriously compromises the utility and sanctity of assessment.[7] Faculty development programs and hands-on training workshops of the faculty on FA can bring transparency and credibility in the assessment of PGs.

Individual perspective on feedback

The individual perspective on feedback in this study is determined by both ownership and achievement goal orientation. Ownership is an internal drive to make the most of postgraduate training and act accordingly.[8] In our study, ownership of PGs plays a central role in both the motivation of a PGs to ask for feedback and the Ts willingness to start giving feedback.[910] Progressive independence in the training years increases the awareness of personal responsibility along with an active approach of PGs to their learning pathway. However, supervisors expect ownership right from the start of training. It is, therefore, essential that the importance of ownership is explicitly discussed with junior PGs right from the start of training.

Ts revealed little information about their motivation to get actively engaged in FA. Available literature suggests that both a feeling of responsibility for the training of future doctors and beliefs arising from the Ts achievement goal orientation play a role.[11] This is in accordance with our finding that goal orientation is a main determinant of the assessment preference of both PGs and Ts. Whereas people with a mastery goal orientation tend to focus on acquiring and developing competence, taking all feedback as an opportunity to improve their learning, the focus of people with a performance-orientation tends to be on demonstrating one's competence and outperforming others; the latter usually valuing the clear standards and consequences of SA.[12] However, even though goal orientation itself may be difficult to modulate, awareness of both PGs and Ts of their personal achievement goal orientation should be stimulated as this will aid in customizing assessment and preventing frustrations on both sides.

Supportive learning environment

Supportive learning environment acts as an important facilitator in engaging in FA. The need for dedicated assessment moments and teaching time is must because time pressures induce a surface learning approach.[13]

Both PGs and Ts expressed a need for clear standards of performance in the combination with clear consequences for substandard performance. Here, the complex situation is that a doctor on the everyday work floor requires performances which are multi-dimensional, making it very difficult to explicate what good performance is, with the risk of getting lost in detailed, unrealistic lists full of desirable attitudes and skills.[1415] Meanwhile, there is a need for a continuing discussion on the minimal level of performance of a PG along with professional values and good clinical practice.

Credibility of feedback and/or feedback giver

Our participants particularly emphasize the importance of credible feedback and feedback givers as perceived by PGs and the importance of authentic assessment as perceived by Ts. However, as Ende already points out in 1983: Without feedback young doctors tend to develop a system of internal validation that excludes validation from external sources.[16] There is evidence that physicians are poor at self-assessment, making it imperative to develop a system of external evaluation of learner performance that participants trust and use.[17] FA is more than just giving feedback on a single occasion. For FA to exert an effect on learning an action plan follow-up and an opportunity to demonstrate improvement should be part of the process.[18]

Combining the perspectives of both trainees and supervisors made it possible to explore the roles of both stakeholders in the process. Using the focus group technique has provided us with rich quality data and made the complexity and multi-dimensionality of FA in daily clinical practice evident. The limitations of the study could be participants with strong views who share more information than others. This bias has been minimised by questions, answers, and reporting but still moderator bias to an extent is unavoidable.

CONCLUSIONS

The educational impact of FA is multidimensional and actively engaging in assessment for learning is quite a challenge to both PGs and Ts. Honest and fair assessment structure will encourage PGs to excel. Ownership and achievement goal orientation determine active engagement in assessment for learning. Scheduled assessment moments and clear standards, procedures, and consequences facilitate assessment for learning.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.