Evaluation of a performance appraisal framework for radiation therapists in planning and simulation.

INTRODUCTION: Constantly evolving technology and techniques within radiation therapy require practitioners to maintain a continuous approach to professional development and training. Systems of performance appraisal and adoption of regular feedback mechanisms are vital to support this development yet frequently lack structure and rely on informal peer support.
METHODS: A Radiation Therapy Performance Appraisal Framework (RT-PAF) for radiation therapists in planning and simulation was developed to define expectations of practice and promote a supportive and objective culture of performance and skills appraisal. Evaluation of the framework was conducted via an anonymous online survey tool. Nine peer reviewers and fourteen recipients provided feedback on its effectiveness and the challenges and limitations of the approach.
RESULTS: Findings from the evaluation were positive and suggested that both groups gained benefit from and expressed a strong interest in embedding the approach more routinely. Respondents identified common challenges related to the limited ability to implement suggested development strategies; this was strongly associated with time and rostering issues.
CONCLUSIONS: This framework successfully defined expectations for practice and provided a fair and objective feedback process that focussed on skills development. It empowered staff to maintain their skills and reach their professional potential. Management support, particularly in regard to provision of protected time was highlighted as critical to the framework's ongoing success. The demonstrated benefits arising in terms of staff satisfaction and development highlight the importance of this commitment to the modern radiation therapy workforce.

Introduction

In an era of advancing technology and evolving practice health practitioners need to update practices continually in order to provide quality healthcare.1,2 Radiation therapists (RTs) are required to maintain skills and knowledge in current practice while adapting to this ongoing change. The challenge of establishing and maintaining consistency of practice in a dynamic radiation therapy department highlights the importance of having defined practice standards and a reliable process for assessing performance. Well-defined standards can provide a clear statement of what is important in performance and be used to motivate staff and provide feedback on performance.3–5 Performance appraisal must also provide adequate support and training for development needs.3,6 A tool that encompasses these features will facilitate effective performance appraisal and promote staff satisfaction and professional growth.

Radiation Therapy Performance Appraisal Framework development

A Radiation Therapy Performance Appraisal Framework (RT-PAF) was developed to promote a culture of performance appraisal that was fair, objective, transparent, based on evidence and focussed on skills development. The focus of this initiative was on radiation therapy planning because of the independent nature of practice and the inherent difficulties in providing feedback on performance. Development began with base-grade skill and was designed to complement existing work processes rather than add to them. Planning practice was structured such that the one RT was responsible for the CT simulation, dosimetry and plan finalisation of each patient allocated to them. The RT-PAF was developed incrementally and implemented on a trial basis so that lessons learned from each stage could inform development of ensuing stages. Skills sets for each area of practice were identified and defined by performance indicators; these were then incorporated into case-based feedback forms.

The appraisal process commenced once the RT had been in planning for a minimum of 2 weeks and was conducted over a 4-week period. The RT was given a series of forms to include with every patient case they completed during this period. These forms were used to evaluate dosimetry, CT/simulation and plan evaluation, and were completed as part of the routine peer review process for each area of practice. Peer reviewers were chosen as experienced in the scope of the review irrespective of relative seniority to reduce the perceived effect of power relationships. At the completion of the review period, a senior RT collated the completed forms to finalise the feedback. Final feedback included additional elements of performance such as professional attitude, time/workload management, commitment to quality and technical communication which were assessed by observation and collaboration with peer reviewers. Documented guidelines recommended that feedback be delivered within 1 week of the review period ending to allow time for the RT to respond and negotiate a development plan while still rostered in planning. The goal was to integrate the RT-PAF into the routine practice and repeat during each planning rotation; however this depended on frequency and length of rotation.

An integral part of the development process was to provide clearly documented protocols and process guidelines to support consistency in practice. Rigour was placed on defining skill levels for each skill set to support a consistency and transparency in the assessment process. Figure1 presents a sample of the RT-PAF with guidelines for rating.

Figure 1

Sample RT-PAF and ratings.

RT-PAF evaluation

The purpose of this study was to evaluate the effectiveness of the RT-PAF as experienced by RTs. Feedback quality was evaluated in terms of timely delivery and whether equal focus was given to identifying strengths and highlighting areas for development. The quality of the process was measured in terms of overall satisfaction, the perception of objectivity and fairness and how effectively the development of skills was enabled. This included identifying any obstacles encountered to meeting these goals and seeking recommendation for improvement.

Methods

Data collection

In March 2011, after a 4-year period of development and implementation, ethics approval, reference number HREC/11/QPAH/047, was granted by the hospital's Human Research Ethics Committee to conduct a formal evaluation of the framework. A retrospective review was designed using survey-based mixed methods evaluation tools. Surveys were designed for two groups: the reviewers and the recipients. Reviewers comprised any RT who had given feedback and recipients were any RT who had received feedback through this process. Reviewers evaluated process quality in terms of the usefulness of guidelines and objectivity of the process, and level of demand imposed by the process. Feedback quality was evaluated based on timeliness, confidence in feedback delivery and the ability to respond to development needs. Recipients evaluated process quality in terms of the clarity of guidelines and transparency of the process. They also assessed feedback quality in terms of timeliness, effectiveness to provide insight to performance, openness of communication and the response to development needs. All respondents were asked to nominate their appointed level and the role(s) in which they gave or received feedback. Likert scale responses were gathered relating to the feedback and process quality with timeliness and identification of strengths and development needs assessed with simple ‘yes’ or ‘no’ responses. Additional short answer questions gathered qualitative data for triangulation with the quantitative findings and identification of themes. The evaluation was conducted over a 6-week period commencing in July 2011. Table1 presents the definitions for terminology that were supplied to support consistent interpretation of terms. Qualitative analysis of participant responses was performed adopting a reflective bracketing approach to minimise the effect of researcher bias.

Table 1

Definition of terms

Transparent	Readily understood; characterised by visibility or accessibility of information especially concerning practices
Clear	Free from obscurity, ambiguity, or unnecessary complexity
Effective	Producing or capable of producing a desired result
Objective	Dealing with facts without distortion of personal feelings, prejudice or interpretation
Subjective	Modified or affected by personal views, experience or interpretation

Participation

Eligibility to participate in the surveys was determined by involvement in the feedback process in any capacity, which made some RTs eligible to complete both. A participant information and consent letter was distributed by email to all RTs and followed up with information sessions on the study. Surveys were facilitated by SurveyMonkey® (Palo Alto, California); a web-based program which allows electronic survey distribution, and confidential information storage and analysis. Individuals were left to determine eligibility and responses were submitted electronically. Participation was voluntary and anonymous and consent was implied by the submission of a completed survey. Responses were required within 14 days of the surveys being distributed, however clinical circumstances delayed response and the deadline was postponed by a further 14 days.

Results

There were 14 respondents to the recipient survey, and 9 to the peer reviewer survey, indicating response rates of 67% and 88% respectively. Figure2 provides an overview of the roles to which feedback was given, with the majority being given to CT simulation. Of the recipients, 100% found the process to be satisfying and effective and only one reviewer found the process unsatisfying. Figure3 shows the obstacles experienced by recipients to ongoing development and Figure4 shows the obstacles identified by reviewers to offering strategies for development. Having insufficient time was reported by 100% of reviewers as an obstacle to delivering timely feedback and by 75% as an obstacle to offering development strategies. This was echoed by recipients where over 85% cited reliance on rostering to provide adequate time in the area to implement development strategies. Additional obstacles to timely feedback identified by the reviewers are shown in Figure5. Several additional themes were extracted from the qualitative data, including: the effectiveness of the RT-PAF, the impact of implementation, and recommendations for ongoing improvement. These were triangulated with the quantitative data and provide themes for the following discussion.

Figure 2

Roles offered feedback by reviewers.

Figure 3

Recipient-identified obstacles to ongoing development.

Figure 4

Assessor-identified obstacles to offering development strategies.

Figure 5

Assessor-identified obstacles to timely feedback delivery.

Discussion

The goal of this evaluation was to determine how effective the RT-PAF was in providing meaningful feedback on performance and guidance for development, how it was experienced by RTs, and how it could be further developed.

RT-PAF effectiveness

The RT-PAF was designed to demonstrate strengths and areas for development to RTs with an emphasis on providing support for professional development.7,8 When conducted well, a performance appraisal process can enhance learning, improve practice and encourage professional development.9 In this study although 100% of recipients had received suggestions for development and 78% had been offered development strategies, only 67% of reviewers felt they had offered strategies for development. This supports Finlay and Maclaren's supposition that the feedback process (and not just the content) could prompt the individual to consider their own ideas for development.9 The experiences of respondents indicate the effectiveness of the RT-PAF with comments including:

This is an excellent tool and should become a part of day to day practice in order to encourage and empower all staff to achieve their potential.

Feedback from RTs who have undergone the RT-PAF has been very positive - invaluable in their professional development and establishes a clear path for their development

Learning what my strengths were and where I could improve…provided some direction for me to focus on.

Others reported that they had found reward in the recognition of their work by their peers. It was clear that the process enabled staff to validate the skills of others and motivate them to continue to develop:

Not only senior RTs but more junior RTs believe in you and your judgment. Also, to demonstrate that one does have the skills to achieve higher levels in the department. It also helps…to set realistic goals and know that with commitment these goals can be achieved.

One respondent discussed the importance of reaching their own objectives and career goals and relished the input of a peer to provide guidance and motivation with this. Through the RT-PAF experience they:

…could see the potential and value in the process. For staff to develop and realise their potential this process should be mandatory.

How it was experienced by RTs

Effective feedback should also be delivered in a timely manner to ensure relevance and currency, and that sufficient time is given in the rostered area for it to be acted on. As one recipient found, this was not always achieved:

More time is needed in the rostered area in order to develop the areas which need to improve.

Recipients indicated that they received feedback within the recommended time frames in 93% of cases, although only 33% of reviewers achieved this. One explanation for this discrepancy could be that the majority of recipients received their feedback from the minority of reviewers who ensured it was delivered on time. Comments from respondents identified the primary obstacles as being:

The lack of resource for the person coordinating the feedback meant that some sessions were rushed as they had their own workload to complete plus the feedback.

…the challenge in completing more work/forms/feedback. Achieving the right balance can be difficult.

being time poor and having minimum time to spend on feedback.

Delaying the feedback creates a risk that the feedback will no longer be relevant, or participating staff will be rostered out of the area before having the opportunity for adequate response.

Feedback that balances affirmation of strength with development plans can build confidence by providing evidence of competence, and contribute to ongoing professional development.10 The RT-PAF was designed to encourage this balance and according to all respondents, this goal was achieved, and was rewarding for those involved. Useful feedback is specific, highlights areas for improvements, identifies strength, and provides guidance for development.11 For it to be effective, however, there must be trust in the process and this is achieved through promoting a culture of performance appraisal that is fair, objective and transparent.12 Objectivity can be enhanced through input from multiple contributors to the feedback13 and comments from reviewers reinforced this. One of the most satisfying aspects of participation, as a reviewer cited, was being in a position to:

remove doubts about the quality of some staff's work and remove subjectivity from assessment, which allows staff to feel more comfortable that what is being said about them is true.

Another assessor felt that the process was:

developed to encourage consistency and transparency, which is imperative to the success of the process.

Some recipients, however, found it to be unsatisfying to:

know that every action you do can be commented on in the report.

and

be under the microscope for a period of time.

When asked to rate their confidence in delivering feedback, the majority of reviewers felt confident giving feedback in most circumstances, but less confident delivering difficult feedback. One reviewer who felt uncomfortable delivering feedback commented that:

not having received feedback on own performance throughout my career affects my personal confidence level in giving feedback - hard to empathise or know how it feels for recipient.

Conversely another felt that participation in this process had helped with the development of what is an increasingly important skill:

this process is also a valuable educational tool for more senior staff in compiling and delivering effective and successful feedback.

Challenges of implementation

Documentation contributes to transparency and accountability in the process, but it also has the potential to compromise confidentiality. Therefore care must be taken with storage and future handling of the information. Regardless of who controls the information, trust is required that it will be handled with professional integrity and used for staff development. The choice to give the recipient control of their information is a choice to empower and respect them as professionals who are committed to personal learning and professional development. The most rewarding aspect of this process for one reviewer was the ability to give ownership of the feedback and professional development to the individual RT. Another noticed that giving staff control over their information led to:

increased confidence of planners in their own ability and the extension of their skills as a result.

The most demanding aspect of the process was reported as managing case-specific forms and finalising the feedback. All respondents agreed that appropriate allocation of time, training, and managerial support for the process would increase its effectiveness. A performance review process is often seen as time consuming in the already busy schedules of RTs and so the provision of protected time is important to achieve a successful feedback process.10 Emphasis was also placed on the need to be empowered with opportunity to follow through on development strategies. One reviewer found that they were:

not always in a position to follow through on development needs due to lack of knowledge of rostering for me and the recipient.

Identifying the need for managerial support raises a worthy discussion of the balance between managerial responsibility and personal responsibility in professional development. While the support of management is necessary to empower staff in achieving their development goals, shared responsibility must be emphasised to foster a culture where individuals are supported to take responsibility for their own development.

Suggestions for improvements

Achieving consistency of approach is a challenge in the implementation of any peer review process.10 Guidelines were provided to support consistency in managing this process; however it was acknowledged that reference to them was required in order for that to be effective. Suggestions for improving the guidelines included; defining individual responsibilities, clarifying feedback follow-up and access to mediation. These aspects of a process contribute to the perceptions of justice and impact on whether the feedback is accepted or not. Fairness is achieved through full disclosure of the process, allowing input from the recipient before the feedback is finalised, and basing all judgements on evidence.12,14 One recipient supported the value of input by identifying the lack of it:

There is no real pathway to discuss with senior RTs what aids junior RTs in learning and development – this may help in developing the process further.

While both groups felt that transparency and objectivity had largely been achieved, it was suggested that the presence of a third party during discussions and documentation of conversations would improve this in some situations. Training was also recommended for reviewers to maintain objectivity and thus improve the quality of the feedback.

While adequate attention was given to both areas of strength and development needs, this was weakened by the limited ability to respond to strengths with pathways for progression, and to weaknesses with meaningful development plans. The need to train all staff in the goals of staff empowerment and skills development were also recurrent in comments made by both groups, and highlighted in the comment:

It is often difficult to give constructive feedback on areas of weakness if RTs are under the impression that this can be used against them in developing a career path. It needs to be reinforced that this is more a tool for development and empowerment and…will be used to develop them further.

One recipient expressed frustration in:

being given the feedback and knowing what needed to be worked on, then not having the resources, time and support to set goals and improve in these areas.

Two recipients indicated the need for greater evidence on performance but did not indicate where evidence was lacking. Evidence was provided through individual forms, and the management of these was cited as being the most onerous tasks in the process. Documenting additional evidence may be ideal, but could add to the complexity of planning practice and the demand on the existing workload.

Limitations of results

Stake12 discusses the inevitable presence of bias in any evaluation, where the desire to see the reviewed process supported or further developed may influence respondents in a particular direction. Positive responses to the framework may indicate that those who participated were inclined to support the process and promote its development. Participation was, however, inclusive of all RTs and contribution was voluntary. Researchers may have been at equal risk of such bias, however, as health professionals, RTs are challenged to continually reflect on their actions in order to nurture a professional and ethical approach to all aspects of practice. This paradigm is particularly relevant to the concept of bracketing where reflection on all stages of the PAF development, implementation and evaluation allowed an objective approach to be maintained. Accordingly the professional and academic authors of the piece have drawn on these skills to engage in personal and group reflection on the different project stages. A reflective journal approach to bracketing was proposed by Tufford and Newman.15 Although this research relates to social work, it can be seen that the prevalence of reflection in the radiation therapy profession can enable a more active and group approach to using reflection to support bracketing.

Retrospective evaluation relies heavily on personal interpretation12 and recall. This approach was taken in order to gain insight into individual experience of the framework. RTs experienced the PAF at different stages of development and implementation, which may have influenced both experience and memory of it. Finally, all reviewers indicated appointment at a senior level. This potentially excluded RTs who may have participated as peer reviewers to individual aspects of feedback but were not instrumental in finalising it. In hindsight, criteria for eligibility may not have been clear, limiting the respondents to only appointed senior RTs.

There was additional interest in identifying how many respondents contributed to feedback while acting at a higher level, to determine how this may have impacted on feedback satisfaction and confidence. Ethics requirements had this omitted to ensure anonymity.

Conclusion

This project was developed from a need to establish consistent practice and active development among RTs in a dynamic work environment. Clearly defined expectations for practice and a process for providing feedback on performance were necessary to achieve this. The RT-PAF was developed to meet these needs and to promote a culture of performance appraisal that was consistent, objective, transparent, based on evidence and focussed on skills development. Evaluation of this framework revealed its value in empowering staff to maintain an appropriate level of skill and reach their professional potential, in an environment of continually evolving technologies and practice change. This framework achieved its aims of defining expectations for practice and providing a fair and objective feedback process that focussed on skills development. Obstacles to its success were related to the support and time resources required for RTs to be able to develop meaningful development plans and implement strategies to meet these goals. The provision of protected time to conduct the review, provide feedback and complete development plans in a timely manner was highlighted as critical in ensuring the relevance of the feedback. Ongoing management support of this framework was needed through rosters and training to meet development needs. Training in the principles of the process and in feedback delivery skills was also recommended to ensure consistency in quality of the feedback delivered and how the process was conducted. The goal of empowering and developing staff required emphasis to reduce RT concerns of having their performance reviewed and the feedback documented. Further development of the RT-PAF requires commitment to ongoing development of the framework, and a commitment to empower staff to allow for realistic goal-setting and development of skills as they keep pace with changes to technology and practice. The benefits arising in terms of staff satisfaction and development, however, highlight the importance of this commitment to the modern radiation therapy workforce.