Clinical perceptions of radiation therapy undergraduate competency standards.

INTRODUCTION: The multifactorial nature of clinical skills development makes assessment of undergraduate radiation therapist competence level by clinical mentors challenging. A recent overhaul of the clinical assessment strategy at Queensland University of Technology has moved away from the high-stakes Objective Structured Clinical Examination (OSCE) to encompass a more continuous measure of competence. This quantitative study aimed to gather stakeholder evidence to inform development of standards by which to measure student competence for a range of levels of progression.
METHODS: A simple anonymous questionnaire was distributed to all Queensland radiation therapists. The tool asked respondents to assign different levels of competency with a range of clinical tasks to different levels of student. All data were anonymous and was combined for analysis using Microsoft Excel.
RESULTS: Feedback indicated good agreement with tasks that specified the amount of direction required and this has been incorporated into the new clinical achievements record that the students need to have signed off. Additional puzzling findings suggested higher expectations with planning tasks than with treatment-based tasks.
CONCLUSION: The findings suggest that the amount of direction required by students is a valid indicator of their level and has been adopted into the clinical assessment scheme. Further work will build on this to further define standards of competency for undergraduates.

Introduction

The 3‐year Bachelor of Radiation Therapy course at Queensland University of Technology (QUT) has undergone a major curriculum review in recent years in order to maintain currency of content as well as embed evidence‐based pedagogy.1 An essential aspect of the review process has been engagement of a range of stakeholders including students, educators and clinical professionals. The latter group has been particularly involved in the redesign of the clinical assessment strategy for the course as suggested by Gibbs.2 As a result extensive feedback has been sought from all stakeholders, including students and clinical colleagues.

Assessment for the clinical units at QUT includes an academic component as well as the clinical component with the clinical assessment contributing 60% to the overall mark. The clinical assessment historically included an Objective Structured Clinical Examination (‘OSCE’) as well as assessment with the Australian Universities Radiation Therapy Student Clinical Assessment Form (AURTSCAF) criteria, outlined by Giles et al.3 In 2011, a decision was made to remove the OSCE. High stakes staged assessments such as this have been reported to cause high‐anxiety levels4 and thus provide an inaccurate representation of student learning in relation to key clinical skills.5 To measure a more consistent approach to clinical performance, these 1‐day ‘snapshots’ of students are increasingly being phased out. Prior to this study at QUT, this was replaced by a new ‘clinical achievement record’ (CAR) competency assessment as seen in Figure 1.

Figure 1

Clinical assessment changes.

The AURTSCAF aims to measure student performance against a range of attributes and skills related to six domains of practice. The case report encourages students to develop a holistic view of patient care and the reflective journal builds reflective practice into student progression. The CAR aims to measure levels of student competency with a range of defined supervised clinical tasks drawn from the scope of practice of a radiation therapist.

When formulating an appropriate assessment of competence, it was helpful to consider the distinction between a ‘competency’ relating to requirements of a task and ‘competence’ relating to personal attributes.6 Although this article discussed the use of clinical oral examinations to assess competence in radiation therapy (RT) students in Ireland, the definition of competence for RTs in current undergraduate courses is important. This difficulty in defining competency is also highlighted in a study of nursing students which asserts that assessing the clinical practice of nursing students is problematic as a result.7 The purpose of introducing the CAR tool was to measure competency relating to tasks.

Rationale

One of the long‐standing challenges associated with clinical competency assessment is that of student feedback, and evidence from the literature7, 8 suggests that there are discrepancies in marking and assessing, not only between different centres but also between staff in the same centre. These relate to differences as to how clinical mentors perceive the level of student competency and this seemed to be more pronounced after the removal of the staged assessment. Differences in marking were also evident from the AURTSCAF where staff in different centres had rated the same students quite differently. There were also a number of instances where students were told that they are unable to receive a top grade on the AURTSCAF because they are not in third year or not undertaking their graduate development year. This can affect a student's overall Grade Point Average (GPA) score as it makes it impossible to achieve the highest grade in the clinical units. It is clear that different staff members have different perceptions of the level of competency required for each level of student progression9, but the basis for this is unclear and the students are not made aware of this prior to assessment.

These marking differences have traditionally been ameliorated through the moderation process, although this in turn has been shown to be subjective.10 Evidence from the literature11, 12 confirms the value of agreed standards to help reduce the inter‐assessor variability. Smith11 highlighted the value of involving assessors in the development of these standards. The rationale for this study was to seek feedback from a wide range of clinical colleagues to define standards for different levels in order to improve parity for all students.

Aims and objectives

The aim of the study was to identify standards of clinical competency for each level. It is expected that students will progress at different rates, but it is important to establish the minimum level required for each stage of development in order to ensure that students and staff are aware of the minimum level each student should be capable of on each clinical placement. This would improve parity across the various clinical departments. Informal student feedback suggested that there was some difference in the perception of student performance between different grades of staff with suggestions that perhaps seniority of staff made a difference to what was expected of students, so a secondary aim was to determine the strength of any correlation.

Methods

A simple, anonymous questionnaire was used to collate staff opinions of expected student progression in semester 1 2012. The questionnaire consisted of a series of statements relating to common tasks students could be expected to complete. A 5‐point Likert scale with year 1, 2, start year 3, end year 3 and graduate year was used and the participants were asked to assign an appropriate level to each task. In general, higher year level students should be able to complete lower level tasks so staff were asked to select the minimum level at which a student should be expected to complete each task satisfactorily.

Recruitment

The aim was to gain feedback from as many radiation therapists in Queensland as possible so questionnaires were handed out to staff attending a clinical education update. A face‐to‐face paper‐based questionnaire approach was utilised rather than deployment of online survey tools to improve response rates.13 Since not all radiation therapists were able to attend the event questionnaires were also given to the local clinical educators to hand out to other interested staff members. Participants were advised that participation was voluntary and anonymous and the university research ethics committee advised exemption. Data collection was manual with questionnaires being either collected on the day or posted back to QUT by the local clinical educator.

Results

Response rate

Overall 300 questionnaires were provided; the exact number of questionnaires that were accessed is not known due to the method of distribution. Completion of the questionnaire was voluntary and 89 complete responses were received, with a reasonable proportion from each of the centres. Table 1 shows the grades of staff who returned the questionnaires. The highest proportion was at levels 1 and 2, which reflects the largest number of staff in the departments, that is, those who are expected to be working most with the students.

Table 1

Staff response demographics

Grade not supplied	NPDP	HP3	HP4	HP5	HP6
9	10	35	19	13	3

NPDP, National Professional Development Programme/New graduate; HP3, Level 1/junior RT/Band 5 Radiographer; HP4, Experienced RT/Deputy Charge/Band 6; HP5, Senior RT/Charge RT/Band 7; HP6, Manager/Superintendent RT/Band 8.

Levels

Some of the standards that were provided in the questionnaire are shown in Figure 2. From the responses received, each task was assigned to a particular year level. Although there was considerable variation in some areas, one of the themes that the data demonstrates is the correlation between amount of direction and the level of students. Most staff felt that students at the start of year 2 (who at QUT would only have completed 1 week of clinical placement) should be able to prepare the room and follow protocols on radiation safety. At the start of year 3, when students have had between 10 and 12 weeks of clinical placement, staff members believe that students should be able to work as part of the team and operate equipment with minimal direction. This follows onto the start of the supervised practice year where it is felt that they can work independently. In general, staff members agreed that students at the start of year 2 (the first big clinical placement at QUT) would require close direction for all tasks. Year 3 students were expected to be able to work as part of the team with minimal direction and being trusted to work independently was expected at the level of recent graduate. From this, it can be concluded that clinical staff members perceive that amount of direction is a valid measure of student ability related to their level.

Figure 2

Questionnaire standards.

Planning

One interesting finding was the apparent discrepancy between expected levels of planning and treatment skills. At the start of year 3, students are expected to be able to produce a clinically acceptable plan independently, which is in stark contrast to the other year 3 tasks where they are expected to work as part of the team and operate equipment with minimal direction. Results also suggested that staff members thought that students cannot be trusted to take and pass on messages until year 3 and students were not expected to be able to explain procedures to patients until after qualification. These findings suggest that staff members have a higher expectation for planning skills than for localisation or treatment.

Discussion

The rationale for this study was primarily to seek feedback from clinical radiation therapy staff to define standards in an attempt to improve parity for students, who are expected to attend clinical placements in a number of different centres, both public and private. There are a number of limitations with the study and thematic analysis has revealed some interesting issues, which will be discussed.

Limitations

There are a number of limitations with this study. With a 30% response rate, it is challenging to demonstrate validity. Thus, it is difficult to tell if the results are representative of the whole population or if the sample is from people who have a particular interest in student education. It also has to be considered that the staff who responded may have been influenced by the most recent student they worked with and that experience may have influenced their expectations. The other major problem with interpreting the data lies with the proven variability of expectation of assessors. McCarthy and Murphy assert that assessors' interpretation of clinical competence of students varies widely, which makes assessing a student's clinical capability problematic.7 Although this study concerned nursing students, their findings are particularly relevant to radiation therapy students at QUT, who are expected to attend a different clinical site for each placement. A larger sample would potentially provide more concrete data regarding levels of tasks.

Impact

The standard deviations of responses were quite high, so although it was possible to spot trends when looking at the means of the different responses from all staff, these confirm the wide variation of staff perceptions. This reaffirms the initial rationale for the study. Historically a staged assessment was used at QUT to assess students on clinical placement. The guidelines for the AURTSCAF make it clear that the assessment of students should reflect the consistency and standard of performance across the placement, taking into consideration the experience level of the student. The staged assessment did not fulfil this and did not address the issue of capability. Studies have shown that one single assessment method is subjective and therefore not sufficient to determine whether a practitioner is professionally competent, hence the reason for its removal.14 The feedback from this study indicated that ‘amount of direction’ was seen by staff as a useful measure of student competency. This measure underpins the new CAR assessment tool which requires staff to sign students off according to the amount of direction required for each task.

Skill mix

One of the most puzzling findings from the study concerned the apparent difference in expectation between planning and patient communication tasks. High expectations of autonomous planning expertise were expected of students at an early stage of their progression, whereas unsupervised patient interaction skills were not deemed as suitable tasks for the same individuals. This, could, perhaps, reflect the findings of a study into the relative importance of different research topics which postulated that some RTs consider technological skill to be more important than patient care.15 Conversely this could suggest that staff are not willing to trust students with patient interactions due to the lack of oversight or checking procedures that are inherent in planning. Assessment of student competency in relation to patient interaction is challenging and has been addressed partially by seeking patient feedback.16 Despite this, the findings of this study indicate that more research into clinical staff attitudes to various aspects of their role is clearly warranted and could help further inform development of more detailed standards.

Capability

It is generally accepted6, 7 that capability or professional competence is difficult to assess as it includes cognitive and emotional aspects, not just technical aspects of practice. ‘Professional competency is more than factual knowledge and the ability to solve problems with clear cut solutions; it is defined by the ability to manage ambiguous problems, tolerate uncertainty and make decisions with limited information’.17 Although this statement is more than 30 years old, it is especially true today in radiation therapy, where techniques are more complex and require critical thinking skills on a daily basis. With this in mind, a capability statement has been introduced to accompany the students’ clinical achievements, with the clinical educator assessing whether students are able to respond to guidance and feedback and apply new knowledge to new situations.

Validation

Further work is planned to validate this measure by triangulating student progression with their CAR with overall clinical unit performance using a similar method to that of Selim et al.14 Results from this study are expected to inform future development of a national standard for clinical assessment.

Conclusion

The findings from the study have helped determine guidelines for measuring student competence. They have also indicated that there is considerable variation in staff perception of student competence and it is hoped that the introduction of guidelines will address this. There is a strong trend that suggests that the ‘amount of direction’ is a valid measure of the level of student ability. This work has informed the method by which student competence in radiation therapy skills are measured, so that the amount of direction required measures progression. Our limited data suggests that there is no correlation whatsoever between staff grade and student levels. It would be useful to repeat the questionnaire after guidelines have been issued. Planned study at the national level will build on this work and help further standardise clinical assessment in Australian radiation therapy education.

Conflict of Interest

The authors declare no conflict of interest.