Development of the Student Practice Evaluation Form - Revised (Second Edition) (SPEF-R2): The second action research cycle.

INTRODUCTION: The Student Practice Evaluation Form - Revised Edition (SPEF-R) was used across Australian universities from 2008 to 2020 to assess occupational therapy student performance on practice placement. Evolution of practice contexts, placement models and professional competency standards prompted updating of the tool. This paper describes the second and final action research cycle in the development of the SPEF-R2.
METHODS: Cycle 2 included three phases: (a) piloting of the SPEF-R2 and post-pilot survey to determine utility and applicability; (b) post-pilot focus groups/interviews; and (c) final amendments for publication and launch. Quantitative data were summarised descriptively, and qualitative data were analysed using qualitative content analysis and reported using illustrative quotes.
RESULTS: In Phase 1, 23 participants piloted the SPEF-R2 and completed a post-pilot survey. Results indicated participants found the SPEF-R2 relevant to a range of traditional and contemporary settings, easy to understand and an improvement over the previous version of the tool. Most participants found it more concise and less repetitive than the SPEF-R. Participants particularly valued additions regarding culturally responsive practice, student health and well-being and reflective practice. In Phase 2, five post-pilot focus groups/interviews were held, gathering a deeper understanding of its utility. Discussion highlighted desire for an additional core item within the self-management domain (Domain Two). Increased confidence in rating and provision of feedback on student performance were also evident. Reflection on findings led to final amendments and publication of the SPEF-R2.
CONCLUSION: Extensive consultation with the occupational therapy community informed the development of the SPEF-R2, reflecting contemporary practice and meeting the expectations of Australian occupational therapists. Action research was an effective approach to the development of the SPEF-R2. Use of the manual and training website and support from universities are paramount if practice educators are to use the tool effectively.

The SPEF‐R2 reflects contemporary practice and was well received by the Australian occupational therapy community.

Action research was an effective approach to the development of the SPEF‐R2.

Use of the manual and training website and support from universities are paramount if practice educators are to use the tool effectively.

BACKGROUND

Occupational therapy university programmes rely heavily on practice placements to ensure graduates develop appropriate skills, behaviours, and attitudes for the workplace (Gustafsson et al., 2017). The World Federation of Occupational Therapists (WFOT) stipulates that supervised professional practice must be completed by occupational therapy students prior to graduation. Experiences should be congruent with current philosophy, represent a depth and breadth of practice contexts, and involve clear and explicit evaluation of student performance (WFOT, 2016).

In Australia, a number of allied health professions use single, nationally recognised tools to evaluate student practice placement performance (Dalton et al., 2012; McAllister et al., 2013). Since 1998, Australian occupational therapy university programmes have used a tool developed by The University of Queensland to measure this. The Student Practice Evaluation Form (SPEF) was initially designed in 1995 as a response to the need to evaluate student performance in a consistent and reproducible way, across a wide range of practice contexts by practice educators with varied experience (Turpin et al., 2011). The tool was subsequently refined in 2008 and retitled The Student Practice Evaluation Form – Revised Edition (SPEF‐R) (Turpin et al., 2011). Following release of the 2008 version, studies into the tool's construct and content validity, internal consistency, and inter‐rater and test–retest reliability were conducted, providing evidence of its suitability for use across diverse settings (Rodger et al., 2016; Rodger, Coleman, et al., 2014; Rodger, Turpin, et al., 2014).

Since 2008, practice contexts and service delivery models have evolved, leading to emerging areas of practice placements (Whiteford & St‐Clair, 2002; Wilburn et al., 2016). As a result, occupational therapy students need to develop a broad and appropriate range of skills, behaviours, and attitudes to meet the demands of emerging practice contexts. In response to these changing practice demands, the Occupational Therapy Board of Australia released updated competency standards (Occupational Therapy Board of Australia, 2018). The 2018 Australian Occupational Therapy Competency Standards (AOTCS) now reflect the skills, behaviours, and attitudes required by occupational therapists to practice in contemporary settings. Examples of additional contemporary skills include cultural responsiveness and reflective practice. It is therefore imperative that any tool used reflects these contemporary skills and the 2018 competency standards.

An update of the SPEF‐R was considered more appropriate than creation of a new tool for several reasons. First, the SPEF‐R had been widely adopted nationally and was familiar to the Australian occupational therapy community. Second, the SPEF‐R was developed using a programme of research involving consultation with multiple stakeholders and provided a solid platform to build on (Rodger et al., 2016; Rodger, Coleman, et al., 2014; Rodger, Turpin, et al., 2014). Third, significant resources had been invested in developing an online platform for access to the SPEF‐R, and it was considered the pragmatic direction to take.

Considering these social and regulatory changes, it was necessary to update the SPEF‐R so that evaluation of student performance more accurately reflected contemporary practice. This resulted in the Student Practice Evaluation Form – Revised (Second Edition), or the SPEF‐R2. The SPEF‐R2 project was undertaken across two action research cycles. The first cycle was described by Caine et al. (2020) and focussed on mapping the competency standards against the SPEF‐R, identifying essential amendments, conducting a national survey regarding a draft version of the SPEF‐R2, and finalising the 36‐ to 37‐item SPEF‐R2 for piloting. The first cycle found good alignment between the SPEF‐R2 items and the 2018 competency standards and was well received by the Australian occupational therapy community as an improvement on the SPEF‐R. This paper describes the second and final action research cycle, which aimed to determine the perceptions of Australian occupational therapists and university educators in relation to the utility and applicability of the proposed SPEF‐R2. This cycle included piloting the tool and making final amendments for publication of the SPEF‐R2.

METHODOLOGY AND RESULTS

Like the development of both the SPEF and the SPEF‐R (Allison & Turpin, 2004; Turpin et al., 2011), the SPEF‐R2 project was undertaken using action research methodology (Stringer, 2014). Action research uses cycles of action, observation, reflection, and planning in relation to an identified practice situation. Consistent with this approach, the methodology and results are presented together in this paper.

The second action research cycle included three phases:

Phase 1: Piloting of the SPEF‐R2 and post‐pilot survey to determine utility and applicability

Phase 2: Post‐pilot focus groups and interviews

Phase 3: Final amendments for publication of the SPEF‐R2

Prior to conducting the research, ethical approval was obtained from The University of Queensland Human Research Ethics Committee (#201800814) and Griffith University Human Research Ethics Committee (#2019/548).

Phase 1: Piloting the SPEF‐R2

Participant recruitment

Convenience sampling was used to recruit participants. The survey used in the previous action research cycle (Caine et al., 2020), which focussed on perceptions of the draft SPEF‐R2, included a question to identify practice educators interested in piloting the tool. Those who indicated interest were contacted and offered the opportunity to register for participation in the pilot of the SPEF‐R2. Further pilot participants were sought through professional networks. A call for volunteers was posted on the Occupational Therapy Australia (OTA) Research Surveys webpage, and an email was circulated to Australian university programme staff containing suggested wording and links for forwarding to practice educators.

Practice educators were directed to an online registration survey that collected further participant information related to professional registration status, upcoming student placement dates, contact details, and consent. Initially, participants were required to have a confirmed and allocated practice placement of at least 7 weeks' duration timetabled within the pilot period—July to December 2019—to be eligible. Following feedback from potential participants and some universities, the minimum placement length was reduced to 5 weeks. Full‐time practice placements often range in length for Australian occupational therapy students; thus, the minimum of 5 weeks allowed the revised tool to be piloted with a range of placement lengths that reflected reality. Fifty‐eight potential participants started the pilot registration survey, with 47 completing it in full. These participants were then contacted by email with further information regarding participation. They were provided with a copy of the SPEF‐R2 User Manual and Evaluation Form, a fact sheet summarising key changes made to the tool, frequently asked questions/answers, participant instructions, and a post‐pilot reflection tool. Of the original 47 who completed the pilot registration and were contacted by email, 23 piloted the SPEF‐R2 and completed the online survey in full. Those who did not pilot the tool cited workload pressures, changes to role, amended student placement arrangements, or did not respond to researcher emails.

SPEF‐R2 pilot and online survey

During the pilot of the SPEF‐R2, to maintain student equity, participants continued to use the existing SPEF‐R to formally evaluate students. They were asked to then complete a second evaluation of the student using the SPEF‐R2, at either halfway or the end of the placement, to compare the two evaluations. The project team timed email reminders to coincide with individual placement start, midway, and finish dates to encourage completion and provide opportunities for support. The purpose‐designed post‐pilot reflection tool, provided along with other materials for the pilot, included prompt questions regarding what educators liked and did not like about the new tool, as well as how it compared to the existing version and whether or not it was considered an improvement. Participants were encouraged to use the reflection tool in preparation for completion of the post‐pilot survey; this was not collected by the researchers.

Following completion of the SPEF‐R2, participants were directed to a purpose‐designed 33‐item online survey that gathered their perceptions of the utility and applicability of the revised tool. The survey gathered demographic data such as location, years of experience, areas of practice, and experience using the SPEF‐R. Likert‐scale response sets and open‐ended questions with free text boxes were then used to explore opinions and experiences. All participants were required to provide consent before being able to access the survey questions. Table 1 outlines details of the participants.

TABLE 1

Demographic characteristics of occupational therapy practice educator participants (N = 23)

Characteristic	n (%)
Gender
Female	22 (95.6)
Male	1 (4.4)
Age group
18–24	1 (4.4)
25–34	10 (43.5)
35–44	6 (26.1)
45–54	5 (21.7)
55–64	1 (4.4)
State or territory where currently working
Queensland	10 (43.5)
New South Wales	6 (26.1)
Victoria	3 (13.0)
South Australia	2 (8.7)
Western Australia	1 (4.4)
Northern Territory	1 (4.4)
Years working as an occupational therapist
1–2 years	1 (4.4)
3–5 years	3 (13.0)
6–10 years	8 (34.8)
11–15 years	3 (13.0)
16–20 years	3 (13.0)
Over 20 years	5 (26.1)
Current position
Clinician	13 (56.5)
University practice educator	4 (17.4)
Other university role	3 (13.0)
Other	3 (13.0)
Primary work setting
Public hospital	11 (47.8)
Public community	2 (8.7)
Private community	1 (4.4)
University	5 (21.7)
Other	4 (17.4)
Primary clinical/educational caseload or area of expertise
Adult physical	7 (30.4)
Adult mental health	3 (13.0)
Paediatrics	5 (21.7)
Aged care	1 (4.4)
Child and youth mental health	1 (4.4)
Other	6 (26.1)
Number of students supervised on placement in the past
1–2 students	2 (8.7)
3–5 students	3 (13.0)
6–10 students	5 (21.7)
More than 10 students	13 (56.5)
Number of times SPEF‐R used with a student on placement
0 times	1 (4.4)
1–2 times	2 (8.7)
3–5 times	4 (17.4)
6–10 times	5 (21.7)
More than 10 times	11 (47.8)
SPEF‐R2 piloted at the halfway or final evaluation
Halfway	5 (21.7)
Final	18 (78.3)
SPEF‐R2 stream used to evaluate student
Stream A (direct service provision)	19 (82.6)
Stream B (project management/consultancy)	4 (17.4)

Five participants (22%) piloted the SPEF‐R2 at the halfway point of their student's placement, and 18 (78%) at the final evaluation. The SPEF‐R2, like the SPEF‐R, is separated into two streams: (1) Stream A: Direct service provision and (2) Stream B: Project management/consultancy. The majority (n = 19, 83%) used Stream A, with the remainder (n = 4, 17%) using Stream B. The pilot sample included representation from six states and territories across Australia, with highest participation rates in Queensland (n = 10, 44%), New South Wales (n = 6, 26%), and Victoria (n = 3, 13%). Placements occurred in the final years of students' programmes, across a variety of practice contexts and supervising practice educators reported varied experience in using the SPEF‐R. Numerical identifiers from 1 to 23 were used for participants. Quantitative data from the survey were summarised descriptively, and qualitative data were analysed using qualitative content analysis (Neergaard et al., 2009) and reported using illustrative quotes.

SPEF‐R2 pilot survey results

Quantitative data from the pilot survey are summarised in Table 2. The online survey revealed positive feedback about the SPEF‐R2 from practice educators. Overall, almost all (96%) pilot participants agreed or strongly agreed that the additions, deletions, and amendments within the SPEF‐R2 represented an improvement on the SPEF‐R. This was reflected in additional comments from participants throughout the survey, many of which have been included within the core feedback topics below.

TABLE 2

Clinical educator pilot survey quantitative responses (N = 23)

Question/statement, n(%)	Strongly agree	Agree	Neither agree or disagree	Disagree	Strongly disagree
The SPEF‐R2 is applicable to my current OT role and setting	16 (69.6)	7 (30.4)	0 (0.0)	0 (0.0)	0 (0.0)
The SPEF‐R2 covers the breadth of OT skills and attributes for contemporary Australian OT practice	13 (56.5)	10 (43.5)	0 (0.0)	0 (0.0)	0 (0.0)
The SPEF‐R2 accurately reflects the 2018 AOTCS in practice	9 (39.1)	12 (52.2)	2 (8.7)	0 (0.0)	0 (0.0)
I found the SPEF‐R2 easy to use	16 (69.6)	4 (17.4)	3 (13.0)	0 (0.0)	0 (0.0)
The SPEF‐R2 domains and items are easy to understand	12 (52.2)	10 (43.5)	1 (4.4)	0 (0.0)	0 (0.0)
It was easier to distinguish between the domains and items in the SPEF‐R2 (as compared to the SPEF‐R)	11 (47.8)	8 (34.8)	4 (17.4)	0 (0.0)	0 (0.0)
Additions relating to culturally responsive practice enriches the SPEF‐R2	14 (60.9)	7 (30.4)	2 (8.7)	0 (0.0)	0 (0.0)
The more explicit references to reflection in domain eight add value to the SPEF‐R2	16 (69.6)	5 (21.7)	1 (4.4)	1 (4.4)	0 (0.0)
I found it easy to adapt from the SPEF‐R to the SPEF‐R2	17 (73.9)	6 (26.1)	0 (0.0)	0 (0.0)	0 (0.0)
I found it easier to generate student feedback with the SPEF‐R2	9 (39.1)	6 (26.1)	7 (30.4)	1 (4.4)	0 (0.0)
Overall, I feel the additions, deletions and amendments within the SPEF‐R2 constitute an improvement over the SPEF‐R	10 (43.5)	12 (52.2)	1 (4.4)	0 (0.0)	0 (0.0)
How did your student score on the SPEF‐R2, as compared to the SPEF‐R?	Better on the SPEF‐R2 3 (13.0)	Worse on the SPEF‐R2 0 (0.0)	About the same 20 (87.0)
Did your student pass or fail on the SPEF‐R and the SPEF‐R2?	Passed both 21 (91.3)	Passed SPEF‐R, failed SPEF‐R2 0 (0.0)	Failed SPEF‐R, passed SPEF‐R2 0 (0.0)		Failed both 2 (8.7)
Looking ahead, how would you expect the time taken to complete the SPEF‐R2 will compare with the SPEF‐R in future?	SPEF‐R2 quicker 8 (34.8)	SPEF‐R quicker 0 (0.0)	About the same 15 (65.2)

Contemporary practice

Contemporary additions, for example, references to social media, electronic documentation, and diversity, as well as clearer and more consistent reference to evidence‐based practice and reflection, were a feature of the SPEF‐R2. All participants agreed that the SPEF‐R2 covered the breadth of occupational therapy skills and attributes for contemporary practice. “I found it more current and relevant to today's practice” (Participant 15). Furthermore, 91% felt that the SPEF‐R2 accurately reflected the AOTCS 2018. The contemporaneousness of the SPEF‐R2 was reflected in multiple comments from participants, with one commenting “the language used is contemporary and considers the needs of occupational therapists in the future” (Participant 19). All pilot participants agreed that the SPEF‐R2 was applicable to their current role and setting.

Clarity and understanding

Ninety‐five per cent of participants agreed that SPEF‐R2 domains and items were easy to understand. “The language is clear and concise—there is no ambiguity in what is being assessed from the clinical educator perspective” (Participant 19). Others commented it was “much easier to understand than the previous SPEF‐R” (Participant 12), and “the descriptions and examples are more relevant and more inclusive” (Participant 11). Ninety‐three per cent believed it was easier to distinguish between domains and items in the SPEF‐R2 as compared to the SPEF‐R, for example, “language used for each item is clearer. Examples under each item are more specific to practice …” (Participant 13). Some free text comments about the amount of repetition and duplication favourably compared the pilot version of the SPEF‐R2 to the earlier version reviewed in Cycle 1 (Caine et al., 2020). For example, participants stated the SPEF‐R2 was “a better document with less duplication” (Participant 7), “there seemed to be less doubling up of information” (Participant 2), and “items in each domain are more succinct and some domains are ‘tighter’ (less items now)” (Participant 22).

Ease of use

The SPEF‐R2 was rated as easy to use by 87% of participants. All participants said they found it easy to adapt from the SPEF‐R to the SPEF‐R2, with almost two‐thirds (65%) also reporting it to be easier to generate student feedback with the updated tool. Being able to more confidently rate student performance emerged as a key point for others, for example, “much easier to rate” (Participant 12), and “as I was assessing the student's performance, the items helped me decide more emphatically how the student was performing” (Participant 22).

Cultural inclusions

Six items were enhanced with references to cultural responsiveness, for example, “uses culturally safe and responsive communication”, and a new item was created within the professional behaviour domain to focus on this aspect of practice in the SPEF‐R2. Ninety‐one per cent of participants agreed or strongly agreed that additions regarding culturally responsive practice enriched the SPEF‐R2. Example comments included, “I feel this really is important given the huge cultural focus in their [students] theoretical learning” (Participant 5) and “the SPEF‐R2 had greater cultural sensitivity to the needs of Aboriginal and Torres Strait Islander Peoples, and to people of culturally diverse backgrounds” (Participant 19). This was a commonly reported improvement. Thirteen (57%) participants declared that cultural safety additions were a change they liked within the SPEF‐R2, and an additional three (13%) identified cultural content additions as one of the key changes they noticed. “I think that the SPEF‐R2 has addressed the major gaps in the SPEF‐R and has done well to include a cultural aspect to the assessment. This is especially important for the services that deal closely with ATSI [Aboriginal and Torres Strait Islander] populations” (Participant 21).

Reflective practice

Ninety‐one per cent of participants agreed that the more explicit references to reflection in Domain Eight added value to the SPEF‐R2. Ten (43%) participants specifically mentioned the improved reflective practice content when asked what changes they liked within the SPEF‐R2. One commented, “the increased focus and attention to reflection as part of service evaluation is a great addition” (Participant 4), and another, “reflection is something I've always discussed with students, but until now there's been no formal evidence of this” (Participant 16).

Placement expectations

Use of the SPEF‐R2 as a placement planning tool for setting expectations for students was raised as another area of strength, with one participant commenting, “I believe it would be easier for students to understand expectations and for supervisors to initiate conversations … as it gives greater clarity around expectations within the workplace” (Participant 8). When reflecting upon the updated item examples within the SPEF‐R2, another participant commented that the “examples … give student and supervisor a clearer understanding of what is expected” (Participant 13).

Student perspective and well‐being

Numerous participants made mention of the SPEF‐R2 language and content being more aligned to the needs of students, for example, “explaining the SPEF‐R2 to a student would be more simplistic as the language is user‐friendly to both student and clinician/clinical educators” (Participant 19). Likewise, when asked generally what they liked about the SPEF‐R2, 12 participants (52%) specifically cited their approval of changes in wording relating to student health and well‐being. One commented, “the more holistic examples in relation to self‐management and stress management are more aligned to some of the health and wellbeing issues students confront” (Participant 9). Another expanded on this, commenting, “the SPEF‐R2 focused on the student or learner in a more holistic way, for example emphasising the importance of self‐management skills through recognising own mental health and potential impact on clinical performance” (Participant 19).

Time to complete and student evaluation results

Though the time taken to complete the tool was not stated, the majority of participants (65%) felt that the time taken to complete the SPEF‐R2 would be about the same as the SPEF‐R. Approximately one‐third (35%) estimated that the SPEF‐R2 would be quicker. None considered that the SPEF‐R2 would take longer than the existing SPEF‐R in everyday practice. Reasons cited for the SPEF‐R2 being quicker included greater clarity and differentiation between items, improved item examples and more concise content.

Most participants (87%) reported that students assessed during the pilot scored approximately the same on the SPEF‐R2 as they did on the SPEF‐R. Three (13%) participants said their students scored more highly. Reasons for these differences were not provided, although responses to other questions suggest that a greater understanding of the item examples, and increased confidence in understanding the skills and behaviours to which items referred, may have been contributing factors. Of the 23 students with whom the SPEF‐R2 was piloted, two students failed their placement evaluation (i.e. if students did not pass all core items and a minimum number of non‐core items in each domain, this was considered a fail). Pass/fail results were the same across both versions of the tool.

Other feedback

Although the most frequent responses are detailed in the topics above, further qualitative feedback also related to improved item examples (22%), the addition of content regarding student peers (17%), a sense of the SPEF‐R2 being more holistic and inclusive (17%), improvements to the documentation domain (17%), a focus on consumer perspective and engagement (13%), and a sense that the SPEF‐R2 would be better for students (13%).

Concerns and issues

When asked about any amendments that they did not like or concerns that had not been addressed in the new version, 10 pilot participants (43%) made comments. One was concerned about the balance of core/non‐core items within the self‐management domain, “I find it frustrating that self‐management continues to have so many non‐core items when it is such an important part of practice” (Participant 19). Three participants (13%) remained concerned about the length of the SPEF‐R2 and the resulting time taken to complete it. Two participants (9%) raised the rating scale as an issue (unchanged in the SPEF‐R2). One pilot participant (4%) was concerned about the rewording of two specific examples within the information gathering and evaluation/reflection domains and proposed improvements. Another (4%) perceived a need for further inclusive language in the SPEF‐R2.

Phase 2: Post‐pilot focus groups/interviews

Ten (43%) pilot participants indicated an interest in follow‐up focus groups/interviews to further explore trends identified in the post‐pilot survey, with one withdrawing prior to commencement. Participants were sent emails detailing the purpose of the focus groups/interviews, seeking consent, and providing a variety of options for attendance: video call (Zoom), telephone, or in person.

The timing of the post‐pilot focus groups/interviews coincided with rapid escalation of the COVID‐19 pandemic in Australia. As such, all participants chose to attend via Zoom video calls. In total, five sessions were held to accommodate all nine attendees. One session involved four participants, one involved two, and there were three individual interviews.

Participants represented both urban and regional/rural settings and used various supervision models, such as apprenticeship (one student: one practice educator); shared (two practice educators); collaborative (more than one student); multiple‐mentor (multiple students and multiple practice educators); long‐arm (supervision at a distance); and role‐emerging (student placed in a non‐traditional OT role, or no local OT present).

In preparation for the focus groups/interviews, questions and discussion points were developed to further explore the findings from the survey results. These included questions regarding supervision model used, balance of core (must pass) and non‐core items, comparison of student results between the two versions, educator manual use, and impact on users' confidence to rate and provide feedback. Each session was recorded and transcribed verbatim. A summary of the focus groups/interviews was sent to all participants for member checking purposes prior to the project team analysing the data using qualitative content analysis (Neergaard et al., 2009). Responses confirmed it was an accurate record and no changes were made.

SPEF‐R2 pilot focus group/interview results

General feedback from focus group/interview attendees about their experience of using the SPEF‐R2 during the pilot was largely positive, consistent with the survey, with a small number of concerns raised. Feedback included confirmation of strengths such as reduced repetition, increased clarity, ease of use, and valuable additions. Concerns related to the tool length and rating scale of the SPEF‐R2. Key topics and ideas arising from the survey that the project team felt warranted a deeper understanding were explored further, as outlined below.

Core items related to self‐management

In the SPEF‐R, two out of five items in Domain Two were considered core items, that is, they must be passed. The majority of focus group participants agreed that an additional core item should be considered for this domain in the new SPEF‐R2, and this aligned with survey comments. The research team discussed which of the other items was most non‐negotiable in terms of student behaviour and the general consensus was that Item 2.3—Demonstrates initiative and responsibility for actions—was the most suitable as an additional core item. Participants indicated that this item was linked to student learning, seeking of feedback and meeting expectations “… and demonstrating initiative, you know, it's all part of the learning process to gain and seek that feedback” (Participant 5). This item was considered by many to have an overarching impact on student performance across multiple domains and items within the SPEF‐R2.

SPEF‐R2 manual and training website use

The five participants (56%) who had read the SPEF‐R2 manual reported that it was adequate, although some said it was difficult to identify what had changed because of the length of the document. Some were familiar with the SPEF‐R training website, but many had not accessed it for some time, and one did not know it existed. Those who reported the website to be helpful cited the video vignettes, with examples of educators and students using the evaluation tool, as the most helpful components. When asked what actions might increase the likelihood of educators reading the manual and using the training website, participants identified a need for reminders from universities about their existence. One suggested that if the 42‐page manual was shorter, they would be more likely to use it.

Confidence to offer and provide placements

Most considered that, although the SPEF‐R2 was not likely to alter their confidence in their capacity to offer a placement, it would enhance their ability to supervise a student. This included increased confidence in providing clear placement expectations, “I have more confidence in knowing exactly what these domains mean, which means I have more confidence in providing direction to the students” (Participant 3); selecting ratings, “I just felt that I had more evidence to fail her confidently” (Participant 3); and generating feedback, “I think because the items and domains are quite nicely differentiated, again it just makes you able to give clearer feedback” (Participant 5). Some believed familiarity with the SPEF‐R2 would mean students were more informed, enabled, and engaged, thus improving quality of student placements from both educator and student perspective.

Other comments and concerns

Additional comments or concerns were raised by some participants. Multiple participants felt the improved content of the SPEF‐R2 may reduce the amount of time required to develop workplace‐specific examples by practice educators—as many of the new and updated item examples are readily applicable to their local setting. One participant elaborated further on a survey comment that information gathering in practice is “not a linear process … you are continually gathering information” (Participant 5) and that this should be reflected accordingly in the SPEF‐R2. The three other members of the focus group agreed.

Two participants commented on the rating scale and difficulty differentiating between the levels. Another expressed uncertainty and a lack of clarity from universities around expectations of student performance at halfway versus final placement stages. This led to general discussion about what information students are provided with from universities regarding placement expectations and evaluation. At times, discussion suggested that some educators were applying the scoring criteria in ways other than that suggested in the manual, for example, letting students know at the commencement of placement that they would not use the higher scoring ratings, regardless of performance.

Phase 3: Final amendments and publication of the SPEF‐R2

The project team reflected on findings from the pilot and subsequent focus groups/interviews, and in response, changes and minor adjustments were made to the SPEF‐R2. These included an increase in the number of core items within Domain Two: Self‐Management to include Item 2.3: Demonstrates initiative and takes responsibility for actions and representation of information gathering as an ongoing process in Domain Six: Information Gathering. Additional clarification regarding scoring was added to the manual, for example, domain‐specific rating descriptors, but the length of the manual was not reduced as there was no indication of redundant material. This resulted in the finalised SPEF‐R2, ready for publication and launch. A set of resources to assist educators to transition to the updated tool were prepared, and national webinars were held to introduce universities and educators to the SPEF‐R2. The online platform was also updated to include the new version of the evaluation and some small improvements in functionality, and amendments to the training website were finalised.

DISCUSSION

This research enabled the development of a contemporary national evaluation tool for occupational therapy practice placements that reflects changes within the Australian competency standards and the needs of practice educators. Although the updated competency standards provided the impetus for the SPEF‐R2 development, an equally important benefit was the expansion of the tool's applicability to contemporary practice contexts. Although there exists a range of standardised assessments of core competencies for students in other health professions (Dalton et al., 2012; McAllister et al., 2013; Yaqoob Mohammed Al Jabri et al., 2021), the SPEF‐R2 remains the most established means of competency measurement for occupational therapy students in Australia.

This second cycle of the research project to update the SPEF‐R was a vital stage in the development of the revised tool. It provided an opportunity for practice educators to trial the tool within contemporary practice contexts, in real time, and give feedback that could be used to refine the tool. Pilot participants represented a broad demographic range, including geographic location—both urban and regional/rural—years of practice, experience as a practice educator, and experience with the previous SPEF‐R. This participant profile helped to ensure that the experiences and feedback relayed to the project team were representative of the broad diversity of Australian practice placement settings.

Overall, feedback gathered through both the post‐pilot survey and focus groups/interviews demonstrated the utility and applicability of the SPEF‐R2. Key areas of perceived strength included applicability to contemporary practice; improved clarity and understanding; ease of use; reduced repetition and duplication; expanded cultural content; reflective practice inclusions; clarity and guidance around placement expectations; representation of the student perspective; and improved language around student well‐being. Importantly, this feedback came from educators who were not using this type of tool for the first time but were comparing it to the previous SPEF‐R with which they were familiar. This resulted in rich reflections on the suitability of items, examples, and core competencies. The SPEF‐R2 was also piloted with students of varying performance levels, across both direct service and project style placements, and at both halfway and final points of the placements. This allowed the research team to gain perspectives of educators in a diverse range of contexts and adds further robustness to the testing.

As was the case in Cycle 1, there were some instances of conflicting feedback within the survey and focus groups/interviews. Some residual tension remained between aspects of the SPEF‐R2 that some users perceived as strengths, for example, comprehensive item examples, diverse content, and contemporary additions, and others identified as limitations, for example, length, complexity, and time to complete. In keeping with earlier stages of the project, the rating scale was not amended or re‐evaluated as part of the pilot. This decision was based on prior research (Rodger et al., 2016; Turpin et al., 2011) and explained further in the paper describing Cycle 1 (Caine et al., 2020). A proportion of pilot participants mentioned difficulty differentiating between points on the rating scale. Though clarity regarding use of the rating scale has been enhanced in the manual, the challenge remains to ensure that practice educators have opportunities to clarify their uncertainties either by consulting the manual or the training website (https://spef-r.shrs.uq.edu.au/). However, should educator uncertainty persist within the broader occupational therapy population, further research on the rating scale and useability of the manual is indicated.

More broadly, previous research has demonstrated that practice educators may not always be well equipped to provide feedback on students' competency during placement (de Beer & Mårtensson, 2015). This finding underscores the importance of training in the use of the SPEF‐R2 to promote accurate measurement of competencies and provision of quality feedback. Previous research on the SPEF‐R indicated that targeted training of practice educators was required for it to be considered a reliable tool (Rodger, Coleman, et al., 2014). In addition, in the current project, the use of authentic and contemporary examples assisted educators to substantiate their measurement of student performance. Consequently, the video vignettes developed for the SPEF‐R training website (Rodger, Turpin, et al., 2014) will require updating, and further reliability studies on the SPEF‐R2 should be conducted following a period of widespread use.

Study limitations

Action research is a time‐consuming process that requires consultation with a broad range of stakeholders and was made possible through support provided from licencing funds. In designing the methodology for the SPEF‐R2 pilot, the project team wanted to ensure that the process of participating in the study would not compromise or unduly impact on students' official practice placement evaluation experience or results. As such, the decision was made not to directly involve students in this cycle of the research. Although this remains a limitation of the SPEF‐R2 pilot, the student perspective was incorporated through nationwide consultation in Cycle 1, in which 63 students shared their perspectives on their experiences of the SPEF‐R and the initial draft of the SPEF‐R2 (Caine et al., 2020). Further research examining student perspectives of the SPEF‐R2 in practice placements would provide an additional perspective beyond those provided by practice educators in this pilot.

It may be important to consider the timing of when practice educators completed the SPEF‐R2 with their current student in mind. Completing the SPEF‐R2 within days of evaluating their student using the existing version of the tool may have made them more aware of their student's performance in relation to evaluation items. This may have positively impacted on their ability to score and provide feedback on the new tool, possibly giving them a false sense that it was the amendments that had made the difference. Two pilot participants recognised this and flagged it in their survey responses. Although the project team used questioning strategies in the focus groups/interviews to mitigate this possibility, it is acknowledged that the order of use of the two tools may have impacted results.

Finally, the emergence of the COVID‐19 pandemic in early 2020 impacted Phase 2 of this cycle (focus groups/interviews). Restrictions relating to travel, physical distancing, and adhering to protocols meant that all sessions had to be held online via videoconference. It is possible that the global pandemic also affected the number of practice educators available to participate in the pilot. The sample size of the participants in relation to the population of Australian occupational therapists needs to be considered in terms of generalisability of findings. In addition, participants volunteered for the focus groups, which may constitute a sampling bias.

Implications and recommendations for further research

Investigation of the psychometric properties of the SPEF‐R was carried out by Rodger, Coleman, et al. (2014) and Rodger et al. (2016) with key findings confirming strong evidence for its construct validity. Given the substantial changes to item descriptions and examples, as well as new and upgraded core items in the new SPEF‐R2, it would be valuable to repeat this research in several years, when sufficient experience with, and data from, the tool is available. Reconsideration of the rating scale was not considered appropriate at this time but might be usefully revisited in the future, potentially including an inter‐rater reliability study. A broader survey of the efficacy of the manual and available training for effective use of the tool may also be indicated once there is longer term, widespread use across real‐world contexts.

CONCLUSION

An action research methodology was used to review and update the core occupational therapy student practice placement evaluation tool in Australia—the SPEF‐R. Through this, the SPEF‐R2 was developed, and perceptions of the Australian occupational therapy community regarding the acceptability, utility, and applicability of the SPEF‐R2 were explored. The first cycle of this action research project was described in an earlier paper (Caine et al., 2020). This paper describes the second action research cycle involving quantitative and qualitative feedback from a pilot of the SPEF‐R2. It contributes evidence that the SPEF‐R2 provides an accepted means of evaluating occupational therapy student placement performance that aligns with contemporary practice contexts and Australian competency standards.

CONFLICT OF INTEREST

The University of Queensland holds the copyright to the SPEF‐R2 and licences the tool, for a fee, to universities in Australia and overseas. Individual authors hold no financial interest.

AUTHOR CONTRIBUTIONS

AC and CH managed data collection, analysis and interpretation, and development of the manuscript. All authors contributed to the conception of the work, data analysis and finalisation of the manuscript for publication.

FUNDING

This project was made possible with funding from The University of Queensland's School of Health and Rehabilitation Sciences. Funds were made available from SPEF‐R licence subscriptions to support the research.