Management of Radioactive Spills in Nuclear Medicine; Teaching and Assessing with Objectively Structured Assessment of Technical Skills.

Routine work in nuclear medicine requires the careful elution of radioactivity and its subsequent, storage and handling. Though all effort is maintained to prevent any "spill" of this radioactivity, accidents are bound to happen. The response to this spill is a methodically worked out a plan that is written and adopted as a "standard operating procedure." This protocol is taught to all involved in the area of working as a mock drill/apprenticeship model. No formal evaluation of learning is in place except for the mock drills. The objectively structured assessment of technical skills (OSATS) is a variation on the Objective Structured Clinical Examination, which is a form of workplace based assessment. The OSATS is cited in the Accreditation Council of Graduate Medical Education evaluation toolbox on the website as the most desirable evaluation tool for the patient care topics. It is the objective of this paper is to introduce the "OSATS" for teaching, and assessment of the learning, of the protocol for the management of radioactive spill. As a review of the literature on the subject failed to reveal any such teaching protocol/material/document for this important technical skill, we hope that it may act as a landmark for the development of teaching and assessment of other technical skills also.

Problem Statement and Context

Nuclear medicine (NM) is a unique field of medicine in the sense that it utilizes “open radiation” for the purpose of its diagnosis and treatment of illnesses. Everyday work in a NM Department requires the careful elution of radioactivity form a “generator” and its subsequent, adequate storage and handling. Though all effort is maintained to prevent any “spill” of this radioactivity, accidents are bound to happen. This “nuclear spill,” which maybe a minor or major one, is identified as an emergency and coded. An adequate response to it is mandatory to prevent any radioactive adversity. The response to this spill is a methodically worked out a plan that is established and followed accordingly. This plan is written and adopted as a “standard operating procedure (SOP).” The radioactive spill protocol is taught to all involved in the area of working of the radioactivity (residents, technical staff and radiation protection officer) as a mock drill/apprenticeship model. However, no formal evaluation of learning is in place except for routine mock drills.

As the realization for the training of medical experts to ensure reliability and also safe medical practice is realized, so has the need to develop teaching and assessment strategies for the same.

The traditional clinical apprenticeship model relying on experiential teaching and learning with subjective observational assessment of clinical skills has proven itself to be insufficient in terms of not providing enough opportunities for assessors to observe trainees in the course of clinical/technical encounter, hence limiting the evaluation of skills with little or no feedback.[1]

Major changes have been seen in the undergraduate, postgraduate medical as well as paramedical and allied education around the world. No longer is the old adage of “apprenticeship learning” valid and it has been seeing a lot of challenges in the recent past.[12] Different fields, especially the surgical modalities have realized the need for developing teaching and learning strategies and assessment tools to teach, evaluate and assess the procedural skills of the students. The major changes that have been seen are the introduction of the “workplace based assessments.” Workplace based assessment refers to “the assessment of day-to-day practices undertaken in the working environment.”

Although many forms of assessment can be used to show a doctor's/para medical staff knowledge or competence, there is evidence that competence does not reliably predict performance in clinical practice. One major advantage of workplace based assessment is its ability to evaluate the performance in the context.[3]

Workplace-based assessment has the advantage that it involves the observation and assessment of performance on naturalistic setting that is, clinical situations. Judgments are made on clinical procedural competence as well as other aspects of practice such as professionalism, decision making etc., with the provision of feedback to ensure achievement of a required standard. It targets the top two levels of Miller's pyramid that is from “shows how” to “does” [Figure 1].[4]

Figure 1

Millers pyramid of evaluation of competencies

Included in the workplace based assessments are the direct observation of procedural skills and objectively structured assessment of technical skills (OSATS). These have the provision and advantage that they can be used for assessment of the students' performance, provide feedback and identify areas and gaps in teaching and learning that require improvement.[1]

The Objective Structured Assessment of Technical Skills

The OSATS was developed by Martin et al. in 1997.[5] OSATS is a variation on the Objective Structured Clinical Examination. The OSATS is cited in the Accreditation Council of Graduate Medical Education evaluation toolbox on the website as the most desirable evaluation tool for the patient care topics including interviewing, counseling, preventive services, and performance of physical examinations. This can also be extended to include the technical skills.[6]

It offers structured, proficiency and competency based assessment of clinical skills in the work place for regulatory purpose and provides formative feedback and supervised training opportunity. The rationale is to assist and support learning that is, “assessment for learning” and this can be used as evidence of trainee's progress in the annual review that is an “assessment of learning.”[7]

The main objective of an OSTAS is to describe the physical tasks and the subsequent steps that are involved in the decision making process of the performance. In actuality it provides a knowledge base which then facilitates learning. This ultimately leads to the avoidance of mistakes and risks in the future.

Nuclear medicine has various technical procedures which require competence in terms of technical skills and also require knowledge of theoretical and practical aspects. Some of the technical skills to consider here are

Elution of the technetium generator

Formulation of the radiopharmaceuticals

Management of the radioactive spill

Quality control of the formulated kits

Quality control of gamma camera instrumentation.

As in all other fields the successful outcome is broadly defined as completion of a task and attaining the desired end point without complications.

In NM, the OSATS in this regard can prove to be helpful in the teaching of the various skills to the students.

Objectively Structured Assessment of Technical Skills in the Management of the Radioactive Spills

Extensive search was carried out to search for any available training method on the subject. Literature is replete with a description of the various types of nuclear spills and their management; however no formal teaching and evaluation scheme was identified.[891011]

It is the idea of this paper to develop OSATS for the radioactive spill, based upon the “SOP” and use it as a formative and then as a summative assessment for students and technical staff. Effort has been made to develop a task sheet (OSATS) for the purpose of formative evaluation and feedback [Annexure 1]. This can, then subsequently, be used as a summative tool in the training of the residents and technical staff. It will not only help in the evaluation of learning, but also provide feedback and identify areas for improving performance and filling in identified gaps.

Development of the item was made from the best practices guidelines available at different resources and expert opinion.[1011121314]

Every effort was made to include, very precisely and succinctly all crucial steps of the process.

Special Considerations

Before the induction of this teaching and assessment format, it must be remembered that a formal education to the process and management of the radioactive spill should be given didactically as well as practically by the instructing faculty (credentialed NM physicians and radiation protection officer).

The OSATS is performed in accordance to the steps in the form developed for the purpose (OSATS rating scale) and which has been taught accordingly. The steps should to be observed by a faculty/radiation protection officer and scored for the purpose of providing feedback for formative assessment.

Feedback is for identification of agreed strengths and areas for development. The weak areas should be overcome with further dedicated learning, observation and practice. This feedback should be given immediate after the assessment.

Here, it is important to realize that the training of the assessor regarding the various aspects (observation, scoring, avoiding and preventing biases in scoring etc.) are also important for the proper evaluation and ability to provide feedback. The assessor must be familiar with the form developed for the OSATS and have the expertise for the procedure being performed.

Rating Scale

The scale used for the developed OSATS was the Juster scale which is a variant of the adjective scale that is, which uses descriptors along a continuum. It combines adjectival descriptors of probabilities with numerical ones and have superior psychometric properties. The reason for the use of this scale is that it is said that the attainment of a technical skill lies on a continuum and if a categorical judgment or the check list is used, it may result in measurement error because of limited choice of response levels. This will lead to loss of efficiency of the instrument and a decrease in its correlation with other measures and hence reduces reliability.[615]

Errors of rating

There are common errors that have been noted in the ratings by the observer and special precautions are needed to avoid them. These errors include personal bias, halo effect, and logical errors.[16]

Personal bias errors[2]

These errors occur when the rater develops a tendency to rate all residents at approximately the same position on the scale. The problem arising from such a rating is that reliable discriminations are not provided due to scores being so close to each other. This can be overcome by a proper education of the assessing faculty and the realization that this error will, despite all effort, remain.

Generosity error

Rater tends to rate all at the high end of the scale.

Severity error

Rater prefers the lower end of the scale for candidates.

Central tendency error

Rater avoids both extremes and rates all as average.

The above three can be avoided by educating the rating faculty adequately.

Halo effect[2]

This error results from a, previous general impression of the candidate on the rater. If the impression is good, the rater inevitably scores the student at a higher level and vice versa, regardless of the actual performance.

As the identity concealing cannot be avoided here, one should realize and be aware that personal bias and prejudice exist and that they may “color” ones judgment. This can then help prevent this effect.

Logical error[2]

This occurs when two characteristics are rated as more or less alike because of rater's belief concerning their relationship. This error is due to a rater's belief and assumptions of relationship between traits that does not actually exist. It is a virtual belief that a student good at one thing will invariably good at the other. This type of error can be reduced by a properly designed assessment.

Statistical Tests towards the Analysis of the Tool

Reliability[17]

Reliability has been defined as the “reproducibility of the assessment data/scores overt time or occasion.”[17]

To test for reliability of the assessment tool the following tests should be performed:

Inter rater

This is assessed when two raters assess same procedure at the same time or an equivalent one performed after a time interval. This can be determined by comparing two raters evaluating the same or equivalent procedure of the same student.

Inter-item reliability

Internal consistency refers to, and assesses the degree of consistency among the items in a scale, or the different observations used to derive a score. Most of the items selected should have a similar level of difficulty.

Validity

The fundamental concept refers to the validity as “whether a test, or a measurement instrument, measures what it is supposed to measure.”[1819] To establish validity various tests are done. For the purpose of establishing the validity for the OSATS under discussion, the following were carried out.

Face validity

Face validity implies that the assessment in question seems right to the reader.

As a check on face validity, test items were sent to teachers to obtain suggestions for modification. Due to its vagueness and subjectivity, psychometricians have abandoned this concept of long.[2]

Content validity

Content validity means that the assessment should measure the extent of knowledge that it is intended to measure that is, it should contain material that should be present for the training it intends to impart? In content validity, evidence is obtained by looking for agreement in judgments by judges. To ensure coverage of content validity the assessment tool had been sent and approved by experts in the field.

Predictive validity

Predictive validity determines how well the student will perform the technical skills during the remaining training period and as an independent NM physician or staff technologist. This should be tested in revisits of the test in the assessments of the following years as well as the key performance indicators. A multisource feedback as well as annual appraisals will strengthen this validity.