Structured and evidence-based training of technical skills in respiratory medicine and thoracic surgery.

Within the last decade, the number of technical procedures in respiratory medicine and thoracic surgery has grown and created a higher demand for effective and evidence-based education. Today, trainees are often allowed to perform procedures unsupervised on patients after obtaining a course certificate from a theoretic course and having performed a fixed number of supervised procedures. However, these methods do not ensure adequate competence. Well-structured and effective educational programmes including validated tests are needed to reduce economic expenses, optimize time spend, and ensure patient safety. The aim of this article is to summarize current state of educational strategies for technical procedures within respiratory medicine and thoracic surgery. Thus, to discuss future recommendations for curriculum development and assessment of competences based on Kern's framework. The approach by Kern consists of six topics, which needs to be considered and evaluated and in order to educate physicians and surgeons most effective and evidence-based. We present a practical guide contributing to future educators' considerations on (I) problem identification and general needs assessment, (II) targeted needs assessment, (III) goals and objectives, (IV) educational strategies, (V) implementation, and finally (VI) evaluation and feedback. 2021 Journal of Thoracic Disease. All rights reserved.

Introduction

Competent performance of technical procedures is essential for both thoracic surgeons and respiratory physicians (1-3). The list of technical thoracic procedures has expanded widely within the last decades due to an evolution of new technologies. This creates massive demands for obtaining new knowledge and skills of the physicians in respiratory medicine and thoracic surgery.

Previously, education and training in technical procedures have often taken place bed-side fully integrated as part of daily clinical work. The extensive growth in number and complexity of procedures combined with increased demand of efficacy, number of patients, and focus on patient safety have made training in new approaches necessary in order to benefit both patients and trainees (4). The traditional apprenticeship model, the “see one, do one, teach one”-strategy, where trainees watch the procedure being performed by experienced colleagues, and then are expected to have gained adequate competences and skills, is no longer considered adequate (5,6). Current guidelines from the international societies recommend volume thresholds when certifying trainees in technical thoracic procedures (2,7). Fixed numbers of supervised procedures prior to independent performance do however not ensure adequate competencies, because trainees differ in learning pace, motivation, and pre-existing skills. Larger syllabuses are often taught in courses, which extend from a couple of hours to days, and often as classroom-based didactic lectures, even though research proves interactive teaching models are far more effective (8). Theoretical knowledge of a given procedure is possible to gain from a lecture, but the necessary essential motor skills are not possible learn in a theoretic session.

Looking at the growing number of published literature within medical education, and education in technical procedures, the previous educational strategy is today not considered as best evidence medical education. Researchers have searched for structured and evidence-based ways of not only developing methods for education, but also ways of ensuring sufficient competence following education.

Kern has presented a six-step, systematic approach for curriculum development (9). The approach is today widely used, because it, in a practical, evidence-based way, emphasizes considerations on how an educational program should be developed, implemented and re-evaluated. It adheres to the requirements of a modern healthcare system, where education is time, and time is money. Kern’s six-step approach complies:

Problem identification and general needs assessment;

Targeted needs assessment;

Goals and objectives;

Educational strategies;

Implementation;

Evaluation and feedback.

To improve the effectiveness, educators need to evaluate and elaborate each step of the approach, see . The aim of this article is (I) to summarize current state of education in technical procedures in respiratory medicine and thoracic surgery, and (II) based on Kern’s framework, discuss and elaborate on future recommendations.

Figure 1

Kern’s six-step approach for continuing curriculum development. The steps are not consecutive, but should be evaluated fluently as curriculum development is a dynamic process.

Problem identification and general needs assessment

First step is the identification and analysis of the problem, issue or need, that is to be addressed by the curriculum (10). The identification should be based on a review of existing literature and available information on e.g., current postgraduate medical education, and the educators must elaborate on how the problem is currently being addressed. The difference between how the educational issue is addressed and how it should be addressed is the general needs assessment. The complete problem identification and needs assessment must be done at a national or international level in order to ensure generalizability.

Generally, national learning objectives, both technical and non-technical in respiratory medicine and thoracic surgery, are provided by the National Health Services in collaboration with the societies for the specific speciality, and statement of aims rely on the syllabus established by the international societies e.g., European Respiratory Society (ERS) or American Board of Thoracic Surgery (ABTS) (11-13). In most cases, the presented statements of aims, are lists of knowledge and skills needed in order to handle, diagnose or treat patients with pulmonary or thoracic diseases, but it is unclear what the evidence behind these lists are. A structured approach to identify, which technical procedures should be learned, is desired, rather than one expert’s opinion, financial interests of a company, or availability of simulators or phantoms for a specific technical procedure. Nayahangan et al. present a general needs assessment of procedures suitable for simulation-based training (SBT) (14). The stringent and structured study design is judiciously and deliberated, and future general needs assessment can advantageously be carried out as this study, which additionally encourages cross boarder collaboration in order to generalize curriculum development, and provide comparable results.

In March 2018 ERS published the “Update of the ERS international Adult Respiratory Medicine syllabus for postgraduate training” (11), and following in May 2018, the report of general needs assessment in thoracic surgery was published by the European Society of Thoracic Surgeons (ESTS) (15). The aim of the work and publications was to define minimum standards to address the heterogeneity of training across Europe, and enable a generalizable education in respiratory medicine and thoracic surgery. A Delphi-like process, as described by Nayahangan et al. (14), was in both cases performed.

No specific or definitive number of respondents is needed when performing a Delphi-process, it depends on the complexity of the topic evaluated, but response time and deadlines can influence the choice and the number of participants (16). It is recommended that the number of experts should be large enough to provide a representative gathering of the opinions on the content (16). It is unclear how many respondents were used in the ERS publication, but the process of the preparation of syllabus in thoracic surgery included 334 respondents (representing 36 countries) in first round. Trainee members of ESTS were invited, because the task force had experienced dissatisfaction among the younger surgeons due to missing education and training in specific procedures they found relevant. Therefore the trainees could contribute to the overall assessment of general needs relevant in a daily clinical work, even though a Delphi process is said to include experts only (16).

Hence, this way of identifying and prioritizing the educational modules and syllabus, meets the recommendations by Kern, and is a commonly used and favoured technique for problem identification and needs assessment, but requires adaption into the setting. The structured, international general needs assessments performed in respiratory medicine and thoracic surgery are available, and can form the foundation of targeted needs assessments on a national or institutional level, when having the geographical differences in disease prevalence, and resources available in mind.

Targeted needs assessment

The definition of the targeted needs assessment, is “the process by which curriculum developers apply the knowledge learned from the general needs assessment to their particular learners and learning environment” (9). Main focus on this step is to identify the target trainees and targeted learning environment, e.g., are the target trainees of the curriculum in the specific technical procedure junior surgeon residents, or experienced surgeon with competences in similar procedures. A well-done targeted needs assessment prevents inappropriate decisions that can cause additional work, bad feedback, and continuing restructuring with massive work load. Examples of possible avoidable issues/problems that could have been solved with a sufficient targeted needs assessment are; duplicated objectives and tasks (e.g., which is taught in another course), or wrong aligned level of knowledge and skills (e.g., the tasks are either too easy or too difficult to the targeted trainees). If such problems arise, it will decrease trainees’ motivation, the outcome benefit will be lower than expected, and the financial expense will be wasted (10). Therefore, a curriculum developer should identify resources to align the strategy and assess the targeted trainees’ experiences, expectations, existing proficiencies (including cognitive, affective and psychomotor), and the environmental opportunities (already existing curriculum, needs of stakeholders, financial barriers, and resources). Understanding the targeted trainees’ environment also contributes to understanding trainees’ behaviour and attitude towards a specific procedure or skill. Different methods are being used for targeted needs assessment, and can be done by questionnaires, tests, focus groups interviews or observations (9). These methods can be time consuming but sufficient considerations and reflections can benefit the next parts of the curriculum development, and at the end be timesaving and cost-effective.

Goals and objectives

When relevant procedures are clarified for the targeted trainees and targeted environment, characterization of the structured goals have to be made. The goals are often broad educational objectives of the curriculum, and describe an end toward which, an effort is directed, and are usually not measureable (10). Contrary, in order to prove the effect of an educational intervention, clear measureable objectives (and outcome measures) are needed, and may include cognitive, affective and psychomotor skill objectives to the trainee, especially if the curriculum covers technical procedures.

Presented in is a modified version of the framework for assessing clinical competences by George Miller, Millers’ pyramid (17). The lower level represents the basic facts (knows), followed by the applied knowledge (knows how), the hands-on (shows how), and performance (does). The framework acknowledges the consecutive development when learning a new skill with increased complexity when moving towards the top, and invites to the use of competency-based learning and assessment.

Figure 2

A modified version of Miller’s pyramid for assessing clinical competences.

Trainees have different and individual learning paces, as well as different prerequisites when training a new technical procedure. Competency-based training allows these differences so that all trainees reach same adequate, and pre-defined level of proficiency for a given task; some trainees just use more time or more guidance in order to reach the level. Thus, when writing objectives, considerations must be done on, which step of the Miller’s pyramid the procedure is being taught. It is essential prior to implementation of measurable assessment tools, and additionally if the assessment tool should be implemented as formative or summative test, during the training or at the end. A theoretical test as a measure of a practical competence is not proper, and a practical test in a simulation-based environment does not equal perspective to other findings in a clinical setting.

Already published literature has contributed with assessment tools in technical procedures with solid validity evidence in respiratory medicine and thoracic surgery (18-22). Savran et al. have developed theoretical tests in endosonography and video-assisted thoracoscopic surgery (VATS) lobectomy, because acquisition of sufficient theoretical knowledge provides trainees with the necessary foundation to develop technical skills (19,23). SBT may be a flexible and effective approach for training technical skills. Konge et al. present a study using simulation as a tool for assessment of competences in endobronchial ultrasound (EBUS) (18). The trainees in the study had to identify six anatomical landmarks followed by two transbronchial fine-needle aspirations of lymph node stations, and an EBUS-expert supervised the trainees and assessed competences by a structured score sheet. Jensen et al. developed and gathered validity evidence for the first virtual reality simulator for VATS lobectomy (24). The model is a right upper lobe lobectomy based on the principles of a standardized anterior approach including dissection, division of vessels, fissures and bronchus with staplers (25). Today, new software in the simulators makes it possible to assess technical proficiency, progression, and provide reliable and structured feedback. Because of this, it is possible for the trainee to restrict the training effort on the pitfalls identified until an adequate level is obtained and then subsequently proceed to a clinical setting. SBT are therefore able to replace the apprenticeship training in the beginning of the learning curve, but does however not provide experts in a clinical setting. A structured assessment tool with proven solid validity evidence has been developed for assessment of competences in bronchoscopy in a clinical setting (20). The assessment tool includes several aspects of a technical procedure; anatomical knowledge, technical skills, and the ability to describe the procedure. Thus, the complexity has further increased when compared to the simulation-based assessment. As an example the tool for assessment of competences in thoracic ultrasound, also evaluates on the ability to reflect on indications for the procedure (e.g., tentative diagnoses, information to the patient), and conclusions based on the procure (e.g., integrate the examination findings with the patient history) (26). The VATSAT (VATS assessment tool) was developed by Jensen et al. using the Delphi method and a group of international VATS experts. It consists of eight items with a score between one and five (27). Petersen et al. provided validity evidence for VATSAT in a real clinical situation with VATS lobectomies performed in a Danish nationwide study. Video recordings of 60 VATS lobectomies performed at all four centers of thoracic surgery in Denmark by 18 thoracic surgeons were rated using the VATSAT. The two raters were blinded to surgeon and center (28). Observational assessment in a clinical setting is still needed, and can be done by direct observation using a structured tool filled out by an experienced supervisor.

Gathering validity evidence for assessment tools is vital to their use (29). A significant issue with the increasing number of publications in medical education is the lack of proven validity evidence, wrong or outdated methods (30). Downing and Yudkowsky describe Validity as the single most important characteristic of assessment data, and that “validity refers to the evidence presented to support or refute the meaning or interpretation assigned to assessment data (test)” (29). So, gathering validity evidence for a test is done to ensure that the assessment tool or test measures what it is meant to measure, and in a meaningful way. It ensures that no trainee with adequate level fails, and then spend more time than needed practising, costing the institution time and money unnecessarily. It also ensures that no trainee who does not possess adequate level of competence pass, and potentially put patients at risk by mistakes or wrong interpretations (29,31,32). Pass/fail standards are needed for summative tests, and is an essential part of exploring validity evidence (33).

So, in conclusion, when determining the educational objectives for a curriculum of a technical procedure, considerations must encompass to which level the procedure should be learned, and how should the development and skills be measured.

Educational strategies

Once the goals and objectives have been determined, the next step is to consider which educational strategy is more favourable to accomplish goals and objectives. It is important to consider both the content of the curriculum, and the educational methods. Maintenance of congruence between the objectives and educational methods are crucial, but so are feasible methods in terms of resources and administrative possible solutions. Involved educators must decide how much time and effort they will use on the educational program. Studies have proved that online or web-based sessions like podcasts, are non-inferior to traditional classroom-based education in terms of obtainment of theoretical knowledge, and today, trainees are used to navigate online and respond to web-based learning by increasing satisfaction (34-36). Workload of both educators and trainees is more flexible when using web-based learning, the educators avoid long confrontation hours, which reduce already discounted hours away from the clinical work, but still, this method requires continuously evaluation in order to keep material up to date (36).

Traditionally, courses are being held as classroom-based courses, introducing new technical procedures to the trainees. The trainees then repeat and train the procedure for the day, and are then ready to implement the procedure in daily practice. First of all, a fixed timeframe or a fixed number of procedures do not ensure adequate competences. Secondly, the effect of whole-day courses is inferior to shorter task training sessions (37), and thirdly, if the procedure trained does not correspond to the daily work done by the trainee, the effect could potentially be minimized. One potential way to address these issues is by using SBT. Several studies within respiratory medicine and thoracic surgery have proven that trainees can benefit from SBT and that is possible to use SBT for assessing competences in a solid and valid manner (18,21,38). SBT addresses the challenges of patient safety, making it possible for novices to train motor skills at the stage where the movements are slow and inconsistent, and where large parts of the movements are controlled consciously without using patients as hands-on training facilities () (39). Additionally, when the basic skills have evolved, and the movements are done automatically, it is possible to repeat and train more complex high-stake cases in a calm environment, which increases trainee motivation and confidence (40). The cost of acquiring a simulator can be significant, and SBT is not the answer to all issues in the teaching of technical procedures. The implementation and use of SBT must be deliberate and embedded into a full curriculum, because SBT is not suitable for knowledge acquisition (know) or for implementing the procedure into context in a clinical setting (Does from Miller’s pyramid). Konge et al. recommend a four-step approach for the development of competency in video-assisted thoracic surgery (VATS) lobectomy including SBT. The approach complies the basic principles of the mastery learning concept, and as seen in , covers the aspects of Miller’s pyramid (41). The trainees will train to mastery, and the training is individualized, thus, the depended variable is not the test score but numbers of examinations performed during the training period.

Figure 3

A trainee practices bronchoscopy using a simulator, supervised by an instructor in the Regional Center for Technical Simulation at Odense University Hospital, Denmark.

Figure 4

The four-step mastery learning approach for medical simulation-based training of technical procedures, recommended by Konge et al. (24). The approach is compared to the levels from Millers pyramid.

Implementation

Support and flexibility from the institutions are prerequisites in order to implement, and thus benefit from SBT as an educational strategy in a curriculum of technical procedures. The institutions must accept the time away from the clinical ward as an investment, which will pay off in effective time (working hours), decreased number of complications or calls for assistance, but the win is not directly and immediately measureable. The “Danish model” comprises four regional Simulation Centres, where physicians from hospitals and departments within the same region share the same models, simulators, instructors and material. Each centre evaluates the use and need of courses, and can decide to move a specific course to a smaller hospital for a period of time. The centres are more cost effective, than if each hospital should offer courses within the same procedures, and the model also contributes to more uniform education and the possibility for educational research.

It is a necessary process to go through all the steps in Kerns’ model, because curriculum development is time-consuming, and therefore the result must be validated, systematic and as effective as possible. The infrastructure and technology e.g. simulators, are not enough to comply the demands of teaching technical procedures. A challenge is to effectively utilize the tools whether it is simulators, phantoms, or other devices used for education. A major component of the challenge lies in attracting institutional members to apply this strategy of education. Several technical procedures within respiratory medicine and thoracic surgery are also done by physicians from other specialities, e.g., pleural drainage performed by anaesthesiologists, and thoracic ultrasound performed by emergency physicians (42,43). One method to streamline, and make a curriculum more generalizable, and thus cost beneficial, is to explore the multi-speciality procedures. The utilization of resources could be increased if these procedures are identified and the needs assessments and subsequently steps are done with a broader variety of specialities.

A benefit of SBT when considering the implementation strategy, is the possibility of self-directed training. Self-directed learning is trainee-centred, individualized, and flexible to working hours. The trainee can be introduced to the procedure and simulator by the expert (educator), and is afterwards able to train on their own or supervised by an institute member or medical student. Many simulators are able to assess the trainee automatically which allows identification of pitfalls or incomplete whole or part tasks, and the expert supervision can be saved for more complex tasks, or for assessment after implementation of the procedure in a clinical setting. A well-defined simulation-based curriculum in endobronchial ultrasound-guided transbronchial needle aspiration (EBUS-TBNA) has been implemented using medical students as supervisors during self-directed SBT (22,44). Studies are needed to sustain the fact that experts are not necessary during SBT to acquire the greatest possible benefit from SBT—actually, self-directed learning might increase retention (45).

A recurrent issue in published literature within medical education is the patient-related outcome or benefit of the educational intervention (46). Such educational interventions are hard to measure because of the irregular links that potentially exist between the intervention and patient outcome. Research warns against relying on surrogate measures, but no persuasive and convincing methods have been presented in order to address this issue.

Evaluation and feedback

The sixth step “Evaluation and feedback” closes the loop in the curriculum development cycle, and provides information to guide the curriculum or individual into improvement so that (I) the individual meets the objectives defined in step III, and (II) the curriculum evolves and remain up-to-date (10). Kirkpatrick developed a model for evaluating and analyzing educational programs, which is widely used in the assessment of educational interventions (47). The model describes four levels (). Level 1 is the trainees’ reaction, and trainees’ satisfaction with the educational intervention. The objective of this level is a straightforward quantitative analysis, but it is important to remember that an optimistic reaction to a curriculum does not equal a positive learning outcome, but an unfavourable reaction definitely makes it less likely that the trainee paid attention to the curriculum. Level two of the learning outcome criteria, is the evaluation of obtained knowledge, skills and behaviour (individuals meet the objectives). The one-group pre-test and post-test design is often used (48-50), but is today considered obsolete due to minimal internal and low external validity (51). When using assessment tools, like described under Goals and objectives, including minimum pass/fails standards, the evaluation of learning can be assessed effectively and evidence-based. The evaluation of behaviour (Level 3) assess whether the skill or competence is transferred to a clinical setting. Trainees must apply the learning into context, and change behaviour either immediately or as a continuous remodelling if the teaching of a technical procedure should be effective. Clear design on how this outcome should be measured, is highly relevant in order to decrease the subjective judgement by educators related to the educational programme, which must be expected as a factor that can affect the reliability and consistency of the evaluation. The use of e.g., identical SBT with uniform assessment tools across respiratory medicine and thoracic surgery, can lead to a large amount of data for future research in the teaching of technical thoracic procedures. Hopefully, the gathering of comparable data can also make the patient-related outcome more feasible.

Figure 5

The four levels of educational evaluation (Kirkpatrick’s model).

Conclusions

In the future, structured curricula have to be developed when introducing and educating trainees to perform new thoracic technical procedures. ERS and ESTS have made a substantial contribution by performing the needs assessments, and this work can form the foundation to forthcoming programs, but educators have to elaborate on more than just the topic in order to establish a structured, evidence-based educational program. Clear goals and objectives must be established, assessment tools with solid validity evidence are crucial, and educational strategies must be decided with available recourses in mind.

The authors have presented a practical guide, using a recognized framework, on how the development can be manufactured, and the future can potentially be as told by Vozenilek “see one, simulate many, do one competently and teach many” (4).

The article’s supplementary files as