Metrics Development for Minimal Invasive Unilateral Laminotomy for Bilateral Decompression of Lumbar Spinal Stenosis With and Without Spondylolisthesis by an International Expert Panel.

STUDY
DESIGN: Prospective study.
OBJECTIVES: To develop, operationally define, and seek consensus from procedure experts on the metrics that best characterize a reference approach to the performance of a minimally invasive unilateral laminotomy for bilateral decompression (ULBD) for lumbar spinal stenosis.
METHODS: A Metrics Group consisting of 3 experienced spine surgeons (2 neurosurgeons, 1 orthopedic surgeon), each with over 25 years of clinical practice, and an educational expert formed the Metrics Group that characterized a lumbar decompression surgery for spinal stenosis as a "reference" procedure. In a modified Delphi panel, 26 spine surgeons from 14 countries critiqued these metrics and their operational definitions before reaching consensus.
RESULTS: Performance metrics consisting of 6 phases with 42 steps, 21 errors, and 17 sentinel errors were identified that characterize the procedure. During the peer review, these were evaluated, modified, and agreed.
CONCLUSIONS: Surgical procedures can be broken down into elemental tasks necessary for the safe and effective completion of a reference approach to a specified surgical procedure. Spinal experts from 16 countries reached consensus on performance metrics for the procedure. This metric-based characterization can be used in a training curriculum and also for assessment of training and performance in clinical practice.

Introduction

Neurosurgical and orthopedic spine surgery programs rely on cadaveric and operating room training as their standard teaching methods. As these training forms can be limited by either institutional infrastructure and/or biohazard regulations with operating room time even more limited by work time regulations, legal issues, and high costs.[1,2] Furthermore, the surgical learning curve is associated with an increased risk of complication and higher morbidity and mortality for the patient.[3,4] Given that cadavers and animals tend to have different characteristics when compared to a living patient training options of specific procedure steps on cadavers or animal models have limitations and new options would be valuable.

Surgical training, at both trainee and master levels, intends to enable a surgeon to acquire the theoretical and practical knowledge necessary to perform a designated surgical procedure skillfully, reliably, and safely. Many experienced surgeons who expertly perform a specific procedure are able to identify and agree on the essential “steps” to be completed and “errors” to be avoided, whereas they rarely think about the procedures they perform with the level of detail needed to identify those key features.[5]

By psychologist definition those key features have to be subjected to a detailed task analysis specifying the order, duration, and result of a specific action and determine them to be unambiguously measurable to be called metrics.[6] The metrics do not have to capture every aspect of performance, but should allow differentiation in the way developed by Dreyfus and Dreyfus[7] to serve as a tool to evaluate effectiveness with a quantitative score.

The systematic implementation of a simulation curriculum in neurosurgery or orthopedic training programs has been proven to be feasible, is favorably regarded, and has a positive impact on trainees of all levels.[8] Therefore, a few academic centers around the world have focused on the role of highly realistic spine surgery simulation models to train spine surgeons[9-14] and even though simulation training is not a substitute for operating room experience, it may decrease the time teaching intraoperatively and reduce the morbidity secondary to resident surgical errors.

Simulation-based medical education with deliberate practice was proven to work even better in technical skill acquisition and maintenance for several clinical skills than traditional apprenticeship-styled clinical education.[15]

The aim of this study was (1) to develop and objectively define performance metrics that comprehensively characterize a minimal invasive unilateral laminotomy for bilateral decompression of lumbar spinal stenosis with and without spondylolisthesis (Figures 1 and 2), (2) to see if the metrics are assessible on a simulator, and (3) to seek consensus on the completeness and appropriateness of the metrics from a panel of experienced orthopedic and neurological surgeons.

Figure 1.

Illustration of the incision site on the right side for a L4/5 minimal invasive unilateral laminotomy for bilateral decompression of lumbar spinal stenosis.

Figure 2.

Illustration demonstrating the removal of ligamentum flavum through a tubular retractor as part of a unilateral laminotomy for bilateral decompression of lumbar spinal stenosis.

Material and Methods

Metric Definitions

As described before metrics are composed of different parts, such as phases, steps, errors and sentinel errors.[16]

Metric: A standard of measurement of quantitative assessments used for objective evaluations to make comparisons or to track performance.

Procedure phase: A group or series of integrally related events or actions that, when combined with other phases, make up or constitute a complete operative procedure.

Step: A component task, the series aggregate of which constitutes the completion of a specific procedure.

Error: A deviation from optimal performance.

Sentinel error: An occurrence involving a serious deviation from optimal performance of a procedure that exposes the patient to serious risk but falls short of being a critical error (ie, life threatening)

Metrics Development

Three experienced spine surgeons (2 neurosurgeons, 1 orthopedic surgeon), each with over 25 years of clinical practice, and an educational expert formed the Metrics Group that characterized a lumbar decompression surgery for spinal stenosis as a “reference” procedure. In the first step, all 3 experts individually described their classic lumbar decompression procedure. As it became evident that this resulted in a nondetailed description with just few steps identified, the next task was to explain it as they would explain it to a resident or fellow. Besides seeking consensus in the description and reviewing the current literature,[17-20] a series of 9 videos of lumbar decompressions available online (Google, YouTube, VuMedi, PubMed) were reviewed in detail. Based on a detailed task analysis and deconstruction process[6] as well as decades of experience practicing and teaching spinal surgery the units of performance that are integral to skilled task performance were identified. During individual face-to-face meetings, phone calls, and four 1.5- to 2-hour online conferences along with countless email exchanges metric units were identified and then unambiguously defined so that they could be subsequently scored as either occurring or not occurring with a high degree of reliability. For the online sessions, the use of Skype videoconferencing (Microsoft, Redmond, WA; available at www.skype.com) enabled the investigators who reside in different geographic locations to look at videos simultaneously and discuss results facing the same screens. The metrics included the phases in general order as well as the specific steps and the instruments used. The metrics also described for each procedural step what should not be done (errors) thus characterizing behavior that deviated from optimal performance.[21] The intent again, was to create unambiguous operational definitions (rather than descriptions) for each performance unit namely procedure steps, errors, and sentinel errors. The goal was to characterize a “reference” minimal invasive decompression procedure without unusual or complex pathology and subsequent unexpected obstacles.

Metric Stress Testing and Reliability of Identification

With the Metrics Group content with characterization of the procedure, the metrics were “stress tested” by subjecting them to an analysis of reliability assessing how independently scored in blinded fashion. Eight video recordings of complete minimal invasive decompression procedures that were performed by surgeons using either tubular or spatula retractors were independently reviewed and scored in binary fashion as either occurring or not. Differences in the scoring of each metric by the reviewers were discussed and if necessary, definitions of steps and errors were clarified, with the items modified, or dropped and new ones added until the Metrics Group was satisfied that the metrics accurately and unambiguously characterized the specifics of a minimal invasive decompression procedure. The extent of agreement between 2 raters ranged between 0 = no agreement and 1 = complete agreement.

Experts Perform the Procedure to Evaluate the Metrics

To test the metrics further each expert performed 2 procedures on a simulator while being scored individually by the other 2 using the metrics. The procedures were recorded, and the records reviewed by all experts conjointly. Differences in the results and scores once again were discussed with steps missing or unnecessary deleted or modified in 3 phases (Table 1).

Table 1.

The Steps of the Reference Procedure of Minimal Invasive Unilateral Laminotomy for Bilateral Decompression of Lumbar Spinal Stenosis and the Changes Agreed on After Expert’s Procedure.

Procedure Phase	Name	Steps Before Experts’ Procedures	Steps After Experts’ Procedures	Added	Deleted	Modified
I	Starting point, skin incision, and access	8	8	0	0	1
II	Identification of ipsilateral hemilamina, ipsilateral hemilaminotomy, and identification of ligamentum flavum	9	9	0	0	1
III	Flavectomy	4	4	0	0	0
IV	Complete ipsilateral decompression	9	8	0	1	0
V	Contralateral hemilaminotomy	11	11	0	0	0
VI	Hemostasis and closure	5	5	0	0	1

Modified Delphi Panel to Evaluate the Metrics

The metrics were then sent for appraisal to a group of surgeons who were also expert in the performance of the procedure. A questionnaire was developed by the Metrics Group seeking consensus for each of the procedural steps and potential errors that needed to be evaluated individually. The questionnaire was then sent by the educational specialist via SurveyMonkey (Dublin, Ireland; available at http://surverymonkey.com) to every surgeon who agreed to participate.

While the metrics developed by the Metrics Group for a reference procedure for unilateral approach to a bilateral decompression for lumbar stenosis might not depict the exact techniques used by the individual panelists, it was agreed that the operative steps accurately reflect the essential components of the procedure.

An affirmative vote by a participant indicated that the metric definition presented was accurate and acceptable as written but not necessarily that it was the manner in which that particular panelist might have chosen to complete the step. Therefore, panel members were encouraged to comment on the metrics even though they might not perform the steps in the exact same way. “Consensus” meant that panel members voted unanimity and that a particular metric definition was “not wrong or inappropriate.”

Subjects

Twenty-six experienced spine surgeons from 14 countries (Table 2) were asked to take part in an expert panel. Participants were selected (to ensure regional representation) from the international and regional faculty of AOSpine as well as Society for Minimally Invasive Spine Surgery (SMISS) and by their experience in minimally invasive spine surgery providing an equal distribution of orthopedic surgeons and neurosurgeons.

Table 2.

Number of Surgeons From Each Country Represented in the Delphi Panel.

Country	No. of Surgeons
Australia	1
Austria	1
Brazil	2
Canada	1
Colombia	1
Egypt	1
France	1
Germany	3
Italy	1
Jordan	1
South Korea	1
Spain	1
United Kingdom	1
United States	10

Results

Procedure Phases

The procedural metrics resulting from task deconstruction were grouped into 6 phases. Each phase (eg, “Identification of ipsilateral hemilamina ipsilateral hemilaminotomy, and identification of ligamentum flavum” or “Contralateral decompression”) contains a series of related, unambiguously defined, observable procedure steps with specific beginning and ending points (Table 3).

Table 3.

The Operational Definitions of the Beginning and End of the Different Phases of the Reference Procedure of Minimal Invasive Unilateral Laminotomy for Bilateral Decompression of Lumbar Spinal Stenosis.

Procedure Phase	Name	Begins and Ends
I	Starting point, skin incision, and access	Begins	Preparation phase above has been completed
		Ends	Retractor positioned on bone in correct level
II	Identification of ipsilateral hemilamina, ipsilateral hemilaminotomy, and identification of ligamentum flavum	Begins	Surgeon ready to identify lamina
		Ends	The ligamentum flavum has been identified
III	Flavectomy	Begins	Phase II completed
		Ends	Central portion of ipsilateral ligamentum flavum has been removed
IV	Complete ipsilateral decompression	Begins	Ipsilateral medial flavum has been removed
		Ends	Lateral recess is opened
V	Contralateral hemilaminotomy	Begins	Ready to start contralateral decompression
		Ends	Contralateral decompression and flavectomy performed
VI	Hemostasis and closure	Begins	Hemilaminotomies have been performed
		Ends	Hemostasis applied and wound closed

By processing the information from the expert operations and the Delphi panel survey all phases were accepted with changes, but no extra phase was proposed or was a phase deleted (Table 4) while phase V was renamed. Phases I and VI were agreed on with just linguistic changes, while wording and content was altered within the other phases (II-V)

Table 4.

The Procedural Phases of the Reference Procedure of Minimal Invasive Unilateral Laminotomy for Bilateral Decompression of Lumbar Spinal Stenosis.

Procedure Phase	Name
I	Starting point, skin incision, and access
II	Identification of ipsilateral hemilamina, ipsilateral hemilaminotomy, and identification of ligamentum flavum
III	Flavectomy
IV	Complete ipsilateral decompression
V	Contralateral decompression
VI	Hemostasis and closure

Procedure Steps

Within the reference procedure, 44 steps were identified by the Metrics Group. After the expert operations and reevaluation of the steps, 1 was added, 1 was deleted, and 3 were modified. These were distributed unequally within the phases with the most occurring in the fifth phase (“contralateral decompression”). After discussion of the feedback from the expert panel, 2 steps were deleted and 11 were modified. (Table 5).

Table 5.

The Steps of the Reference Procedure of Minimal Invasive Unilateral Laminotomy for Bilateral Decompression of Lumbar Spinal Stenosis and the Changes Agreed and Voted on by the Delphi Panel.

Procedure Phase	Name	Steps Before Delphi	Steps After Delphi	Added	Deleted	Modified
I	Starting point, skin incision, and access	8	8	0	0	3
II	Identification of ipsilateral hemilamina, ipsilateral hemilaminotomy, and identification of ligamentum flavum	9	9	0	0	0
III	Flavectomy	4	4	0	0	3
IV	Complete ipsilateral decompression	7	6	0	1	1
V	Contralateral hemilaminotomy	11	10	0	11	2
VI	Hemostasis and closure	5	5	0	0	2

Procedure Errors and Sentinel Errors

All potential errors identified had been noted to occur during the stress testing of the metrics. Identical errors and sentinel errors could occur during different phases of the procedure (eg, “dural tear with CSF leak”), but were named within the phases individually. Overall, 18 errors and 20 sentinel errors were defined by the Metrics Group and affirmed. No errors and sentinel errors were added, but 1 sentinel error was deleted and 1 sentinel error occurring in 2 phases was changed to an error instead of a sentinel error. Furthermore, changes were made in 3 errors adding information or changing wording. In the end, 21 errors and 17 sentinel errors were defined (Tables 6 and 7).

Table 6.

The Errors of the Reference Procedure of Minimal Invasive Unilateral Laminotomy for Bilateral Decompression of Lumbar Spinal Stenosis and the Changes Agreed and Voted on by the Delphi Panel.

Procedure Phase	Name	Errors Before Delphi	Errors After Delphi	Added	Deleted	Modified
I	Starting point, skin incision, and access	2	2	0	0	0
II	Identification of ipsilateral hemilamina, ipsilateral hemilaminotomy, and identification of ligamentum flavum	6	7	1	0	0
III	Flavectomy	3	3	0	0	0
IV	Complete ipsilateral decompression	2	3	1	0	0
V	Contralateral hemilaminotomy	4	5	1	0	0
VI	Hemostasis and closure	1	1	0	0	0

Table 7.

The Sentinel Errors of the Reference Procedure of Minimal Invasive Unilateral Laminotomy for Bilateral Decompression of Lumbar Spinal Stenosis and the Changes Agreed and Voted on by the Delphi Panel.

Procedure Phase	Name	Sentinel Errors Before Delphi	Sentinel Errors After Delphi	Added	Deleted	Modified
I	Starting point, skin incision, and access	2	1	0	1	0
II	Identification of ipsilateral hemilamina, ipsilateral hemilaminotomy, and identification of ligamentum flavum	5	4	0	1	0
III	Flavectomy	2	2	0	0	0
IV	Complete ipsilateral decompression	5	4	0	1	0
V	Contralateral hemilaminotomy	5	4	0	1	0
VI	Hemostasis and closure	1	1	0	0	0

After affirmation of the panel suggestions the metrics were agreed as an accurate characterization of a minimal invasive unilateral laminotomy for bilateral decompression for lumbar spinal stenosis. An example of the complete metrics and steps for a phase is shown in Figure 3.

Figure 3.

The agreed metrics for Phase IV of the reference procedure (starting point and entrance angle). Fields in white show the metrics that can be scored by an assessor.

Discussion

Assessment of medical education and competence is always difficult and becomes even more challenging within postgraduate training. Ideally, the assessment of competence should provide insight into actual performance as well as the capacity to find and generate new knowledge and improve overall performance. Competence is contextual, reflecting the relationship between a person’s abilities and the tasks he or she is required to perform in a particular situation in the real world.

Medical training and its assessment have to meet certain criteria. The first criterion, reliability, refers to the reproducibility, consistency, and generality of the results over time. The second criterion, feasibility, reasons that, in order to be accepted, a new tool should be cost-effective, easy to execute, and not time-consuming. The third criterion, validity, asks if the tool assesses what it is supposed to assess. Validity is divided into 5 subcategories: predictive validity, which asks if the tool can predict future performance; face validity, which asks if the tool reflects real life; construct validity, which asks if the tool measures what it is designed to measure; content validity, which asks if the domain that we are assessing is assessed by the tool; and criterion-related validity, which asks if the result assessed by a tool correlates with that measured by the current gold standard tool.

Metric development is therefore crucial as the metrics build the fundament of good medical training. Metrics define how the training should be characterized and the procedure performed by the trainee as well as grant the opportunity for assessment of the trainee’s performance and progress.

In contrast to other, mostly quantitative approaches, metrics provide for an individual not associated with the initial development to use them to score performance reliably as they provide precise definitions of performance and a binary scoring system.[6,22,23]

Especially training in spine surgery varies a lot due to regional and national differences as well as differences between neurosurgical and orthopedic approaches to surgical procedures. Furthermore, with duty hour limitations and regulatory pressure for enhanced quality and outcomes limitations occur in trainees’ access to fundamental skills training. In addition, a cultural shift is seen from competence to mastery learning.[24] Fundamental skills training as part of a curriculum might help mitigate this challenge to surgical education.[12,25] This is what we aim to achieve.

Conclusion

With a total of 29 surgical experts from 14 countries achieving consensus on the metric-based definition of a reference procedure, this study validates that definition and exact characterization of the phases and steps depicting a reference procedure for a minimal invasive unilateral laminotomy for bilateral decompression of lumbar spinal stenosis is possible. Furthermore, face and content validity for the resulting steps and error metrics can be obtained by simulator-based procedures by experts and through review by a panel of experts in a modified Delphi panel.