Protocolized Training of Advanced Practice Providers for Robotic Surgery Improves the Quality of Intraoperative Assistance.

BACKGROUND: The expansion of robotic surgery requires identifying factors of competent robotic bedside assisting. Surgical trainees desire more robotic console time, and we hypothesized that protocolized robotic surgery bedside training could equip Advanced Practice Providers (APPs) to meet this growing need. No standardized precedent exists for training APPs.
METHODS: We designed a pilot study consisting of didactic and clinical skills. APPs completed didactic tests followed by proctored clinical skills checklists intraoperatively. Operating surgeons scored trainees with 10-point Likert scale (< 5 not confident, > 5 = confident). APPs scoring > 5 advanced to a solo practicum. Competence was defined as: didactic test score > 75th percentile, completing < 5 checklists, scoring > 5 on the practicum. The probability of passing the practicum was calculated with Bayes theorem.
RESULTS: Of 10 APP trainees, 5 passed on initial attempt. After individualized development plans, 4 passed retesting. Differences in trainee factors were not statistically significant, but the probability of passing the practicum was < 50% if more than four checklists were needed.
CONCLUSIONS: Clinical experience, not didactic knowledge, determines the probability of intraoperative competence. Increasing clinical proctoring did not result in higher probability of competence. Early identification of APPs needing individualized improvement increases the proportion of competent APPs.

INTRODUCTION

Utilization of the robotic platform for surgical procedures has increased dramatically over the last 10 years. Reasons include increasing number of surgical trainees gaining exposure to this approach, faculty adoption of robotic instrumentation allowing more complex minimally invasive procedures, and patient demand. Accordingly, surgery programs have recruited robotically trained surgeons to meet clinical demand and provide educational opportunities for surgical trainees.

Unfortunately, surgical trainees are often relegated to bedside assistance during robotic procedures due to lack of designated bedside assistants. In our center, surgical oncology fellows are board certified general surgeons, and many seek additional training to apply robotic surgical skills to complex oncologic procedures in their future careers. Community robotic surgery programs without surgical trainees have met the need for robotic bedside assistance with Advanced Practice Providers (APPs).[1] APPs represent a resource that can provide quality surgical assistance and allow surgeons in training more robotic console time. Having a consistent APP available for operating room (OR) preparation (positioning and room-set up) and intraoperative postoperative care (closing incision, room turnover, and patient transitions) helps to decrease overall surgical time.[2] Even APP specific training programs are recognizing the importance of exposure to robotic surgery into their physician assistant training program.[3]

Very little has been written about determining APP competency for robotic surgical assistance, and much information comes from established community robotic surgery programs. The basic roles of a robotic APP first assistant include: placing robotic ports, docking the patient cart, safe exchange of instruments, retracting tissues, suctioning and irrigating, retrieving specimens, and closing skin incisions.[4,5] Expert opinion from surgeons who have created robotic surgical programs cite potential benefits from having physician assistants as first assists. In comparison to using a second surgeon in the OR, there is now an increasing trend of utilizing APPs in the OR.[6] The perceived benefits include an economic benefit by freeing up a second surgeon from serving as first assistant, reproducible quality from a consistent APP first assistant, and the potential educational role for APPs to training residents as bedside assists.[1] One author described a clinical pathway for training bedside assists accomplished in three phases. Phase 1 is didactic online education regarding system basics and troubleshooting. Phase 2 is hands-on system training involving docking, port placement, instrumentation and exchanges in a dry lab, and eight hours with a training specialist in a wet lab where the assistant must meet all requirements with a post assessment checklist. Phase 3 is clinical practice of the skills acquired in the OR.[4] At our institution, APPs are intricately involved in the entire care continuum of cancer patients; from outpatient clinic encounters to intraoperative assistance with open and laparoscopic surgery, to inpatient and postoperative management. However, no formal precedent exists for training APPs on the robot. Therefore, we designed a pilot program based on the model described above.

The purpose of this study is to examine the outcomes of a protocolized APP first assistant robotic surgery pilot program developed to quantify competence. Our hypothesis is that competency will be associated with the frequency of clinical exposures, and the primary goal of our pilot is to quantify the number of clinical exposures needed to achieve competency to first assist.

MATERIALS AND METHODS

Screening by the Quality Improvement Assessment Board was performed, and no formal review was needed. We identified APPs in the department of surgical oncology who were interested in formalized robotic surgical training. All APPs with OR responsibilities were eligible for the pilot. Recent graduation, limited clinical experience, area of surgical specialization, and previous robotic experience were not exclusion criteria. Exclusion criteria included APPs who could not commit sufficient time to computer modules, dry box training, or a minimum of five proctored surgical cases.

Competency was measured as a composite outcome. Trainees were considered competent if they passed a didactic test on robotic surgery, completed no more than five proctored clinical checklists, and passed a practical examination as a first assistant. The didactic test was designed to test the APP’s baseline knowledge of robotic surgery. Testing content included identification of robotic surgical instruments, sequences for safe instrument exchanges, troubleshooting bedside patient and instrument collisions, and basic anatomy for surgical procedures performed at MD Anderson Cancer (see Appendix 1). The median test score was determined, and at the completion of the training pilot, the APP trainee retook the didactic test and needed to score above the 75th percentile. The selection of the 75th percentile represented a reasonable goal to demonstrate acquisition of didactic knowledge because it represented the upper quartile of baseline knowledge.

Clinical competency was tested by having the trainee complete clinical checklists highlighting skills specific to bedside assisting in robotic surgery (see Appendix 2). The trainee completed these checklists under the supervision of experienced robotic surgical APPs (AL, HG, CM). These three APP proctors each had a minimum 12-month experience with surgeons whose practices were more than 50% robotic. Due to the significant time commitment needed to proctor multiple APP trainees, only 5 clinical checklists were provided per trainee. Clinical checklists were provided intraoperatively, and trainees needed to stay for the duration of the surgical case. APP proctors remained scrubbed in at the same time to assure patient safety. At the completion of the case, the attending physician of record graded the confidence they placed in the trainee based on a 10-point Likert scale. A score of 1 – 4 indicated the operating surgeon was not confident that the APP trainee could first assist solo. A score of 5 indicated the operating surgeon was ambivalent or unsure if the trainee could assist solo. A score > 6 indicated the surgeon was confident that the trainee could assist independently. A score > 8 indicated that the trainee could assist independently and perform advanced skills such as management of advanced energy devices, stapling devices, mesh placement, and specimen retrieval. Trainees were asked to grade themselves on the same l0-point Likert scale, and they had to self-identify three deficiencies they needed to address after each checklist was completed. Trainees who scored greater than 6/10 on the Likert scale were allowed to proceed to a solo clinical practicum.

APP trainees who performed a solo clinical practicum performed first assistant duties for an entire case. The APP proctors were not scrubbed in, but they remained present in the room for safety. The proctors were only asked to scrub into the case if the trainee was unable to perform first assistant tasks requested or performed a critical error. Trainees failed if they performed unsafe instrument exchanges resulting in potential tissue damage. Trainees passed the solo clinical practicum if they achieved a score > 6 on the previously described 10-point Likert scale.

Trainees who were not able to advance to the solo practicum after five cases or who initially failed the solo practicum were given an individual development program (IDP). The IDP was developed by the APP proctors and was based on deficiencies identified. The length of time to complete the IDP was agreed upon mutually by the trainee and proctor. When the IDP was completed, the trainee performed another solo practicum and was scored accordingly.

We compared the characteristics of the APP trainees who passed vs. failed on their first attempt. We anticipated that the number of trainees would be < 30 and used nonparametric statistical testing. Demographic variables, didactic test scores, and Likert scale scores were compared using t test and Fisher’s exact test to evaluate continuous and categorical data respectively. An α of 0.05 was considered statistically significant. We anticipated that the number of clinical checklists needed would not generate enough power to demonstrate a statistical difference, however given the resource intensive nature of clinical checklists, we wanted to know the probability of passing the program given the number of checklists needed prior to proceeding the solo practicum. We used Bayesian inference to determine this conditional probability, and a noninformative prior was assumed (i.e., both passing and failing probability are 00.5).

RESULTS

Ten APP trainees were identified and completed the training program. Trainees were predominantly women, age 30 years (range 26 – 51 years) (). The median time from graduation was 5.4 years (range 00.3 – 20 years), and the median length of time with OR experience was 4.5 years (range 0 – 14 years). They specifically supported general, hepatobiliary, gastric, colorectal, melanoma, breast, and endocrine surgeons. The majority of APP trainees (70%) reported having previous experience with robotic surgery. The participants needed a median of 6.6 months (range 3–10 months) to complete the pilot program.

The median didactic test score on an 18-question multiple choice test was 70% (13/18 questions). This median score was used to determine interquartile ranges, and 84% (15/18 questions) represented the 75th percentile. All trainees scored above the 75th percentile on retesting at the conclusion of the program, with a median retest score of 94% (17/18 questions).

APP proctors had extensive experience in robotic general surgery and colorectal surgery. Attendings who participated in the APP training came from diverse areas of expertise (2 general surgery, 1 hepatobiliary, 1 pancreatic/gastric, 1 colorectal). The clinical checklists were conducted over 64 robotic cases. Procedures performed were cholecystectomy, appendectomy, ventral incisional hernia repairs, bilateral inguinal hernia repairs, hepatectomy, gastrectomy, pancreaticoduodenectomy, low anterior resection, partial colectomy, parastomal hernia repair, small bowel resection, distal pancreatectomy with adrenalectomy, and a duodenectomy, with median time of 195 minutes (range 21 – 571 minutes).

Fifty percent (5/10) of APP trainees obtained clinical competency on their first attempt. summarizes the characteristics of APP trainees who passed on their first attempt vs. those who failed. There were no statistically significant differences in age, sex, year postgraduate, years of OR experience, previous robotic experience, or time needed to complete the pilot program. Similarly, no statistically significant differences exist between didactic test scores, median number of clinical checklists needed prior to proceeding to the solo practicum, and median final practicum score.

also demonstrates that the median clinical checklist Likert scores per number of clinical checklists performed followed no discernable pattern and did not have enough power to determine any reliable statistical signal to determine any correlation between trainee scores and attending scores.

Given anticipated limitations with power to detect a statistically significant difference, and the significant time commitment of surgeons, APP trainees, and APP proctors to perform clinical checklists we decided a priori that a conditional probability of passing the solo practicum should be estimated. shows the probability of passing given the number of clinical checklists performed prior to a solo practicum. The probability of passing is > 50% if less than four clinical checklists are needed prior to proceeding to a solo practicum.

Of the five trainees that failed, four completed an IDP and all passed the solo clinical practicum when retested. The trainee who did not complete the IDP cited lack of dedicated time to complete the program. Thus, the total proportion of trainees obtaining competency to assist in robotic procedures was 90% (9/10).

APP trainees self-identified 162 deficiencies throughout the training program. The top four APP trainee self-identified deficiencies were difficulties with spatial orientation (19/162, 12%), proper scope handling (19/162, 12%), efficient docking of robotic ports (18/162, 11%), and proper port insertion (5/162, 8%).

DISCUSSION

Our pilot program demonstrates that clinical training is the factor most strongly associated with demonstrating competency for APP first assisting in robotic surgeries. APP trainees who passed the program on the first attempt needed < 4 clinically proctored experiences to pass a solo practicum. APP trainees who needed individualized development plans prior to retesting passed after specific deficiencies were identified and addressed.

Because our institution has no precedent for training APPs to assist in robotic surgeries, the purpose our of pilot was to determine factors associated with competence rather than producing a program with a high first-time pass rate. Prior to the pilot, APPs interested in assisting robotic surgeries shadowed in the OR without clear endpoint or timeframe to meet predetermined goals. We designed our pilot to formalize robotic training more rigorously within this precedent. While a 50% first-time pass rate may not seem impressive, it represents the high standards expected to pass the training protocol. Analysis of trainees failing after their first attempts revealed valuable information regarding specific deficiencies in basic laparoscopy, rather than didactic knowledge or robotic specific skills, which were quickly addressed. After completing focused individual development programs, the pass rate increased to 90%.

The parameters we measured to determine competency were determined pragmatically. The didactic test administered at the start of the pilot served as a needs assessment and tested what we felt was necessary for baseline knowledge to assist robotically. We felt that APP trainees, at minimum, needed to know the names of robotic surgical instruments, basic anatomy, and what steps to instrument exchanges were considered safe. We used initial scores as a baseline needs assessment to determine the trainees existing didactic knowledge, and on retesting, we felt that scoring above the 75th percentile was reasonable to demonstrate acquisition of didactic knowledge because it represented acquisition of greater than half of their baseline knowledge. On retesting, all APPs scored quite high with the median score on retesting being 94% (17/18 questions). In future iterations of the training program, minimum passing score on retesting may be readjusted to this score.

The clinical checklist requirement required the greatest time commitment from surgeons, APP trainees, and APP proctors. All surgeons who volunteered to assist in APP training were aware that allowing a trainee may increase OR time; however, few reported feeling the burden of having an APP trainee because trained proctors could perform skills trainees were still acquiring; in addition, surgical trainees were also involved in the case. They derived a great benefit from this training program by being liberated from bedside assist duties and spending more time at the console. The APP proctors absorbed most of the burden during the pilot. APP proctors often had to perform clinical checklists outside of their regular clinical duties to accommodate the changing OR schedule. The proctors estimated that they dedicated a significant amount of time outside of their normal job requirements. Between the three APP proctors, 228 hours were dedicated to training, an average of 76 proctoring hours per trainer. Additionally, APP trainees had to find time outside of their prespecified clinical duties to perform intraoperative checklists. Discussion with APP proctors prompted us to set five clinical checklists per trainee as a maximum offering prior to determining that an individual development plan was needed. Our probability analysis demonstrates that the minimum number of clinical checklists associated with a > 50% probability of passing the solo practicum is four checklists. The higher probability of passing likely represents APP trainees who acquire the skills more quickly or trainees who practiced clinical skills independently in between scheduled checklists. Admittedly, the amount of time between checklists was left open to the learners and decay of skill sets may have occurred between clinical testings. This was a necessary concession to prevent disruption of patient care in their prespecified duties and represents a real-life barrier to implementation of this program. However, we have since considered time commitment needed, identified the minimum number of checklists predictive of passing, and fit these factors within a prespecified period to develop an accelerated and remedial pathway for future trainees. We believe this will increase first time pass rates, optimize resource usage, and increase training efficiency in the next iteration of this program (please see ).

An accelerated and remedial pathway were developed to increase the efficiency of training in a shortened time period. trainees with a high probability of passing followed an accelerated pathway, and trainees with a low probability complete an individualized development plan prior to completing the program.

Most trainees who initially failed the training pilot passed retesting after completing an IDP. The IDP varied from trainee to trainee depending on the specific deficiency identified by the proctor. IDPs often consisted of practicing spatial awareness by performing laparoscopic skills borrowed from the Fundamentals of Laparoscopic Skills testing required of surgical residents or scrubbing into minimally invasive cases where they needed to assemble laparoscopes and place ports.[7,8] Review of the trainee self-identified deficiencies supports this finding. Three of the top four deficiencies directly related to basic laparoscopic skills rather than skills related directly to robotic surgery.

To our knowledge this is the first published protocol, academically or by industry, specifically for training APPs to assist in robotic surgery. There are limitations to this study, with the primary ones being selection bias and limited sample size. While 70% of the APP trainees self-reported prior robotic experience, we could not quantify the quality of prior experience and it did not seem to confound the results ultimately. There were no statistically significant differences between first-time pass vs. fail trainees. The quality of prior robotic experience may have been highly variable, and rigorous expectations of trainers may have been strong enough to negate weaker prior trainee experience. APP robotic training programs do exist in other institutions, but this may be the first protocol to utilize Bayesian estimation in anticipation of a smaller sample size and potential challenges in detecting statistical significance. All eligible OR APPs participated in training, and the small sample size represents the maximum number of participants eligible for training at the time of our study. The Bayesian approach to analysis allowed us to determine a conditional probability of obtaining competence given the number of proctored clinical checklists and has the advantage of being continuously updateable in future iterations of the project with future trainees or the addition of other surgical specialties that utilize the robotic platform.

Our pilot program is being used to guide the future training of APPs for robotic surgery. Training APPs to assist in robotic surgeries requires a significant time commitment and resources. The trainee who did not complete the pilot cited overwhelming clinical burden. Future iterations of the program will be used train new APPs hired to our department during an onboarding period where they have fewer clinical responsibilities and more time to dedicate to robotic training. Furthermore, by determining who is likely to fail the program early and providing a focused IDP in timely fashion, we reduce resources needed in order to obtain higher first-time pass results. Cost was not accounted for in our study; however, all participants entered voluntarily into the program with the hope of developing a larger core of trained APPs who could relieve each other in complex robotic cases, thereby compensating for upfront time commitment. Cost savings associated with a core group of APPs is a variable that may be estimated in the future.

Exploring alternative training options with industry-sponsored programs as a form of cost savings is tempting but has limitations in the context and culture of our institution. The first limitation is the paucity of options specifically oriented to APPs. Most industry courses are focused on recruiting surgeons to use the robotic platform, not APP bedside assisting. The second limitation is quality control and direct application of training. While industry may be able to provide a dry box or cadaveric experience, direct clinical experience with surgeons who actually perform the procedures and their associated expectations simply cannot be replaced. APP participants often stated that accessibility of training with local resources made them more likely to participate.

Our pilot serves as a foundational starting point to determine parameters needed to determine competency but will need many iterations to determine the precise factors associated with achieving competency. While the internal validity of this program is still being tested, graduates of the first iteration report improved confidence with robotics and attending surgeons note improved quality in bedside assistance. Additionally, a graduate of the second iteration was able to pass the program quickly using the accelerated pathway in under three months. This shows promise that dynamic changes based on probability can be effective and efficient despite small numbers. We are hopeful external validity will follow as elements of the program are refined, published, and tested independently. Our institution received this pilot well and it will be a formal part of privileging new APPs to assist robotically not only in the department of surgical oncology, but also other surgical services in the division of surgery that utilize the robot. For future iterations, we plan periodic analyses to demonstrate internal validity across disciplines as part of a learning healthcare system, to continually improve the quality of robotically trained APPs and to be mindful of the resources needed to create competent assistants.

CONCLUSION

A protocolized APP training program focusing on clinical experiences and early identification of deficiencies can produce competent beside robotic first assistants. Further iterations of this program will be needed to determine number of clinical exposures associated with the highest probability of passing our training program.

Table 1.

Demographic Characteristics of All Advanced Practice Provider Trainees

Characteristics	N = 10 Trainees
Age, years (median, range)	30 (26 – 51)
Sex, female (N, %)	9/10 (90%)
Years post graduate (median, range)	5.4 (0.3 – 20)
Years operating room experience (median, range)	4.5 (0 – 14)
Previous robotic experience, yes (N, %)	7/10 (70%)
Time to complete program, months (median, range)	6.6 (3 – 10)
Time to complete clinical checklist portion, months (median, range)	5 (1 – 8)

Table 2.

Univariate Analysis Comparing Advanced Practice Provider Who Passed the Training Program on the First Attempt vs. Those That Did Not

	Pass	Fail
	N = 5	N = 5	p-Value
Age, year (median, range)	29 (29–36)	44 (26–51)	0.15
Sex, female (n, %)	5 (100%)	4 (80%)	1.0
Years postgraduation (median, range)	4 (1–12)	10 (0.3–20)	0.35
Years in operating room (median, range)	4 (1–12)	8 (0–14)	0.51
Previous robotic experience, yes (N, %)	4 (80%)	3 (60%)	1.0
Time to complete program, months (median, range)	6.7 (2.6–9.1)	6.5 (3.8–9.8)	0.75
Pretest didactic score (median, IQR)	78 (75–84)	63 (63–69)	0.31
Post-test didactic score (median, IQR)	88 (88–100)	94 (87–100)	1.0
Number of clinical checklists needed (median, range)	4 (1–4)	4 (2–7)	0.4
Attending Score vs. APP Score (median, range)			NA
Checklist 1	6 (4–10) vs. 7 (3–8)	5 (3–6) vs. 6 (3–6)
Checklist 2	6 (5–7) vs. 7 (6–7)	6 (4–8) vs. 6 (4–8)
Checklist 3	5 (5–7) vs. 5 (5–7)	6 (5–7) vs. 6 (5–7)
Checklist 4	7 (6–7) vs 7 (6–8)	7 (3–7) vs. 6 (2–8)
Checklist 5	NA	4 (4–4) vs. 6 (6–6)
Practical Score (median, range)	7 (6–8)	6 (3–8)	0.27

APP, advanced practice provider; IQR, interquartile range.

Table 3.

The Probability of Passing the Training Program Given the Number Clinical Checklists

Number Clinical Checklists	Pass	Fail	Probability of Passing
	N = 5	N = 5
1	1	0	100%
2	1	0	100%
3	0	1	67%
4	3	2	63%
5	0	2	50%