By Christina Frangou
Philadelphia—Some senior surgical trainees in the United States are unable to competently perform common core procedures by the time they graduate, according to results from the first multi-institutional study to measure resident autonomy and operative performance.
The investigators reported that, for an appendectomy performed on a relatively uncomplicated patient, a typical chief resident in the week prior to graduation has a 97% chance of being deemed “practice ready.” For more complicated patients undergoing an appendectomy, they had a 92.8% chance of demonstrating competence.
For a partial colectomy on a less complex patient, the same resident has a 91.8% chance of being deemed competent. For a more complex patient, a typical chief resident in the week prior to graduation has an 81.8% chance of being deemed competent, the analysis showed.
“Just to highlight the significance of these numbers, the following week, a resident may be credentialed to perform the procedure independently and on patients that are not always of average complexity,” said the study’s lead author Brian C. George, MD, assistant professor of surgery at the University of Michigan, in Ann Arbor.
He presented the study at the 2017 annual meeting of the American Surgical Association.
The results showed that U.S. general surgery residents are not universally ready to independently perform core general surgery procedures by the time they complete residency training. This is especially true for the less frequently performed core procedures—splenectomy, fasciotomy and gastrectomy—where residents were deemed competent in 74% of procedures analyzed. That is down from the 84% competency reported for chief residents performing the five most frequent core procedures.
This study is the first to show in a systematic fashion that a high percentage of residents are not rated as practice ready for core general surgery procedures by their final year of training. The ratings of residents are based on data submitted by attendings with a smartphone app.
Surgeons who heard the study presented, many of them surgical educators, said they put little stock in the specific numbers reported. Several said the real strength of the study lies in the app used to collect data. It could change the way residents are assessed and evaluated, they said, and offer clues about what works and what doesn’t in surgical education.
“The tool is a means by which we can assess the impact of training interventions. In isolation, though, an app alone is not going to be enough to get us to resident autonomy. It’s a great measurement tool, but it needs to be leveraged with faculty and resident development interventions, which support the development of autonomy in the context of supervision,” said Rebecca Minter, MD, professor and the Alvin Baldwin Jr. Distinguished Chair in Surgery at UT Southwestern Medical Center, in Dallas.
The investigators studied data collected by an app, SIMPL, that measures trainee performance and autonomy.
After a 2012 survey of program directors reported that 42% believed that incoming fellows were not ready to perform 30 minutes of a major procedure independently (Ann Surg 2013;258:440-449), the American Board of Surgery funded this study to measure operative autonomy and readiness for independent practice by categorical general surgery residents. The app’s development was funded by the members of a multi-institutional quality improvement collaborative.
Attendings at 14 university-based training programs with at least 20 categorical residents were asked to evaluate trainee performance via the SIMPL app.
Residents or attendings created an evaluation on SIMPL for each case to be assessed and sent the notification to the counterpart’s cellphone. Raters then completed an assessment of the resident’s performance. As a result, the investigators had data from a subset of cases performed by residents rather than all cases performed by residents.
In total, 1,753 attendings and residents were eligible for inclusion in the study; 88% consented and registered with the study’s evaluation system.
SIMPL’s performance scale has five levels, beginning with when a trainee is deemed unprepared or demonstrating a critical deficiency. The two highest levels—practice-ready and exceptional performance—were rated as competent (J Surg Educ 2016;73:e118-e130).
Of the chief residents assessed, 69% were found to be competent in 2,498 performance ratings for all procedures. For core procedures, the observed proportion of performances deemed competent increased to 77% for fifth-year residents. In comparison, 13% of first-year residents were shown to be competent.
For the five most common core procedures done by trainees in the final six months of training, 96% achieved competency for appendectomy, 86% for cholecystectomy, 85% for ventral hernia repair, 82% for inguinal/femoral hernia repair and 71% for partial colectomy.
Autonomy was assessed using the Zwisch scale, named after Jay Zwischenberger, MD, professor and chair of surgery at the University of Kentucky, in Lexington, who has been using a version of it in his practice for the past 20 years. The four-step Zwisch scale is designed to assess how much guidance is provided to residents in the operating room: show-and-tell, active help, passive help, supervision only. The last two levels indicate residents who have achieved meaningful autonomy (J Surg Educ 2014;71:e90-e96).
With the Zwisch scale, 66% of PGY-5 residents were meaningfully autonomous. For residents in the final six months of training, based on 599 ratings by 136 raters of 71 residents, only 33% achieved “supervision only.”
Steven C. Stain, MD, professor and the Henry and Sally Schaffer Chair of Surgery at Albany Medical College, in New York, argued that residents today are not less well prepared than residents in the past, but they are more closely scrutinized.
Since the early 1990s, resident autonomy has been curtailed by outside factors such as public demands for more oversight of trainees, a change in Medicare billing that now requires an attending surgeon to be present for key parts of the operation, and public reporting of surgeon outcomes, Dr. Stain noted.
He recounted a description once given to him by a chair of surgery. “Today’s residents are better; they’re smarter; they do more cases, but they may not be as confident because they’ve not had the opportunities I had 20 years ago to be autonomous.”
Moreover, autonomy may not reflect a resident’s surgical skill, surgeons noted.
“We know that residencies vary widely in their culture regarding allowing for resident independence in the OR. This variability makes it hard to know how to interpret this data,” said Mary Klingensmith, MD, the Mary Culver Distinguished Professor of Surgery and vice chair for education at Washington University School of Medicine in St. Louis.
“The chief message to me was that the system of applying the Zwisch scale to evaluate procedures via an app is feasible.”
The limitations of the study include selection bias, as all trainees came from university-based programs, and sampling bias, as not all cases were evaluated. Although investigators analyzed thousands of cases performed by residents, the study is based on a miscellaneous collection of cases rather than an in-depth review of all cases performed by the residents in the study.
Carla M. Pugh, MD, PhD, the Susan Behrens Professor of Surgical Education at the University of Wisconsin, in Madison, specializes in the use of simulation technology for surgical education assessment. She said the app could be used earlier in a resident’s training to identify trainees at risk of being outliers by their final year. It could identify second- and third-year residents who were falling behind their peers, and who would benefit from interventions.
But a greater challenge is getting faculty to use the app, she said. “It’s not used as broadly as we would like it to be used, but that’s not uncommon with new technology.”
Selwyn O. Rogers Jr., MD, director of the University of Chicago Medicine’s trauma center and professor and chief of trauma and acute care surgery, cautioned against overreliance on an app to evaluate residents.
“The best way to address subjective issues is through proximity. My best judgment as to whether somebody should practice is direct observation of how they think, how they operate and what they do.”
The full study is expected to be published in an upcoming edition of the Annals of Surgery. It was presented this spring at the 2017 annual meeting of the American Surgical Association, the oldest surgical association in the United States.
Orgininal Article: General Surgery News
