In A First, A Robot Listened To Spoken Instructions And Performed Surgery – Just Like A Human Would

Robotic surgery has reached a new milestone after a robot successfully – and autonomously – performed gallbladder removal operations while listening and responding to voice commands. Just as a human trainee would be directed by a senior surgeon, the robot was able to act on verbal instructions after having been initially trained on videos of the procedure.

It was even able to respond to unexpected emergency scenarios, a vital skill that human surgeons must learn.

“This work represents a major leap from prior efforts because it tackles some of the fundamental barriers to deploying autonomous surgical robots in the real world,” said lead study author Ji Woong “Brian” Kim in a statement. 

Teaching a robot to execute a complex, multistep surgical procedure is a daunting prospect. You have to know every individual action the robot will need to take to perform the operation successfully, including things like where not to put the scalpel (pro tip: steer clear of big blood vessels), but then you also have to communicate all of that information to the robot.

“That’s what makes this so so humongously difficult, because how do you write this down? How do you write that in code?” Dr Mathias Unberath, John C. Malone Associate Professor at Johns Hopkins University, recently told IFLScience when we explored the topic in CURIOUS magazine.

“Can I watch a human learn what I’m supposed to be doing? If the answer is yes, things become, not trivial, but they’re considerably easier.”

That's called imitation learning, and it's what this latest study relied on. Supervised by Unberath’s colleague at Johns Hopkins, Professor Axel Krieger, Kim and a team of collaborators from across the United States began by training the system on videos of human surgeons performing gallbladder removal (cholecystectomy) on pig cadavers.

Last year, Krieger’s team revealed at a conference and in a preprint that they’d been able to train a robot to manipulate a surgical needle, lift sections of tissue, and suture, all with just a few seconds of learning. Gallbladder removal is a significant step up, requiring 17 individual stages of identifying tissues, making precise incisions, and placing clips. All of these come with individual challenges.

But unlike previous iterations of autonomous surgical robots, which Krieger likened to training a robot to drive a carefully charted route, Krieger explained that the latest system – Surgical Robot Transformer-Hierarchy (SRT-H) – “is like teaching a robot to navigate any road, in any condition, responding intelligently to whatever it encounters.”

Gallbladder removal, therefore, presented little issue. After watching the video tutorials, which included explanatory captions, the robot could perform the procedure with 100 percent accuracy. It may have taken longer than it would for a human surgeon, but the results were comparable. The 17 steps were executed successfully on eight different pig gallbladders ex vivo (that’s outside the body).

Because it has natural language capabilities – powered by the same kind of machine learning that underpins ChatGPT – the system can respond to verbal instructions given by a human surgeon in real time, such as “move the left arm a bit to the left”, learning from this feedback as it goes.

Even when the study team deliberately threw in a curveball, like moving the position of the robot or adding dyes to change the colors of some of the tissues, the robot wasn’t fazed.

“To me it really shows that it’s possible to perform complex surgical procedures autonomously. This is a proof of concept that it’s possible and this imitation learning framework can automate such complex procedure with such a high degree of robustness,” said Krieger.

AI-powered robots autonomously performing surgery? No longer the preserve of science fiction.

“Our work shows that AI models can be made reliable enough for surgical autonomy – something that once felt far-off but is now demonstrably viable,” said Kim.

The study is published in Science Robotics.