Digitalizing Surgery with VR & AR | Surgical Twins in the Operating Room

Surgical Digital Twin standing in a virtual operating room in front of a OR table.

Digitalizing the Operating Room

From Simulation to Reality – Digital Twins as Tools in Surgical Education and Training

Digital technologies like Virtual Reality (VR) and Augmented Reality (AR) have already been applied in different fields of medical education. They offer a much more interactive and immersive learning experience, but when it comes to the context of the surgical procedure, in existing applications the environments and realistic appearances are often limited. Because of this, authentic learning opportunities are sometimes lacking.

In response to these difficulties, the research project TRANSFORM was conducted from January 2023 to December 2024 at the OR-X alongside other projects with the aim of pushing further improvements in this field. With the support and funding from the University of Zurich, TRANSFORM pursued multiple goals over the course of two years, all focusing on establishing the OR-X as a center for modern research and training, and testing technologies for the digitalization of surgeries.

Why surgical digitalization is needed?

A surgeon wearing an AR headset in the OR, holding a spine model in his hand.

Why surgical digitalization is needed?

The idea behind the digitalization of surgeries is to create a virtual replica of a real-world operating room, which combined with VR and AR can then be used to create advanced learning opportunities. Surgical digitalization is an important step for the future of education in the medical field. It will enable medical students and surgeons to train and practice different surgical techniques in a realistic, interactive, and virtual environment without the need for real anatomical models.

Currently, wet lab training on real specimens is the closest form of hands-on surgical practice. However, this method also poses some challenges, like ethical questions, limited availability, and costs. Using virtual environments instead offers a practical alternative for learners to acquire essential surgical skills without compromising a patient's safety or relying on human specimens.

The Surgical Digital Twin

In the TRANSFORM project, the digitalization of surgeries was the primary focus, centered around the development of the so-called surgical digital twin (SDT).

The concept of a SDT is a complex and specific application of digitalization. The goal is for the SDT to be a perfect virtual representation of the physical world and to create a high-fidelity representation of the OR. For the creation of an SDT the integration of three core elements is needed: the physical object or process and its environment, a digital replication of that environment, and the data and communication links that connect the two. For the alignment of physical and digital entities, there needs to be a common spatio-temporal representation, where data that come from different modalities get fused together.

At the core of the SDT lies a detailed 3D representation of the OR, which serves as the spatial framework for integrating all additional elements. This digital model is then embedded into a VR-system. The OR setting does not only include static elements like surgical lamps and other infrastructural objects, but also includes dynamic entities like medical staff, surgical instruments, and anatomy.

The described VR-System of this project is the first that enables the user to experience a virtual dynamic 3D environment that can be seen as a replica of a real surgery, by encompassing all the different structures. Unlike previous approaches, this project aimed to capture a full segment of a spine surgery instead of only individual anatomical structures or specific surgical interventions.

Capturing this level of complexity requires the integration of various advanced technical devices and sensor systems, which collect data that is then fused with prior domain knowledge to generate an accurate digital model. In this case, five RGB-D cameras brought the surgeon to life in 3D, while a cutting-edge high-end camera captured the anatomical details. Surgical instruments were precisely tracked in real time with an infrared stereo camera, and a laser scanner mapped out the entire operating room, this way seamlessly merging all data into a unified SDT, as shown in the demo video.

The potential and challenges of a surgical digital twin

THE POTENTIAL AND CHALLENGES OF A SURGICAL DIGITAL TWIN

The SDT, which was the central component of this project, has the potential to significantly narrow the gap between virtual simulations and real-life surgical practices. It provides not only a new way for surgeons to train, but also a controlled setting where future applications or technologies can be tested before clinical deployment. Additionally, the digitalization of surgeries would not just help with training, but it would also enable a more standardized and structured data collection process.

In its current state, the project could already be used for educational purposes, yet only for short surgical steps or simple interventions. For a full surgery, which is the long-term goal, additional research and testing needs to be done. Having multiple cameras and laser scanners distributed across 21 locations within the room, certain parts in the OR-X were challenging to reconstruct. For instance, the operating table where the legs had to be modeled and added manually, due to the high reflectivity of certain surfaces.

While certain components still require manual modeling, future advancements could enable a seamless integration with VR and AR technologies. Ultimately, the goal is also to encourage the further development of additional methods for the digitalization of surgeries and the creation of SDTs. The OR-X is a pioneer in developing SDTs and virtual surgical modelling. These innovations have already been presented at leading conferences and featured as an immersive virtual reality demo, earning significant recognition. Further research on this technology is currently under review for publication.