Digitalizing Surgery with VR & AR | Surgical Twins in the Operating Room

Surgical Digital Twin standing in a virtual operating room in front of a OR table.

Digitalizing the Operating Room

Digital Twins as Tools in Surgical Education and Training

Digital technologies like Virtual Reality (VR) and Augmented Reality (AR) have already been applied in different fields of medical education. They offer a much more interactive and immersive learning experience, but when it comes to the context of the surgical procedure, in existing applications the environments and realistic appearances are often limited. Because of this, authentic learning opportunities are sometimes lacking.

In response to these difficulties, the research project TRANSFORM was conducted from January 2023 to December 2024 at the OR-X alongside other projects with the aim of pushing further improvements in this field. With the support and funding from the University of Zurich, TRANSFORM pursued multiple goals over the course of two years, all focusing on establishing the OR-X as a center for modern research and training, and testing technologies for the digitalization of surgeries.

Why surgical digitalization is needed?

A surgeon wearing an AR headset in the OR, holding a spine model in his hand.

Why surgical digitalization is needed?

The idea behind the digitalization of surgeries is to create a virtual replica of a real-world operating room, which combined with VR and AR can then be used to create advanced learning opportunities. Surgical digitalization is an important step for the future of education in the medical field. It will enable medical students and surgeons to train and practice different surgical techniques in a realistic, interactive, and virtual environment without the need for real anatomical models.

Currently, wet lab training on real specimens is the closest form of hands-on surgical practice. However, this method also poses some challenges, like ethical questions, limited availability, and costs. Using virtual environments instead offers a practical alternative for learners to acquire essential surgical skills without compromising a patient's safety or relying on human specimens.

The Surgical Digital Twin

In the TRANSFORM project, the digitalization of surgeries was the primary focus, centered around the development of the so-called Surgical Digital Twin (SDT).

The concept of an SDT is a complex and specific application of digitalization. The goal is for the SDT to become a highly realistic virtual representation of the physical operating room environment. For its creation, three core elements are required: the physical object or process and its environment, a digital replication of that environment, and the data and communication links connecting the two. To align physical and digital entities, a common spatio-temporal representation is needed in which data from different modalities can be fused together.

Building the Virtual Operating Room

At the core of the SDT lies a detailed 3D representation of the operating room, serving as the spatial framework for integrating all additional elements. Embedded into a VR system, the environment includes not only static infrastructure such as surgical lights and other objects, but also dynamic entities including medical staff, surgical instruments and anatomy.

The VR system developed within this project is the first to enable users to experience a virtual dynamic 3D environment that replicates a real surgical procedure by encompassing all relevant structures. Unlike previous approaches focusing only on isolated anatomical structures or specific surgical interventions, the project aimed to capture an entire segment of a spine surgery within one connected digital environment.

Capturing this level of complexity required the integration of various advanced technical devices and sensor systems, collecting data that was fused with prior domain knowledge to generate an accurate digital model. Five RGB-D cameras captured the surgical team in 3D, while a high-end imaging system recorded detailed anatomical structures. Surgical instruments were tracked in real time using an infrared stereo camera, and a laser scanner mapped the entire operating room environment, seamlessly merging all data into a unified SDT.

Watch demo video

Future Potential of Surgical Digital Twins

As the central component of the TRANSFORM project, the Surgical Digital Twin has the potential to narrow the gap between virtual simulations and real surgical practice. Beyond surgical education and training, the system could also provide a controlled environment for testing future technologies before clinical deployment, while enabling more standardized and structured surgical data collection.

In its current state, the system can already be used for educational purposes and selected surgical workflows. However, achieving the long-term vision of replicating complete surgeries still requires further research and technical refinement, particularly in the reconstruction of complex operating room environments and reflective surfaces.

The OR-X is among the pioneers in the development of Surgical Digital Twins and virtual surgical modeling. The project has already been presented at leading conferences and featured as an immersive virtual reality demonstration, with further scientific publications currently under review.