Sunday ACC 2019: Mixed Reality Offers Advantages of Virtual and Augmented Realities

Virtual reality (VR) and its less immersive kin, augmented reality (AR), are gaining traction in some medical applications. But a new entrant, “mixed reality,” may have the inside track in the interventional suite or operating room.

Mixed reality allows the user to interact with both the real world and digital data, according to Jennifer N. A. Silva, M.D., a pediatric cardiologist atWashington University School of Medicinein Saint Louis, Mo., who spoke during Future Hub at this year’s annual meeting (#ACC19) of theAmerican College of Cardiology.

With ready access to both digital and physical reality, a surgeon or interventionalist can simulate a procedure on a digitally created model that is updated in real time – then perform the actual procedure on the patient.

“Mixed reality affords the ability to remain in your natural environment,”SilvatoldImaging Technology News (ITN)andDiagnostic and Interventional Cardiology (DAIC). Yet the users of mixed reality can take advantage of a digital world otherwise found only in VR.

Mixed reality is somewhere in the middle of what Silva described as a spectrum of “extended realities.” At one end is virtual reality (VR), which completely immerses the user in a digital world. Because VR is a completely separate and alternative reality, it “does not allow you to have a meaningful interaction with your natural environment,” she said.

At the other end of the spectrum is reality, characterized by what we sense with our eyes. In between these two ends are augmented reality (to which mixed reality is closely related). In AR, the user cannot interact with but rather only look at digital data imported into the natural environment, Silva said.

“Mixed reality allows you to have a meaningful interaction with digital dataandwith your natural environment,” she said. “So instead of just looking at a hologram, you can do something to it -- you can touch it; rotate it; turn it upside down.”

Fitting Round Pegs Into Round Holes

The technology – and the type of reality it harnesses – “has to fit what you are trying to do,” she told ITN and DAIC. “There has to be a match between the technology and the need.”

Different types of reality may be useful in one or more of the four major application areas: education, pre-procedural planning, rehabilitation and intraprocedural visualization. VR offers little value during procedures, because it immerses the user in a completely digital world and, therefore, blocks out the natural environment. For this same reason, however, VR may be useful in rehabilitation, as it may replace the user’s perception of the real world with a digital one.

VR may also be useful when practicing a surgical or interventional procedure. Augmented reality has proven useful in actual interventions, presenting data that a surgeon or interventionalist can use during the actual procedure. But mixed reality offers the ability to interact with digital data and with the real world in the same context and time frame.

“如果我在做一个手术，我不能完全离开那个环境(在VR中互动)。I need something that will allow me to interact with my natural environment and with the digital data,” said Silva, who was one of the moderators during March 16 ACC Future Hubsession题为“临床虚拟现实和增强现实准备好黄金时间了吗?”

她告诉ITN和DAIC:“操纵和控制数据的能力可能是扩展现实的最大增值。”理论上，混合现实允许这种操纵和控制。

“We are on the first step of what I think will be a long road,” said Silva, who is one of the founders of a mixed reality company, calledSentiAR.

Silva and her colleagues right now are struggling to achieve connectivity among the different pieces of equipment found in an EP interventional lab. Examples of this equipment are fluoroscopy systems, vital signs monitors and cardiac mapping systems. “A lot of them are made by different vendors, so it lends itself to poor interoperability,” she said. The goal is to integrate the data from the equipment, as well as their workflows.

“I want to make the platform the glue that starts to allow me to interact with all my pieces of equipment,” she said.

How Mixed Reality Might Impact EP

Mixed reality may prove particularly useful during electrophysiological procedures, when interventionalists map electrical activity inside the patient’s heart, then ablate tissues associated with electrical signals that cause arrhythmia. The device being developed by SentiAR creates a 3-D model of the patient’s heart that shows the map of electrical signals in real-time, said Silva who specializes in pediatric cardiology. The mixed reality platform being developed at SentiAR can be used on patients of any age, she noted.

通过一种名为HoloLens的头戴式技术，用户可以看到3d图像和其他数字数据，也可以看到现实世界。HoloLens是微软公司生产的。Silva将其描述为“你头上的电脑”，用户可以看到叠加在自然环境视图上的数字结构。同时，内置在头戴式技术中的传感器记录头部运动，允许用户点击投射到用户视野中的数字菜单，或控制介入性实验室中的不同设备。

虽然HoloLens已经被证明是有用的，但该公司与微软的技术并没有千丝万缕的联系。相反，SentiAR对可能出现的新技术持开放态度。“市面上有很多混合现实头盔，还有更多正在研发中，”她说。

Silva said the product being developed by SentiAR is a long way from finalization. “As good technology does, it will continue to evolve,” she said, noting that future technologies may harness new types of reality that represent different points on the “extended realities” spectrum.

“This (spectrum) is where innovation is going to happen.”

Greg Freiherr is a contributing editor to Imaging Technology News (ITN). Over the past three decades, Freiherr has served as business and technology editor for publications in medical imaging, as well as consulted for vendors, professional organizations, academia, and financial institutions.

Additional Content:

Two Technologies That Offer a Paradigm Shift in Medicine at HIMSS 2017

VIDEO: Augmented Reality for Surgical Planning

VIDEO: Users Can Touch This Virtual Reality Heart

A 40,000 Foot View of Trends in Cardiology

Applications for Artificial Intelligence in Cardiovascular Imaging

Machine Learning Approaches in Cardiovascular Imaging

Technology Report: Artificial Intelligence (Video reportpublished January 2019)

PODCAST: Shortcomings of CTA in Cardiology

PODCAST: How Technology Is Changing Cardiology

ACC.19 Future Hub Hosts “Shark Tank” of Emerging Technologies In Cardiology