Microsoft HoloLens in academia unlocking new surgical procedures

Last week we had a great event at Microsoft London with a number of UK Academic institutions. These Universities have been working on Windows Mixed Reality and HoloLens.

During this session, a select number of invited UK universities were invited  to share the work that they have been doing with HoloLens across different curriculums, with a specific focus on industry projects.   

A number of the leading Mixed Reality Program Partners also attended to share the work they have been doing with academia across an even broader breadth of disciplines.    

• A brief introduction and overview by Microsoft around the goals and objectives for mixed reality within academia  
• Each university & partner to provide update on the work that they are doing with HoloLens & Industry, and what they look to be doing in the future
• Moderated round-table
• Over networking and drinks, we will had HoloLens demo’s from Microsoft Mixed Reality Partners.  

One of my favourites showcases was the work from Imperial College London. Which they have been HoloLens and Surgery,

had

The team at  Imperial College is aiming to prove how HoloLens can augment the skills of surgent.

The team have been working with Imperial at a London  hospital, where  HoloLens is being used to help surgeons reconstruct damaged tissue, overlaying a virtual version of a patient’s anatomy before they start slicing. The technique has already amassed eight successes on real patients.  

"You talk about a picture being worth a thousand words, I can’t tell you how many words mixed reality is worth."Dimitri Amiras, a musculoskeletal consultant radiologist

During many reconstructive surgeries a large chunk of flesh or a flap is used, a flap is typically used to fix a wound. The flap is a transplant tissue from another part of the body. To fix the flap in place, is NOT as simple as pressing it into the hole, but instead the flap needs to be carefully plumbed into the blood vessels around the wounded area.

“The key in the operation is blood supply, You can't just transfer a piece of tissue to the wound. It will die. It needs to be oxygenated. So these vessels supplying oxygen to the flap need to be plumbed into the existing blood supply. “But you can't see those vessel superficially, looking at the [skin’s] surface. This is the key: can we use mixed reality to accurately locate those vessels and save time during surgery?” Dr Philip Pratt, research fellow at Imperial College’s department of surgery and cancer.

Currently, surgeons use Doppler ultrasound readings, CT scans and pen markings to locate blood vessels. This isn’t always accurate, and means that time can be wasted in hunting for vessels that aren’t adequate for reconnection, or may have been damaged by whatever caused the wound. For a procedure that can take 10-12 hours to complete, those wasted moments are crucial in directing the success of the surgery.

(CTA imaging showing the location of perforating artieries with yellow arrows. Credit: Imperial College)

Using the  HoloLens.

Instead of translating scan data into two-dimension images for surgeons to consult, the mixed-reality headset allows this imaging to be projected as a 3D hologram onto the body. In a paper published in January, in European Radiology Experimental , the researchers behind the project describe it as “allowing the surgeon to ‘see through’ the patient’s skin”.

The work has resulted in eight successful clinical cases, all focused around operations on limbs. “We've proven feasibility,” says Pratt. “Now it's time to scale it up.”

The Enhancement of AI

“You'll have all the anatomy segmented out, with a tumour identified so the surgeon knows where to cut. That's the dream.”Dimitri Amiras, a musculoskeletal consultant radiologist

While projecting holograms from patient scans has been proven and tested. The process is currently “too long”. A human expert still needs to segment the scan data and identify potential vessels before a 3D model can be made. With the use to artificial intelligence, the researchers hope to be able to automate this part of the procedure.

“The idea is that you could take your CT scan straight from the scanner, stick it through an algorithm, and out pops out a bunch of voxel models,” Dr Philip Pratt, research fellow at Imperial College’s department of surgery and cancer.

Placement of the Hologram and future AI considerations

(An virtual overlay on a patient's leg, using a HoloLens. Credit: Imperial College)

The holographic overlay  currently needs to be manually controlled by the surgeon, moved around and rotated using hand gestures. The hope is that future iterations will automatically snap the 3D model to its real-world counterpart, letting the surgeon focus on where they need to cut instead of wafting a 3D model through the air.

“You can essentially simulate the effect of gravity, So you start in a particular position, then you turn [virtual] the leg upside down and the simulation will change the position of the tissue. Dr Philip Pratt, research fellow at Imperial College’s department of surgery and cancer.

The proof

With eight cases under their belt, and potential to start building HoloLens into medical school training, the researchers at Imperial are adamant that mixed reality has a crucial role to play in the future of surgery. You can read more at paper published in January, in European Radiology Experimental , the researchers behind the project describe it as “allowing the surgeon to ‘see through’ the patient’s skin”.