HoloLens: Inception - The real world within mixed reality
Disclaimer: I'm not on the HoloLens team
This week the HoloLens team announced SpectatorView which addresses three key scenarios, including allowing an audience to view what the HoloLens wearer sees on a 2D screen. This offers a professional grade solution to video capture the holographic content and video stream it in real-time to an external screen. Essentially the use case for this scenario is:
“How can I create a way for people to see what someone sees when they are wearing a HoloLens from a third-person perspective? I want people to not only see what I see when wearing a HoloLens, I want people to feel like they are part of the experience!”
To solve this, there is now an open source library available and some additional hardware necessary: https://developer.microsoft.com/en-us/windows/holographic/spectator_view.
What I'm sharing here is a different use case, that allows you (the HoloLens wearer) to see the real world within mixed reality in real-time ie. a "RealWorldView". So if the question is one of the below, keep reading:
“How can I see myself from a third person view within mixed reality. I want to see how I interact with holograms from the outside-in.” eg. Training scenarios
“How can I monitor another room in the real world whilst I'm in a different mixed reality one.” eg. CCTV scenarios
“How can I control a physical camera within mixed reality" eg. AV scenarios
Why not just stand in front of a mirror?
To address part of the first question, you could just stand in front of a mirror (whilst wearing the HoloLens) if you just need to see your reflection interacting with Holograms. But what if you need to see behind you, or from a different angle? Or something completely different, perhaps in another room, that isn't in your field of vision? This solution will show you what is in the camera lens and display it in 2D within mixed reality. Further more, you can take control of the cameras functions - control zoom, focal point and take photos onto the camera.
Keep talking, tell me what I need?
So, similar to SpectatorView there is an additional camera and some software required so this solution won't work for everyone. However, the end result is a fully untethered rig with no additional computer:
| Hardware/Software component | Description |
|---|---|
| Sony digital camera compatible with Smart Remote Control | Tested with Sony RX100M3, other Sony cameras with a compatible API should also work |
| Sony RX Remote - Windows Store | Windows Store UWP application, recently updated for the HoloLens |
So then, what does this achieve?
This setup will allow you to see whatever the camera can see in a 2D window within mixed reality. If the camera is pointing towards you (as the HoloLens user) it's akin to standing in front of a mirror, you'll see yourself but without the holographic content. But you'll be able to continue interacting with any other holograms eg:
How is this useful?
Lets consider you have created a virtual or augmented world that captivates the user, but the user needs to be aware of how they appear in the real world. For example, imagine a hologram of a golf or baseball - I want to swing at the ball to interact with it, but at the same time I want to see what my real swing actually looks like. Filming yourself practicing sports is one of the best ways to improve and correct mistakes or bad habits. Another example is monitoring or CCTV, where you have a requirement to watch a physical entity in the real world whilst you are immersed in a digital one. This is an untethered solution, the camera and the HoloLens are not physically connected and there are no additional computers or devices required - meaning this can be setup easily in most environments.
What's going on here? How does this actually work?
The Sony RX100 is one of a range of Sony cameras that support Sony's Smart Remote Control. The camera acts as a wifi hotspot and exposes an API that can be used to read data from the camera or send commands to it. A while ago I developed Sony RX Remote originally for Windows Phone 8.1, this has since been updated to Universal Windows Platform (UWP). Therefore, it runs on many different types of Windows 10 device - including the HoloLens.
But I don't have a compatible Sony camera, can you just show me what happens?
Here's an example of a what the experience is like - recorded using Mixed Reality Capture (MRC).
[video width="854" height="480" mp4="https://msdnshared.blob.core.windows.net/media/2017/02/Inception-spaceman.mp4"][/video]