How to access camera preview frames with MediaCapture class in Unity?

Valentin 40

Using the Windows MediaCapture class we can capture videos and images on the HoloLens 2. The class also offers preview capabilities (see here). However, I am not sure how to get the preview stream within Unity and then display it to the user. In the docs they suggest to connect a XML CaptureElement to the mediaCapture object. What's the equivalent for use in Unity/on Hololens 2? A similar question without an answer has been asked 7 years ago here

Roy Li - MSFT 33,496 Reputation points Microsoft Vendor

2024-12-02T01:21:44.2133333+00:00

Any updates about this issue?
Valentin 40 Reputation points

2024-12-02T07:56:53.09+00:00
Not yet as I am unsure which way to go. Is there a definite working solution for this issue? Possible but not confirmed approaches include:

Get a preview frame manually in a certain interval, convert it to a format displayable in Unity, and then displaying it

Using a MediaFrameReader. Not sure if this works in addition to using the MediaCapture API for capturint video and image or one would need to replace that workflow with the MediaFrameReader
Ying Li - MSFT 1,300 Reputation points Microsoft Vendor

2024-12-02T09:05:16.5133333+00:00

Hi Valentin.

You can use the ways of MediaCapture to capture photos and videos, and MediaFrameReader to imeplement the preview function. Using MediaFrameReader, you need to initialize the MediaCapture object - Process media frames with MediaFrameReader - UWP applications | Microsoft Learn.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Valentin 40 Reputation points

2024-12-09T13:29:03.69+00:00

Thank you for your support! I am currently looking into this. For the MediaFrameReader initialization, do I need to setup a MediaFrameSourceGroup as well or is a VideoProfile automatically associated with a source? I manually select a video profile before initializing the MediaCapture object. Do I need to manually select a source group as well or not then?

I am also struggling to retrieve preview frames. Basically, I am already successfully using the MediaCapture API to capture videos or images to local disc on HoloLens 2. Now, it would be nice to be able to preview the stream before saving.

1 answer

Ying Li - MSFT 1,300 Reputation points Microsoft Vendor

2024-11-18T09:37:58.8433333+00:00

Hello, Welcome to Microsoft Q&A,

You can use the WinRT API to capture frame data, convert it to a Unity-compatible format, and then apply it to a Texture2D.

Alternatively, you can also implement camera preview function using PhotoCaptureFrame or WebCamTexture provided by Unity, without using the WinRT API.

If you use the WinRT API with Unity for HoloLens, ensure that the preprocessor directive ENABLE_WINMD_SUPPORT is wrapped around any WinRT-leveraged code. Note that this directive won't work in Unity Play mode. For more details, see - WinRT APIs with Unity for HoloLens - Mixed Reality | Microsoft Learn.

If the response is helpful, please click "Accept Answer" and upvote it.

Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.
Please sign in to rate this answer.
Valentin 40 Reputation points

2024-11-18T14:17:02.6933333+00:00

Thanks for your replay YingLi! I am using the WinRT, MediaCapture API (which is working perfectly fine). Now, however, I am looking for a continuous preview of the camera stream similar to here, so the user knows what exactly will be captured.

Are you suggesting that I get a single frame 30 times/s in this way? Is there now way to access the preview stream as provided by the MediaCapture API?

Ying Li - MSFT 1,300 Reputation points Microsoft Vendor

2024-11-19T07:28:53.3133333+00:00

Hi, Valentin.

A media sink (CaptureElement) must be set before using MediaCapture.StartPreviewAsync. As described on the page you referred to: “The code in this article assumes that your app already has an instance of MediaCapture that has been properly initialized, and that you have a CaptureElement with an active video preview stream.” So, I doubt that the way and these methods on the page can be directly use with Unity. You might want to look into MediaFrameReader , which could be a better choice.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

How to access camera preview frames with MediaCapture class in Unity?

1 answer

Your answer