Hololens 2 Unity app running on desktop and custom sample player don't have the same coordinate system

Question

Hello, I'm currently engaged in a project involving marker tracking using the front camera of the HoloLens 2. My setup involves a Python client that streams camera data, detects ArUco markers, converts their position and orientation to world coordinates, and then overlays a 3D model onto the markers. The transformed data is then sent back to Unity.

Now, let's delve into two different scenarios:

In the first scenario, I have an application running directly on the HoloLens 2, acting as a server to stream data to the client. This setup successfully synchronizes the cube's transformation with the received data. Regardless of my location, the cube is perfectly superimposed, And when i use mixed reality capture i can see the cube perfectly in its position.

However, in the second scenario, I'm running the application on a desktop. Here, I've merged the server application with a holographic sample player, essentially creating an application that serves as both a player and a server simultaneously. Despite mimicking the Python process from the first scenario, I encounter synchronization issues. Initially, the cube appears correctly superimposed, but upon slight movement, it becomes misaligned. Moreover, disconnecting and reconnecting the application results in the cube losing its position entirely. This discrepancy becomes apparent when inspecting the mixed reality capture, revealing a noticeable gap in the cube's positioning despite it appearing perfectly aligned through the lens.

I have seen a link recently talking about system coordinate synchronization between the custom sample player and a custom remote however there is nothing about custom player and Unity application do you any idea or suggestion ? Let me know if you need more details ,Thank you for you answer .

PS: By any chance do you know how i can get rid of the red windows in the first picture ?

Answer 1

It sounds like the coordinate system discrepancies between the HoloLens 2 and desktop application could be due to differences in how each environment handles spatial mapping; ensuring consistent reference frames might help with alignment issues.