Interacting with 2D on 3D in WPF
Interacting with 2D placed on 3D is now possible in v1 of the Windows Presentation Foundation!
In between shipping Vista and planning the next version of WPF, we realized that with a clever implementation it was possible to provide this feature today on v1 bits, and I’m happy to report that as of now, the binaries (and source code to them) needed to do this are available for download (here).
The download also contains two sample applications - InteractiveViewport3DSample and Channel9Demo - which illustrate how to use the code. So if you just want to dive right in to 2D on 3D, that's a good place to start.
The rest of this post will describe how to use this feature as a developer, as well as how it’s implemented behind the scenes. If you want the video version though, which includes a cool demo of an app we made with the 2D on 3D code (photo shown below – source here https://www.codeplex.com/3DTools/Release/ProjectReleases.aspx?ReleaseId=2058), check it out on Channel9 (here).
How do you use it?
There are two main classes needed to add 2D on 3D to you app: Interactive3DDecorator and InteractiveVisual3D. The Interactive3DDecorator handles the majority of the work that allows the 2D on 3D interaction to happen. To use it, you simply place it around the Viewport3D that you intend to make interactive. So in XAML, you have:
<local:Interactive3DDecorator>
<Viewport3D>
…
</Viewport3D>
</local:Interactive3DDecorator>
Then, to actually add interactive 2D on 3D objects in to your scene, you use the InteractiveVisual3D class. InteractiveVisual3D is a subclass of ModelVisual3D and provides the following dependency properties which set up the interactive 2D on 3D:
- Geometry - The Geometry3D that is to become the content of the InteractiveVisual3D .
- Visual – The 2D you want to interact with on the 3D geometry.
- Material – A user specified material to use on the 3D object. By default, a DiffuseMaterial is used.
- IsBackVisible – Indicates whether the material used for the front face should also be mirrored on the back face.
The first two properties, Geometry and Visual, are the primary ones needed. They allow the geometry that should be used to bet set, as well as the Visual that should appear on that geometry. For example, the below XAML code demonstrates how to create an InteractiveVisual3D.
<local:InteractiveVisual3D Geometry="{StaticResource PlaneMesh}">
<local:InteractiveVisual3D.Visual>
<StackPanel>
<Label Content="Sample UI" />
<Button Content="Close Window"/>
<TextBox />
</StackPanel>
</local:InteractiveVisual3D.Visual>
</local:InteractiveVisual3D>
The other properties allow for fine tuning of the object’s appearance. The Material property allows a user to create their own material for the object. To do so, a user can create a material as always, and then specify using the IsInteractiveMaterial attached property which material is intended to be “interactive” (i.e. the VisualBrush created using the passed in Visual is set as the Brush for that material). Note, this attached property must be set to true on at least one material – otherwise an exception will be thrown. If you ever want to disable interaction, you can set “IsHitTestVisible” on the Visual to be false. As an example of setting a custom material, the following code sets the material to be composed of a DiffuseMaterial, which will contain the visual brush, and a SpecularMaterial.
<local:InteractiveVisual3D.Material>
<MaterialGroup>
<DiffuseMaterial local:InteractiveVisual3D.IsInteractiveMaterial="True"/>
<SpecularMaterial Brush="Red" />
</MaterialGroup>
</local:InteractiveVisual3D.Material>
Finally, IsBackVisible sets the material used on the front face to also be used on the back face. Currently, the Interactive3DDecorator does not distinguish between front and back faces, so this allows for there to be interaction with the back face as well.
How does it work?
At a very high level, the interaction with 2D on 3D is achieved by really interacting with a hidden version of that 2D content in 2D. The 2D is positioned such that the point in 3D the mouse is over is the exact same point as the mouse is over on the hidden 2D version. Then, when the user clicks, etc… they are interacting with exactly the same location. If you want to see this for yourself, you can set the “Debug” property on Interactive3DDecorator to true, which makes the hidden layer partially visible.
The Interactive3DDecorator then consists of two elements: the 3D content that is displayed within it and a hidden layer that is used to position and display the 2D content that is being interacted with. Depending on what 2D content on 3D is being interacted with, the hidden layer changes to hold that 2D content. The images below give a visual representation of what is going on. The first photo is just of the 2D content we wish to place on 3D. The second is that 2D content on 3D. And finally, the third image shows how the 2D on 3D interaction takes place. When the mouse moves over the “B” in the button, the hidden layer (shown to be only partially transparent for this example) is moved such that the mouse is over the same point in the hidden layer as it is in the 3D scene.
To figure out where to position the hidden layer works as follows. When the mouse moves in the 3D scene, a ray is shot in to the 3D scene to see if it intersects any object. If it hit an object, and it is an InteractiveVisual3D, we can use the return parameters from the intersection to compute the texture coordinate that was hit. Then from these, we can map from the (u,v) value of the texture coordinate, on to an x,y point on the 2D visual, which is the point we need to place under the mouse. More specifically, the code assumes texture coordinates are all in the range (0,0) to (1,1) – i.e. upper left to lower right of the image (for those planning to use this, this is important to know, since your texture coordinates need to be within this range to enable interaction with the 2D content). Then the point on the 2D object that was hit is simply (u * Width, v * Height).
One of the great things about this method then is that the only event that needs to be tracked is when the mouse moves. There’s no need for catching and forwarding on every type of event: the hidden layer takes care of all of this work.
Positioning With Capture
There’s one very interesting “gotcha” though: what happens when one of the 2D objects grabs capture and then you move off the 3D mesh it is on? For example, you select some text, and then move the mouse above that selected text, or click and hold on a button, and then move away from it. Correct hidden content positioning becomes more complicated in this case. The problem becomes difficult for many reasons. In the normal 2D situation, both the mouse position and the 2D content exist in the same plane. The transformation that is applied to the 2D content can be used to transform the mouse position to the content’s local coordinate system. However in 3D, due to the projection of 3D on to a 2D plane, the mouse’s position actually corresponds to a line in 3D space. In addition, the element with capture could also be mapped to any arbitrary geometry. When the mouse is over the 3D object, hit testing tells us where it is relative to the 2D visual. When it is off the 3D object, due to the above issues, there is no longer a straight forward answer to this question: the 2D point corresponds to a 3D line and the 2D content could be on arbitrary 3D geometry. Also, because the element has capture, it wants to receive all events. Before, we only needed to be sure that the mouse was over the correct object at all times. Now we need to position the hidden visual such that it is in the proper position relative to the object that has capture.
To solve this issue, the code takes the following approach: The overall idea is to reduce the 3D problem back to 2D. In the normal 2D case, the transformations applied to the content can be used to convert the mouse position to the content’s local coordinate system. This transformed position then lets the content know where the mouse is relative to it. In 3D, due to the many orientations of the geometry and texture coordinate layouts, it’s difficult to say where a 3D point is in the relative coordinate system of the 2D content on 3D. To approximate this, the outline of the 2D content on 3D, after it has been projected to screen space, is computed, and then the mouse is positioned based on this projection.
For example, take the 3 images below. The first is the 2D content on 3D. In the second image, text is selected, and the mouse is moved to a point off the object. The third image is the outline of the text box (the object that has capture). This outline is then used to position the hidden visual.
After the outline is available, The closest point on this outline to the mouse position is computed, and then this point on the outline is considered what was “hit” and it is placed under the moue position (hence, highlighting up to the “T” in the middle image below). Since we place the mouse by the closest edge point, the interaction tends to behave as it would in 2D, since we position the hidden content based on what the mouse is closest to on the 2D content on 3D. By placing it at the closest edge point, we are explicitly stating about where we expect the mouse to be relative to the 2D on 3D element’s orientation.
This method helps provide an intuitive response to the interaction, since the interaction happens with the closest point on the object with capture to the mouse.
Enjoy using it! We’ve had a lot of fun building this, and can’t wait to see what kind of incredible applications will be built with it!
-Kurt
Comments
Anonymous
December 13, 2006
Well, it's finally here. As mentioned yesterday, the WPF 3D team had a very special announcement to makeAnonymous
December 13, 2006
Windows Presentation Foundation, sorti en RTM il y a un mois, apporte le support de la 3D comme vousAnonymous
December 13, 2006
[Update: Interactive 2D controls on 3D has arrived! Read the blog , watch the video , then get the codeAnonymous
December 13, 2006
Interacting with 2D placed on 3D is now possible in v1 of the Windows Presentation Foundation! In betweenAnonymous
December 13, 2006
Interacting with 2D placed on 3D is now possible in v1 of the Windows Presentation Foundation! In betweenAnonymous
December 13, 2006
Interacting with 2D on 3D in WPF Holy hotness! Check out details on the WPF3D Blog Check out the Channel9Anonymous
December 13, 2006
Già da qualche giorno si vociferava un importante annuncio da parte del team di sviluppo di Windows PresentationAnonymous
December 13, 2006
With the 3D Tools for the Windows Presentation Foundation library, which is now available on CodePlex,Anonymous
December 14, 2006
The WPF3D team have delivered one of the coolest features yet: Fully interactive 2D controls and contentAnonymous
December 15, 2006
I'm sorry, but this is just too cool. Congratulations and thank you!Anonymous
December 15, 2006
A few links that I missed in being out of the office for about 24 hours :-) You can't blink these days...Anonymous
December 16, 2006
What about Frame control?Anonymous
December 18, 2006
What do you mean by Frame control? -KurtAnonymous
December 19, 2006
News & Highlights BarCamp Toronto Presents Enterprise Camp - The Chicken Test Debug Leaky Apps: IdentifyAnonymous
December 19, 2006
Very good job, guys ! Can you tell us some more about how mixing some 2D Interactive Visual in Model3DGroup GeometryModel3D ? About change from standard nested classes: <ModelVisual3D> <ModelVisual3D.Content> <Model3DGroup> <GeometryModel3D> <..............> <..............> to your custom classes ? AlbertoAnonymous
December 19, 2006
Very good job, guys ! Can you tell us some more about how mixing some 2D Interactive Visual in Model3DGroup GeometryModel3D ? About change from standard nested classes: <ModelVisual3D> <ModelVisual3D.Content> <Model3DGroup> <GeometryModel3D> <..............> <..............> to your custom classes ? AlbertoAnonymous
December 20, 2006
Which things within the Model3DGroup are you looking to make interactive? What you'll need to do is break those that you do want to be interactive out of the Model3DGroup, and make a new InteractiveVisual3D for them. Then set the InteractiveVisual3D.Geometry property to be the same as the Geometry for the GeometryModel3D, and set the Visual property to be whatever interactive content you want to place on that Geometry.Anonymous
December 20, 2006
The comment has been removedAnonymous
December 26, 2006
Primero que nada, Feliz Navidad !!. Ahora si, hay un articulo excelente sobre como colocar los elementosAnonymous
January 02, 2007
There’s been a super exciting update in WPF’s 3D arena that’s been recently announced that I want toAnonymous
January 02, 2007
One of the really cool features of WPF is its ability to place 2D content on 3D surfaces. For example,Anonymous
January 08, 2007
The final frontier… This is the last set of WPF features that I want to cover before moving into a seriesAnonymous
January 15, 2007
somehow the screenshots are not showing.. :sAnonymous
January 16, 2007
Nella versione odierna di WPF non è sempre facile rendere interattivi contenuti 2D che vengono mappatiAnonymous
May 13, 2007
hi all, very good job and we wait more. i have a problem in that. i tried to do that using blend on a cube created by zam3D i created an InteractiveVisual3D element and set it's geometry as one side of a cube and it's Visual as a Grid with some contorls, but it didn't work well the output cube has a mask on this side and no controls on it.Anonymous
June 25, 2007
Found the following interesting links while surfing the web: Snack Tutorials for the Hungry Celso GomesAnonymous
October 09, 2007
I have added a slider to an InterativeVisual but the events dont fire... any clues? thanksAnonymous
October 27, 2007
As I post in this thread : http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=2326393&SiteID=1Anonymous
June 19, 2008
Hi, this is great stuff! Thanks! I have a question regarding interaction with 3D objects. I was able to interact with single 3D objects by the method described here. However when the 3D objects are overlapping, I can see the inner 3D objects by setting the opacity of the outer one, but I could not interact with the ones that are inside. For an example, if I have two cubes, one located completely inside the other, whenever we click on the inner one, the outer would respond. If we set the outer cube's IsHitTestVisible to be false, the clicks would not be able to activate the inner cube's click event handler. I wonder if there is any way that could help me get around this. Any comments would be greatly appreciated. Thanks!