Microsoft Photo Calibration Tool
Siddharth Jain, one of the devs on our team, wrote a tool using VE3D to place photos in 3D and save this information as a tag in the file itself. In short, you open a picture in the program, fly to roughly where the picture was taken from, click on points in the world and in the photo that you know are the same (for example, the chimney of a house, or the edge of a lake), and the program tries to determine where the picture was taken from. You can then view this picture in the world.
We have a picture that another dev on the team took in Las Vegas for you to play with. This picture already has some rough information in it to help you get started, and it might be a good idea to start with it. There's also an image that has already been placed, so you can see how it might look after you're done (load this picture into the tool instead of the first one to see it). Generally speaking your own photos will probably not have any positioning information in them to begin with, unless you have a very fancy camera, so it is important that you know pretty much where you were when you took it. Also keep in mind that this is a tool and demonstration, not a finished product.
Here’s Sid’s writeup of the tool. Enjoy!
Install
Download the zip file and extract its contents to any folder (there is no installer program). As long as all four dll/exe files are in the same directory it should just run by double-clicking the exe file.
Introduction
The Microsoft Photo Placement Tool can be used to find the geographic coordinates of the camera that took a photo. In this way a photo can be geolocated in the 3D world. There are seven parameters which need to be determined to geoposition a photo in the 3D world. Six of them are latitude, longitude, altitude, roll, pitch, yaw which define the position and orientation of the camera. The seventh parameter is the focal length of the camera that took the photo. The Photo Placement Tool allows determination of these seven parameters via manually specified correspondences between points in the photo and 3D world.
System Requirements
Virtual Earth 3D (VE3D) must be installed on user’s machine.
Usage
On running the exe following screen is displayed:
Step 1: User uploads a photo by clicking on Load Image button
Any exif information in the image such as GPS position and focal length is automatically loaded.
Step 2: User navigates in VE3D to a position from where s/he can begin adding correspondences
An example screen is shown below:
Step 3: User turns on calibration by clicking on Turn ON Calibration button
When you are ready to begin adding correspondences, click on the Turn ON Calibration button. This sets the initial camera position and orientation, which is then optimized as correspondences are added. To get back to step 2, click on the Load Image button again.
Step 4: User adds point correspondences between photo and 3D world
As the user adds correspondences via left mouse click, the tool calibrates the photo in the background. The Preview Result button is automatically enabled after adding 3 or more correspondences. An example is shown below:
While in calibration mode it is still possible to navigate in VE3D using the arrow keys, and change the view by holding the shift and ctrl keys while dragging with the mouse. Normal mouse drag is disabled in this mode.
Step 5: Preview the result by clicking on Preview button
When the RMS reprojection error becomes small (say less than 20 pixels), preview the result by clicking on Preview button. The smaller the error, the better the preview would be.
Step 6: Either Accept the result or Revise
User can either Accept or Revise. Clicking on Accept will display the solution and prompt user to save the calibrated photo as a JPEG. The latitude, longitude, altitude, roll, pitch, yaw, and focal length are saved as EXIF tags in the JPEG. Roll, pitch, yaw are relative to the local coordinate system at (lat,lon,alt). All angles (lat, lon, roll, pitch, yaw) are stored in degrees. Next time the photo is opened in this tool, the camera will automatically fly to that location.
By clicking on Revise user can continue to add more correspondences and refine the result. Note that as more correspondences are added, the RMS reprojection error may become better or even worse.
Miscellaneous Features
Use Locate feature to quickly fly to a location e.g. New York
Use the Locate button on the Menu to quickly fly to a location e.g. New York
Deleting a pushpin
Move mouse pointer over pushpin. Its color changes. Press delete key to delete the pushpin. Note that deleting a pushpin also deletes the corresponding pushpin, if any, in the other view.
The Settings Window
By clicking on the Settings button, user can control the parameters to optimize. The user would generally not need to change the default settings. However, for completeness the options provided by this window are described here. If the altitude option is unchecked, the tool will not optimize the altitude of the camera. For example, if you took a photo from the ground and have positioned the Virtual Earth 3D camera on the ground, you may want to uncheck the altitude button as the correct altitude has already been set by you. Similarly, if the focal length option is left unchecked, the tool will not optimize the focal length of the camera. You may want to do this, for example, when you are sure that the EXIF focal length is the correct focal length.
Technical Note: user may find that when a calibrated photo is opened, the camera does not fly to the exact calibrated location. Usually when this happens the camera is positioned too high. This is because the background data in the area, specifically the DEM, is not yet fully loaded, and the program wasn’t able to get to the right spot. Loading the image again should correct the problem.
Acknowledgements
Bill Chen in VE3D Research originally developed a photo geopositioning tool. His work can be found here. The present tool has been developed from scratch and is 100% managed code. I would like to acknowledge Nikolai Faaland, David Buerer, Bret Mulvey for helping me out on this project. David Buerer provided the Hilton Las Vegas image used in this writeup. Jonathan Doughi provided the Smith Tower image used in the writeup.
Comments
Anonymous
February 11, 2009
PingBack from http://www.clickandsolve.com/?p=6453Anonymous
February 11, 2009
Very cool! May I suggest that you show the pixel error on each pin, so it will be easier to locate the pin that messes up the least squares adjustment?Anonymous
February 11, 2009
So we will see a way to then view your collection of photos in VE3D as a fly through slideshow or something?Anonymous
February 12, 2009
Note: apparently there is an issue with the Locate feature. The workaround at the moment is to not use it :) Sorry about that. Note2: Generally speaking we can't discuss the future until it happens. But that doesn't mean we're not listening.Anonymous
February 20, 2009
Sort of the same question as above: What do we do with the calibrated pictures? Does VE3d understand how to display the calibrated pictures (i.e. can it use the altitude, pitch, etc), or will the pictures only work in the calibration tool for now?Anonymous
February 21, 2009
For now, the information is only used in the tool. In the long run, it would be great to have some sort of standard EXIF tag system where a variety of tools would be able to read and write the information, and cameras would populate as much of it as they could (you'd need a compass, accelerometer, GPS, and agreement on a datum and coordinate system). But that's obviously a little beyond the scope here. For this tool we just wanted to show what is possible.Anonymous
February 28, 2009
Greetings! I just came across this entry this morning and was very excited to try it but I'm getting server errors every time I attempt to download the zip. Would someone mind looking into this? Thanks. NateAnonymous
March 13, 2009
As noted, there are not standard Exif tags for some of the data needed. Which tags does the tool use to store the Yaw, Pitch and Roll? Exif Comment, or a Makernote? Thanks, MarvAnonymous
March 16, 2009
following tags are used to store roll, pitch, yaw: Roll = 0x001B Pitch = 0x001C Yaw=0x001D I will also make a subtle correction to the writeup where it says: All angles (lat, lon, roll, pitch, yaw) are stored in degrees. Correction: They are stored in degrees, minutes, seconds as mandated by EXIF specification.Anonymous
March 16, 2009
Thanks siddjain, I would like to suggest that you store the Yaw setting in the 0x0011 17 GPSInfo Exif.GPSInfo.GPSImgDirection field: Indicates the direction of the image when it was captured. The range of values is from 0.00 to 359.99. Since Exif doesn't have standards for Pitch and Roll, maybe an XMP data field would be appropriate. I too am trying to save this data in a standard format, but there doesn't seem to be one.Anonymous
March 16, 2009
Hi, Thanks for the suggestion. Quoting EXIF specification v2.2 (p.56): GPSImgDirectionRef Indicates the reference for giving the direction of the image when it is captured. 'T' denotes true direction and 'M' is magnetic direction. GPSImgDirection Indicates the direction of the image when it was captured. The range of values is from 0.00 to 359.99. It is not obvious that GPSImgDirection is same as the Yaw field of the Photo Placement Tool. It perhaps is not. Our reference for Yaw is the normal vector at the Earth's surface. I am not sure if this equals either 'T' or 'M' of GPSImgDirectionRef. Therefore, I chose not to use this tag.Anonymous
March 16, 2009
Correction to previous post: its page 50 (appears as 54/154 in Adobe Reader).Anonymous
March 16, 2009
I did some research, and according to definition of true north in http://www.princeton.edu/~oa/manual/mapcompass2.shtml, it seems the Yaw angle could be stored in GPSImgDirection field with GPSImgDirectionRef as 'T'. I assume the angle in GPSImgDirection field rotates about the normal vector to the Earth's surface.Anonymous
April 11, 2009
Is anyone else getting an error message after installing the 2009 April 9th VE3D update? I am now presented with the error message whenever I attempt to open calibration.exe: "Unable to find VE3D v3.0 (helios) assemblies. Program will exit" This could be due to any number of changes on my PC since I last opened the tool, but the reference to the version number of the assemblies made me instantly question whether this error was linked to the recent update to VE3D. If I have correctly diagnosed the cause of this error, could we please have an updated version of the tool or directions about what files to modify on our local machines? Thanks so much.Anonymous
April 13, 2009
Hi all, The tool requires a recompile against the new control. Sid will post an updated version as soon as he can.Anonymous
April 15, 2009
Sid has rebuilt the tool, and I've updated the link under "Install". Thanks for your patience!Anonymous
April 29, 2009
Actually, thank you for actually responding! I do have one question. As I'm sure you're aware, in Bill Chen's original tool his interface involved overlaying the photo on top of the VE3D view and then dragging points into agreement, rather than the traditional point for point correspondence marking. For my fellow readers, visit the following link in Internet Explorer: http://research.microsoft.com/en-us/UM/redmond/events/MSRNVideoContent/FacSum08/16080/lecture.htm and jump to around the 48:00 mark for Bill Chen demonstrating the original tool. (If you jump a little further back you'll see him viewing a Photosynth of Smith Tower correctly positioned inside of Virtual Earth... I'm unbelievably jealous!) My question is twofold: When will Sid be given the time to implement Bill's original drag and drop/toggle interface as a secondary option (given that the initial testing recorded that results using that interface were slightly less accurate on average than the current interface), and also when might this photo calibration tool be packaged into Microsoft Pro Photo Tools? Looking forward to hearing back from you guys.Anonymous
June 08, 2009
I 2nd Nate, love to see the overlay version. It seems that others are also trying to automate this entire process using terrain mapping: http://www.opengeodata.org/?p=552 very exciting.