Visualising media search better (Part 2) - Kevin Pfister
The way Image search results look now hasn’t really changed much in the last few years, I ask the simple question ...why? In this article I want to try something that hasn’t been done before; combining Deep Zoom and Live Image Search. I call it DeepSearch ...Sounds awesome!
In the last article we used a pseudo 3D effect in Silverlight to show one way of visualising Images we have searched for.
However just showing the images was a bit bland, even with a 3d effect. We needed to actually be able to manipulate the images better, and so people I give you DeepSearch!
This has only been made possible by the changes made to the MultiScaleImage control (Deep Zoom to us common people) within Silverlight Version 2 which allows us to dynamically generate our own MultiScaleTileSource. In the ye olde days the MultiScaleImage control would link to an external XML file via the Source property; however we can override this via the abstract class MultiScaleTileSource which has a method GetTileLayers() that we can override.
protected abstract void GetTileLayers(
int tileLevel,
int tilePositionX,
int tilePositionY,
IList<Object> tileImageLayerSources
)
Where this gets interesting now for us is creating the Dynamic tiles that we are going to supply to the MultiScaleImage control. Silverlight 2 doesn’t contain really anything in the way of pixel based bitmap manipulation client side, so what we need to do is push all that code over to the server. Mike Ormond and Mike Taulty have both done great blog posts that explain this problem and a way to get round it. I will be using the same type of methods they suggest. We will remote the requests to server side code so using an ASP.Net ashx handle so that we can use System.Drawing to enable us to have the bitmap manipulation we so desperately need, the only problem as both the Mike’s so rightly point out is the problem with the lack of caching causing server side congestion which will cause our code some problems later. Now to start off with I used most of the code base downloadable from Mike Taulty’s blog entry to speed things up a bit... here’s it running with quite a lot of changes made to the Silverlight front end. (One of the ideas of how it could have looked)
Some of you may notice that in Mikes DeepZoom image only showed the number whereas mine also shows the text searched for; in this case Layer. All that had to be done was simply link send the search text to the ashx handler along with the rest of the deep zoom configurations.
string source =
string.Format(
"https://localhost:62586/ClientBin/handler.ashx?" +
"tileLevel={0}&tilePositionX={1}&tilePositionY={2}&tileSize={3}" +
"&imageWidth={4}&imageHeight={5}&imageQuery={6}",
tileLevel, tilePositionX, tilePositionY, tileSize,
imageWidth, imageHeight, QueryString);
Having the bitmaps being generated server side also has a benefit to us as well; we will also be processing the search results server side as well due to Silverlight using a subset of System.Xml and also is a simple way to negate Silverlight’s Cross-Domain problem.
To use the Live Search API’s as part of our project we need to download the Live Search SDK and then acquire an AppID; for more information on the Live Search SDK you can visit the site.
There is a great example as part of the Live Search SDK which shows how to perform a Live Image Search and display the results; we only need to alter the context as it has been designed to run from a console rather than a web service... easy enough you would think.
However there is a big problem that we will have to overcome, the example I mentioned above will just search for a set of images and display the links to the results. If we plug this into the Server code; based on the code example Mike T provides then every time we want to change the depth or viewport we would have to perform the same search again and download the same set of images; thus we need to implement a simple image caching system on the server.
To do this we need to implement SessionState variables in our ashx handler class. This can be done by firstly importing System.Web.SessionState and having the class inherit IRequiresSessionState. What we are going to store in the SessionState is the search results of a given search and also the images that are retrieved. Now this will be A LOT of data if it were pushed out onto a production server however I am only using this method as this is really only a proof of concept. To store the search results I am made a struct to store the information retrieved from a search.
private struct SearchItem
{
public string MediaURL;
public string PageTitle;
public string PageURL;
public Size MediaSize;
}
We need to replace the example Live Search SDK code so it will now store search items in this type of struct and then store them as a List<SearchItem> in the SessionState using the SearchQuery as the key. Also as a sidenote, the SessionState is saved as part of the HttpContext we need to promote the variable to a class variable rather than local.
// List of the image results
List<SearchItem> _SearchItems = new List<SearchItem>();
// Save the Image results.
foreach (XmlNode result in results)
{
SearchItem _Result = new SearchItem() {
MediaURL = result.SelectSingleNode("./mms:MediaUrl", nsmgr).InnerText,
PageTitle = result.SelectSingleNode("./mms:Title", nsmgr).InnerText,
PageURL = result.SelectSingleNode("./mms:Url", nsmgr).InnerText,
MediaSize = new Size(
int.Parse(result.SelectSingleNode("./mms:Width", nsmgr).InnerText),
int.Parse(result.SelectSingleNode("./mms:Height", nsmgr).InnerText)),
ThumbnailURL = result.SelectSingleNode("./mms:Thumbnail/mms:Url", nsmgr).InnerText };
_SearchItems.Add(_Result);
}
// Save the results to the session state
httpContext.Session[queryString] = _SearchItems;
Now to test if this all work I changed the rendering code so instead of displaying text in the deep zoom it would instead display the first image of the search results list; in this case... Bill Gates
Now comes the big question; how do we want to display the images within the Deep Zoom? To give you one idea of what we could do is just panning the images across and then wrapping around at the edges, and so I give you the base of our deep zoom search.
One thing I had to do which I found very interesting was that if I just let told the graphics device to render the bitmap that sometimes it wouldn’t render it to the correct size so I made it render to the size that the search engine returned instead. To make transfer between server and client a little faster I also altered the context.Response.ContentType of the handler response to use jpeg instead of png, hopefully the lossy compression shouldn’t be noticable too much when compared to the bandwidth savings.
context.Response.ContentType = "img/jpeg";
...
bitmap.Save(ms, System.Drawing.Imaging.ImageFormat.Jpeg);
Let’s go back to the UI now seeing as the search functionality is almost done. Now at the start I mentioned that the project was based on Mike Taulty’s underlying codebase, however he didn’t implement the full set of Deep Zoom mouse controls, so we have to add in MouseWheel support which isn’t as hard as you would think as any project created by Deep Zoom Composer already contains this code.
With some final adjustments to the UI with a nice shiny welcome page; making it more like a normal search engine I think I’ve done a good job on dull winter’s afternoon.
Follow the continuing development of DeepSearch at my Blog: www.blogplusequals.blogspot.com
If you want to try out DeepSearch, download the source code from MSDN