Integrating SharePoint FAST Search into Commerce Server

by Marc Bednar and Eric Dunn-Cuillierrier, Commerce Server Development Team

With the evolution of search technology, it’s common practice to find features such as advanced and faceted searches with most modern day ecommerce web sites.  Search based navigation is also a feature that’s much more commonplace.

There are many 3^rd party search engines available that can provide these features, and more, beyond the out of the box search offering with Commerce Server.  For partners that are implementing Commerce Server, the search engine of choice may differ between clients, so it may be practical to devise a search framework that can handle integration with any search engine and not impact your presentation layer.

We’ll discuss an option below, using SharePoint FAST as an example, to illustrate a possible integration between Commerce Server and a 3^rd party search engine.

The basic integration is the same regardless of the search technology:

• Exposing the Commerce Server data that you want indexed
• Crawl the data
• Create operation sequence components to handle queries against the search engine

Exposing Commerce Server Data to Search

FAST contains its own backing store for the data to be used as part of search.  We need to get Commerce Server data into the FAST so that it can be initially crawled and indexed.  In addition, any changes that are made within Commerce Server have to periodically make their way to FAST so that the search data is kept reasonably up to date.

While you can likely imagine several ways to move the data from Commerce Server to FAST, we’ll explore the scenario of using a custom data extraction tool.

A custom tool is written that uses the Commerce Server API to query the catalog system, specifying the properties to crawl, for all categories and products.  Use of the LastModified timestamp would allow FAST to know what has changed and when.

This could be configured to run, on a periodic basis, as either a Windows Scheduled Task or by Commerce Server Staging as a post operation to the staging process.

The main purpose of the tool is to update the data within FAST to the latest data from Commerce Server.  Once available, that data could be placed within a temporary table in SQL Server and a FAST crawl would be initiated.

This solution enables a transformation to be applied to the Commerce Server data before placing it in a temporary table in SQL Server.  The recommended mechanism to get your catalog data from the temporary tables to FAST is to use the SharePoint Business Connectivity Services (BCS) SQL Connector.  You can use SharePoint Designer to define your external content type, and export a bdcm file that can be stored with your project.  The bdcm file contains information about your external system and how to communicate with it.  You need to configure the BCS to use the bdcm file you exported.

Once BCS is configured, a new Search Content Source needs to created under your FAST Content Search Service Application.  You can now point a content source to your BCS setup by setting your Content Source Type to Line of Business Data, and selecting the external data source you defined from the provided list.  When a crawl is initiated using this content source, your SQL table will be indexed through the Business Data Catalog (BDC) and pushed into FAST.

Optionally, you can further manipulate the data as it is being indexed through the use of the Pipeline FAST Extensibility framework.  As each “document” is being indexed, you have the opportunity to manipulate, or even create new, fields.

A practical example of a static refinement is pricing groups ($0 - $20, $21 - $50, etc).  As each document is being indexed by FAST, you could determine which category of pricing the item falls within, and add an appropriate pricing group name as a value into a new field called “Pricing Group”.  Within the FAST Management area of SharePoint, you would create a new managed property called “Pricing Group” and indicate that it can be used for refinements.  You’d map this property to the crawled property named “Pricing Group” and that property can now be used to allow customers to refine products based on static pricing groups.

A possible disadvantage to this approach is the amount of time it could take to export the data and move it into SQL Server.  Obviously, this is dependent upon the size of your Commerce Server data sets.  Performance could be optimized by generating a .sql file and using that to bulk insert the Commerce Server data into the SQL Server temporary table.

The following diagram illustrates the overall data flow for this scenario.

 

Data Concurrency

FAST treats deleted items differently from Commerce Server.  In Commerce Server, deleting a product performs a “hard delete” whereby the product is removed from the database.  FAST will assume that the data item still exists, but just hasn’t been updated.

To ensure FAST knows that an item has been deleted, a flag must be set on the delete items to that FAST knows to remove it from its indices.  It is important to take this into consideration when planning how data will be populated in FAST.  For example, set a flag in the deleted item’s properties as being deleted, then crawl the data, then do the proper delete.

Surfacing FAST Results with Commerce Server Foundation

With the Commerce Server data available in FAST, you need a way to access it for presentation upon the shopper facing web site.

In this section we explore how to define a generic search mechanism used by the front end web server and how one could implement a specific SharePoint FAST search processor in the backend to provide search results and facets.  Using Commerce Server Foundation, you will need to define new models for the search query and the search results.  In addition, leveraging the extensibility points of Commerce Foundation, you will need to create Operation Sequence Components to interpret the search parameters and invoke the actual search against FAST.

An example of models for a SearchRequest and SearchResponse follow:

SearchRequest

Property	Description
Keywords	The keywords entered by the shopper
SpellCheck flag	A flag indicating if spell checked versions of the given keywords should be returned
Search Scope	Scope of the search (e.g. catalog search, content search, etc.)
Search Scope Culture	Culture to be used along with the Search Scope – useful in multi-lingual sites
Refinements	List of refinements or facets to be returned by the search engine (if supported)
Refinement Filter Token	Current refinement token (aka Facet) that was applied to a previous search

SharePoint FAST enables search scopes, which are used to inject additional query filter parameters into the given search.  The parameters allow the isolation of specific rules in search scopes without having to add the specific knowledge through source code.

As an example, search scopes can be used to target different types of content such as product catalog or other content maintained by your site.  The scope can be provided as part of the Commerce Server query and applied accordingly in the back end when issuing the FAST search.

Product catalog search scopes can be defined to add the necessary query filter parameters that target the content type and culture properties that were indexed by FAST.  Take for example the following catalog search scopes:  CatalogSearch_EN_US and CatalogSearch_FR_CA.  These 2 scopes, as the name suggest, support querying catalog content for a specific culture.  In the backend, when Scope and Scope Culture are supplied, the code could include the search scope into the FAST query by combining the search scope name and the given culture.

SearchResponse

Property	Description
SpellCheckedResult	Array of spell checked values
SortableProperties	Array of properties that can be used for sorting
ActualResults	The actual search result represented in a model containing a series of key / value pairs as you would expect from your search engine
AvailableRefinements	A relationship of available refinements or facets that can be applied along with their refinement tokens

Operation Sequence Components

Since the SearchRequest is a generic model, it promotes the creation of generic Operation Sequence Components that could handle request criteria validation or spell checking.

A search engine specific Operation Sequence Component could be created for each search engine supported.  In our example an Operation Sequence Component that invokes a search against FAST.  Its purpose would be to format the given search parameters into a format used by FAST, invoke the SharePoint Query Service with the request, interpret the response and convert the returned items into the SearchResponse model.

Breaking down the Search process into individual Operation Sequence Components allows for the replacement of the SharePoint FAST component with another component targeting a different search engine, yet leaving the common functionality intact and reusable.