Best practices using the FAST Enterprise Crawler

The FAST Enterprise Crawler can be used in situations where the SharePoint 2010 Web Crawler is not crawling as expected. The FAST Enterprise Crawler has been deprecated and should be used as a secondary option. Please file a ticket with support if the SharePoint 2010 Web Crawler is not meeting your expectations.

• Do not use the "sp" collection with with the FAST Enterprise Crawler, let the SharePoint crawler manage that collection. 

• Use the powershell cmdlets for creation, deletion, and clearning collections for the FAST crawler:

  • New-FASTSearchContentCollection
  • Remove-FASTSearchContentCollection
  • Clear-FASTSearchContentCollection
  • Get-FASTSearchContentCollection

• Administration of the crawler should be done via crawleradmin

• The most in depth documentation on the crawler can be found in the FAST ESP Crawler Admin Guide. Just ignore all the user interface (which is non-existent in FS4SP). The latest is Version 6.7 and is dated December 2009 (revision B)

  • http://www.bing.com/search?q=2009+FAST+enterprise+Crawler+admin+guide+5.3&qs=n&form=QBRE&pq=2009+fast+enterprise+crawler+admin+guide+5.3&sc=0-39&sp=-1&sk=
  • http://nesfast1.scot.nhs.uk:16089/help/pdfs/assets/pdf/Enterprise_Crawler_Guide.pdf

• You cannot run PostProcess and CrawlerAdmin crawls at the same time. Only one can be active at any given time.

• Under most circumstances, the amount of content that will need to be crawled will be small (under 100K documents). Running a multi-node FAST Crawler in FS4SP is likely too complicated.

• Backup your crawler XML file!

  Adding a new crawler to a FS4SP node:

  • nctrl add crawler
  • make sure the browserengine is running (not sure if it needs to be on that node), you may need to edit the nodeconf.xml to add it

  TechNet page on the FAST Search Web crawler: http://technet.microsoft.com/en-us/library/ff383296.aspx

  TechNet page on Monitoring the FAST Search Web crawler: http://technet.microsoft.com/en-us/library/ff383268.aspx