Chronicles of an IFilter development - inception to deployment.
I often asked myself the question - How do independent vendors develop IFilters for MS Search Products and what are the challanges they face? It occurred to me that if I could somehow document the development lifecycle of an IFilter developed by someone other than Microsoft, it'd probably provide answers to a lot of baffling questions facing independent vendors developing these components.
I recently had the oppurtunity to have a detailed discussion with Marco van Schagen from CAD & Company, who has just embarked on a fascinating voyage of refactoring the CAD(DWG) IFilter.
This thread is meant to address the issues faced by Marco which may be of broader interest to several of us implementing our own filters.
In Marco's own words:
Currently I am planning a new version of our 2005 version DWG iFilter. This is to support the newer 2007 DWG file format, and address questions on it's operation with SQL 2005 and MOSS (sharepoint) 2007.
Would you be interested in information on this product, please visit http://www.cadcompany.nl/ifilter
As I am unexperienced with iFilter development, I have many questions to find answers for.
In my preparations, Deb Haldar has provided me with crucial information to help me get on the right track. I would like to share this information in his blog to help make this the "One stop shop" for IFilter related issues. Probably a seperate thread will be created to track the development cycle of an iFilter from scratch.
The existing 2005 version iFilter project is coded mostly in C++, in VC 6.0.
Some questions I'd like to discuss:
- Should we use C++ or transfer to dot Net and why?
- What is required for coding a proper iFilter
- Testing for Multithreading compatiability
- Registration with Sharepoint and SQL 2005
I would like to start sharing my information soon, and I am interested in your comments.
Marco van Schagen
Comments
Anonymous
December 24, 2006
Deb, thank you. I will start posting very soon.Anonymous
December 30, 2006
The comment has been removedAnonymous
December 30, 2006
Deb, In our implementation we have a config loader written in VB dot Net. This is called each time the iFilter is called. If I understand you right, calling this dot Net section from our unmanaged C++ iFilter also causes a performance hit? If so, if I cannot avoid this performance hit anyway, would I make things much worse if I write the iFilter itself in dot Net?aAnonymous
December 30, 2006
Marco, Is your config loader used to just initialize the filter with appropriate parameters required by the IFilter:Init() ? If so, I'd still retain the C++ implementation as once the filter is initialized, it'll not have to go through Interop anymore. Typically Init() is called once whereas the other IFilter methods can be called thousands of times depending on the size of the corpus/document.Anonymous
December 30, 2006
Deb, You are absolutely right, the config loader is called only once. So, if I would like to call some code trough then Com Interop - that is perfectly okay. I just need to be very aware to do this only once in the IFilters' lifetime, and not, for example, for each chunk or value.Anonymous
December 30, 2006
The comment has been removedAnonymous
January 25, 2007
We are making good progress on our new DWG IFilter 2007 development project. Here is an update, for those interested in the DWG IFilter: I have completed the switch to a different library to read the dwg files, so now we are fully compliant with AutoDesk. This means we now read dwg files up to the 2007 file format. The IFilter specific part of the code is still based on the smpfilt example. Next step is to finish all other parts and create the completed product. If the tests are succesful we will greate an installer package ready for shipping. You can expect the product to be available as planned in 2007 Q1. Would anyone be interested in beta testing, please drop me an email. We are now collecting requirements for our next release, DWG IFilter 2008. If required we will add support for the 2008 dwg file format. Also we will add support for indexers where not available in our 2007 version. Also, this release is expected to better support SQL 2005. Customers who obtain the 2007 version can expect to update to the 2008 version free of charge. Currently I am collecting information to start planning the 2008 version. Deb, I do have a question on registration. In my current implementation, registration of our filter is based on FiltReg.Hxx, as provided in the original simpfilt example. This works like a charm, I expect this is the perfect solution for most occasions. I wonder to what extend this FiltReg.Hxx also supports the newer indexers; and for which products do I need a different approach. It would be great to have an updated version available if needed.Anonymous
January 25, 2007
Deb, I would like to know about registration for SQL 2005. As you mention here http://blogs.msdn.com/ifilter/archive/2006/12/24/extending-sql-server-2005-full-text-search-with-custom-ifilters-and-iwordbreakers.aspx Shajan has good information on registration an IFilter for SQL 2005. This registration for SQL 2005 seems be a bit of a complex thing. Would there be some helper code or macro available for this task? I do like the RegFilt.Hxx approach as provided with the simpfilt example. I hope these tasks can be fully automated (as with FiltReg), if not, what manual steps do you expect we need to document for our users? Is it required to copy the filter dll file to specific locations for each SQL server instance, or could this be rerouted to our install folder? As posted here http://www.palissimo.de/viewtopic.php?t=36&highlight=ifilter and here http://www.palissimo.de/viewtopic.php?t=39&highlight=ifilter it sounds like it is required to create a signed IFilter dll. I believe dll Authenticode signing is described here http://msdn2.microsoft.com/en-us/library/ms838270.aspx#nativedevicedev_topic7 at 'Project: Authenticode Signing with a Test Certificate" and here http://blogs.msdn.com/shawnfa/archive/2005/07/14/438963.aspx . Can you confirm this information to be applicable to IFilters in SQL 2005? What kind of certificate would we need. Do de use certificates we create ourselves or may it be preferrable to obtain one at a certificate provider? By the way, is a certificatee also required for 64 bit installations? I heard those environments are more restrictive on what 3rd party software is accepted.Anonymous
January 27, 2007
Marco, it's good to know your product is coming along well! For registering the filter, the self registration code kicked off by regsvr32 takes care of the indexing service registration.If you are targeting specific MS Search products, then you need to make sure their registration requirements are also satisfied.For MOSS and WSS, this info is already posted on the blog.Currently, MOSS falls back to indexing service registration if it does not find specific filters registered for the content its crawling.However, this is not a recommended solution as the fallback on indexing server may not be supported on future versions of the product.Please ensure that any keys not set by regsvr32 for the targeted product is set by your installer package. In the near future the Enterprise Search group will the sole distributor of filters to other MS Search products through the filter pack initiative. At that point, we'll be able to provide a more concrete and unified solution to the registration issue. As a sidenote, WDS 3.0 loads via IPersistream and hence the smpfilt might not work there. It's be a good idea to double check this is not the case with DWG filter. Also, a good way to debug registration issues without the pain of hooking the debugger is using Regmon from sysinternals: http://www.microsoft.com/technet/sysinternals/ProcessesAndThreads/Regmon.mspx -Deb.Anonymous
January 27, 2007
Deb, I am wondering, which are the MS Search products and versions we could be targetting now. So it would be great to publish a table listing the Search products and versions, with their IFilter registration issues and requirements. Possibly there will be differences among platforms, or 32/64 bit versions. We would like to see if each will accept default registration, as provided with FiltReg (even if only for backward compatibility). Where will we run into problems when we stick to the simpfilt example. Is digital signing required, optional, or not applicable. If we are forced to have an additional copy of the ifilter dll in a certain location. You could possibly add many more items.Anonymous
January 27, 2007
Deb, For a future release, I'd like to have my IFilter return a different response depending on the Search product that is beeing used. When I would be able to differentiate here, it would be possible to allow an unregistered, free version to be used for, for example, desktop filesystem use only. A registered version would be required for example in a server environment, with certain search product. Would you have an idea how I could do this?Anonymous
January 29, 2007
Marco, I just verified with our Security contact that "Digital Signing is a requirement". We recommed that you use a trusted certificate provider such as Verisign. This is the company which currently countersigns all our vendor binaries as well. Here's a complete list of MS approved certificate providers: http://msdn2.microsoft.com/en-us/library/ms995347.aspxAnonymous
January 29, 2007
The only way that I can think of is as follows: Determine the id of the calling process and map to the calling process name, such as <mssearch> for MOSS or <cisvc> for indexing service. However, if the service name changes from one version of the targeted product to another, the DWG filter will break.Anonymous
January 29, 2007
To make the question posed by Marco suitable for a broader audience, the issue has been broken down into three parts. John Kane, who has worked with IFilters in all versions of SQL Server since FTS was first incorporated in SQL Server 7.0 Beta3, was kind enough to share his experience with us.----------------------------------------- <Deb> For registering a filter with SQL server, is it necessary to to copy the filter dll file to specific locations for each SQL server instance, or could this be rerouted to the install folder? <John> Neither SQL Server 2000 or SQL Server 2005 require the IFilter dll to stored in any specific location, just that it be properly registered and working. However, SQL Server 2005 FTS has added a new level of security for 3rd party developed ISV IFilters. Specifically, SQL 2005 FTS requires certain settings be disabled for security reasons via: "sp_fulltext_service" - "Enabling use of OS resources provides access to resources for languages and document types registered with Microsoft Indexing Service that do not have an instance-specific resource installed" sp_fulltext_service 'verify_signature', 1 sp_fulltext_service 'load_os_resources',0 Additionally, you will need to stop and restart the SQL Server and MSFTESQL services after the above changes are made. Then you can query the sys.fulltext_document_types system table to get a list of all IFilters used by that instance of SQL Server 2005. The above steps are not required by SQL Server 2000 FTS as it will use all successfully installed and registered IFilters. Furthermore, the following related SQL 2005 KB article if they have 'verify_signature' enabled and the server does not have internet access, you will want to disable it via: sp_fulltext_service 'verify_signature', 0 - "You may experience a 45-second delay when you run a full-text query in an instance of SQL Server 2005 that is running on a server without Internet access" <Deb> Do we require third party filter dlls to be signed before they can be consumed from within the search services in SQL? <John> Yes, the IFilters are required to be signed for SQL Server 2005, but not for SQL Server 2000. <Deb> If we have multiple search products installed on a single machine, is there a deterministic way for the filter dll to know which search service invoked the filter dll? I suppose we can always inquire for the name of the calling process but since the name of the search services keeps on changing, a more elegant solution would be ideal. <John> Citeknet provides a very good "IFilter Explorer" that identifies multiple search products installed on a single machine and which Ifilter is related to each search product. John can be reached at SQL 2005 FTS blog.Anonymous
February 04, 2007
The comment has been removedAnonymous
February 04, 2007
In my VS2005 project I have created different configurations for the different debug modes I need to use to run the filter. The standard Debug configuration I use to debug the IPersistFile implementation with IFiltTst.Exe. This method is applicable probably to most IFilter developers. I use this on my C++ project, it may also work fine on other types. How to set this debugging: In your Project properties window, open ‘configuration properties’, ‘debugging’; and enter the following settings: Debugger to launch: Local Windows Debugger Command: C:<your path here>IFiltTst.Exe Command arguments: /i "D:<your project test folder><good file>.<extention to filter>" /v 3 /t 5 /l /d Working directory: /i "D:<your project test folder> Attach: No Leave all other settings as default. Oh, you need to allow Edit & Continue if you like: In your Project properties window, open ‘configuration properties’, ‘C/C++’, ‘General’. At ‘Debug Information Format’, select <i> Program Database for Edit & Continue (/Zl). Remember to not use this setting for your final production build. Now, just set a breakpoint and press the Run button. You can now step trough your code and check your variables. I find this very helpful in my IFilter R&D and development.Anonymous
February 05, 2007
With the public release of Vista a week back, soon developers will be wondering how to write and debugAnonymous
February 05, 2007
Great post Marco! Additionally, please check out the the following posts:
Anonymous
February 05, 2007
Excellent info here Marco! Your IFilter skeleton should provide a good reference for anyone writing this from scratch. Here's a little elaboration of the LOad method while using IPersistStream: SCODE STDMETHODCALLTYPE CMyFilter::Load(IStream* ps) { m_pISream = ps; ps->Addref(); m_pIStream.Set(ps); return S_OK; }Anonymous
February 22, 2007
The comment has been removedAnonymous
February 23, 2007
The comment has been removedAnonymous
February 25, 2007
fyi, i am writing an ifilter for an XML-format file type that contains other common office file types , text, doc, pdf and so on. to "subfilter" those i load the filters for corresponding files, and attempt to do it over ipersiststream first. often, the "subfilter" doesnt support that, and i will fall back to writing the contained document into a tempfile and load subfilter on that temp file. what other alternatives do i have ? btw, i have everything working beautifully under ifilttst, filtdump and so on, but .NET clients absolutely refuse to load my filter, apparently because of threading models mismatch or something .. from what i gather the client insists on getting the imarshal interface or something. i havent quite figured that out yet. I have no problems writing threadsafe code per se, but debugging apartment/com threading issues is a b*tch It would be useful if there was an ATL object template with appropriate threading model stuff specified (i.e. do i use CComMultiThreadModel as the root object, do i set aggregation and so on ? )Anonymous
February 25, 2007
Kert, Did you check with ProcMon, or Process Monitor from the sysinternals tools for anything that gets denied? Just when I started to do so I noticed testing and operational use is done within different security context. I noticed my filter starts loading and then breaks just when trying to write the temp file. If your problem is different from mine; it may also apply to my situation; so I am curious what solution you will find. For temp file workarounds, I found some articles on solutions like file-in-memory or ram-drive like solutions. Also something like making a named stream that you can access filebased trough an url. For all of these I did not find nice sample code to start trying this, but this may get you closer to your solution. Please keep us posted on your findings?Anonymous
February 26, 2007
The comment has been removedAnonymous
February 27, 2007
The comment has been removedAnonymous
March 02, 2007
Hi all, Our development of the DWG IFilter has had some good progress. We will have a first round of beta testing with external parties next week- and then another round for everyone interested. I hope to release the product around start of April. Would you like to join beta testing, drop us an email at ifilter@cadcompany.nlAnonymous
March 05, 2007
The comment has been removedAnonymous
March 05, 2007
Deb, I am happy to hear there is a demand for the new dwg IFilter. I would be happy to have some more consultants join our first beta round. I am preparing this first beta today and will be sending out the info today or tomorrow :).Anonymous
March 08, 2007
The comment has been removedAnonymous
March 08, 2007
The comment has been removedAnonymous
March 15, 2007
In my DWG IFilter Configurator tool I'd like to add some checkboxes to allow for easy registration in target indexers. It seems we can register for MOSS 2007, WSS 3.0, SQL 2005 without having the self registration entries in the standard Indexing Service 3.0 style. This allows me to avoid Indexing Service to pick up the IFilter; avoiding a performance hit on your system. Also this allows me to avoid my CiDaemon issue. I'd like to add a similar registration with WDS 3.0 for the same reasons. As I also am a simple user sometimes, I currently have both the standard 2000/2003/XP Indexing Service and WDS 3.0 active on my XP system, both trying to index my files :) So, I assume more people will find themselves in this situation. I'd like my installer to be able to target only WDS 3.0. Now, what I found on this: The WDS 3.0 registration instruction at http://msdn2.microsoft.com/en-gb/library/bb266533.aspx has overlapping registry entries with the Indexing Service part for WDS 2.x (at http://msdn2.microsoft.com/en-gb/library/aa965717.aspx#filters_to_register). So, if I use this to register for WDS 3.0, I believe I also register for Indexing Service - which I try to avoid. How do I work around this; is there a dedicated WDS 3.0 regisrtation as with MOSS, WSS, SQL2005?Anonymous
March 16, 2007
Just a FYI, my custom IFilter is now up on CodePlex under http://www.codeplex.com/wssDigiDoc its a full Sharepoint customization project, one part of being the IFilter. I never got it working under that .NET test app though, it works charmingly with Desktop Search, WSS and MOSS search. Maybe in a future releases, meanwhile, patches are welcome :)Anonymous
March 18, 2007
bytheway, i found a small useful bit of new info on recommended IFilter implemenation over at http://addins.msn.com/devguide.aspx#WritinganIFilter specifically, it lists the useful PROPIDs to support.Anonymous
March 24, 2007
This might be a far cry - but do you have the XD/NX bit enabled in an Intel/AMD CPU? It might be an interesting experiment to disable them and run the same tests.The XD/NX check can be disabled from the BIOS. More Info: http://www.intel.com/business/bss/infrastructure/security/xdbit.htmAnonymous
March 24, 2007
I will try to find the XD/NX setting, it is a dual core laptop; I will report back on this.Anonymous
April 17, 2007
Hi , I have some problem working with iFilters . i wrote a small app to load a iFilter and check the return value . Code -- void *myIFilter = NULL ; IUnknown *myIUnknown = NULL ; switch(LoadIFilter(L"c:\windows\system32\offfilt.dll" , myIUnknown , &myIFilter )) { case S_OK : printf("1"); break; case E_ACCESSDENIED : printf("2"); break; case E_HANDLE: printf("3"); break; case E_INVALIDARG : printf("4"); break; case E_OUTOFMEMORY: printf("5"); break; case E_FAIL: printf("6"); break; default : printf(" UNknow ERROR "); break; } The problem with the above code is .. it always hit "default" ( i am able to compile and generate a exe for the above app - i just tried declaring - IUnknown myiUnknown ,and IFilter myiFilter; but this gives me lot of compilation errors ) . I am not able to figure out the problem with the above code .. can some one please help me out on this . Thanks, CharanAnonymous
April 17, 2007
What is the specific error code returned by the IFilter?Anonymous
April 17, 2007
Hi Charan, I noticed in the LoadIFilter call you use the filename of the office IFilter dll. Instead, you could try to provide the filename of the content you'd like to filter. Check out the loadIFilter example in the DefaultParser class at http://www.dotlucene.net/documentation/MicrosoftOfficeDocumentsP.html - does this help you? The article also shows links to other code samples. Personally I have no experience with calling the IFilter routines like this. As I am building an IFilter I do like to see cool code samples of how an IFilter can be used.Anonymous
April 17, 2007
I have included all the error codes that are returned by LoadIFilter method in the switch case .. But still the code hits the default case . any way , i checked the error code value turns out that it is - 0x800401f0 i checked the value in winError.h and only the following Macros has the above value tied to them . #define CO_E_NOTINITIALIZED HRESULT_TYPEDEF(0x800401F0L) #define CO_E_FIRST 0x800401F0L Thanks CharanAnonymous
April 17, 2007
never mind , Figured out myself ... just searched for "CO_E_NOTINITIALIZED" error and came to know that we need to initialise the COM using CoInitialize function before using any COM functions .
Charan
Anonymous
April 18, 2007
The comment has been removedAnonymous
April 18, 2007
Hi , I have some problem extracting text from a Power point File (.ppt) The PPT file has a "Organization Chart" and each node in the chart has some label . i dont find that label in the Text i extrated from the PPT file through iFilters . I tried to do a "Find" with that label in the Power Point and i am able to see the hit of that label in the file . (even the "WORD ART" text is not returned by the iFilter , but this is fine since "Find" in POWER POINT doesnt search for such text ) i think this a limitation of iFilter ( for power point ) to return that kind of text . Am i right on this ?? if yes , then where can i find a detailed note on the limitations of office iFilter ( offfilt.dll ) Thanks, CharanAnonymous
April 18, 2007
Hi Marco van Schagen , I didnt have much knowledge on using iFilter Till today evening :-) ... I thought that i need to specify the Office iFilter DLL to the LoadIFilter routine !! ( which is wrong .. of course ) so thats the reason why u found that in my earlier code ... after some research i figured out how things work .. and finally able to build a small working App that uses iFilter routines to extract text from some standard file formats ( .doc .pdf etc ) . I see that u are interested to see code sample which uses the iFilter routines .... the code i have written may help you ... My appologies if i am not supposed to post big message :) void *myIFilter = NULL ; IUnknown *myIUnknown = NULL ; HRESULT rc = 0; if(!(CoInitialize(NULL) == S_OK )) { return false ; } if((LoadIFilter(L"C:\Temp\ab.doc" , myIUnknown , &myIFilter ) == S_OK ) { If(myIFilter) { ULONG initFlags = IFILTER_INIT_APPLY_INDEX_ATTRIBUTES ; ULONG initOutflags =0; if(((IFilter *)myIFilter)->Init(initFlags , 0 , NULL , &initOutflags) == S_OK) { STAT_CHUNK mySC ; memset(&mySC , 0 , sizeof(mySC) ); while(1) { long rcGC = ((IFilter *)myIFilter)->GetChunk(&mySC) ; if( rcGC == S_OK ) { if(mySC.flags == CHUNK_TEXT ) { WCHAR myBuffer[512]; char buf[512]; int i=0; ULONG myBufLen = 0; FILE *fp = NULL; fp = fopen("c:\temp\output.txt" , "a"); while(1) { myBufLen = sizeof(myBuffer)/sizeof(myBuffer[0]); long rc = ((IFilter *)myIFilter)->GetText(&myBufLen , myBuffer); //For English Text - we dont need wchar for(i=0 ; i<myBufLen ; i++) buf[i] = (char)myBuffer[i]; switch(rc) { case S_OK : fwrite( buf, sizeof(buf[0]) , myBufLen , fp); break; case FILTER_E_NO_TEXT: case FILTER_E_NO_MORE_TEXT: case FILTER_S_LAST_TEXT: printf(" END !"); rc = -1; break; default : printf("Unknown ERROR "); rc = -1; } if(rc == -1) break; } fclose(fp); } else { printf("we have a Value-type property here "); } } else if(rcGC == FILTER_E_END_OF_CHUNKS ) { printf("End of Chunks "); break; } else if( rcGC == FILTER_E_EMBEDDING_UNAVAILABLE) { printf("EMbedding problem"); break; } else if( rcGC == FILTER_E_LINK_UNAVAILABLE ) {printf("Link Problem"); break; } else if( rcGC == FILTER_E_ACCESS ) {printf("Access Problem"); break; } else { printf("Unknown Probblem "); break; } }// while(1) } }Anonymous
April 18, 2007
NOtE : i dont handle the value-property in the above code .... it takes lot of time implement the same . I stil need to figure out how to handle the value-property . -- CharanAnonymous
April 18, 2007
Marco, its great news that you're able to drill down to the bottom of this. Regarding " In my case this is the total text as emitted in the first IFilter chunk; where I have only one chunk emitting text. I am not sure if using more GetChunk chunks would allow for more text. " Actually if you're having problems filtering large files, using more chunks is an excellent idea. Also a heads up, the next version of windows will put a security cookie to prevent use of temp files. You might want to start a dialogue with the AutoCad folks to provide stream based loading functionality in their library used by the filter.Anonymous
April 18, 2007
Charan, There is no official document detailing the limitations of offilt. We look into each issue based on its merit and resolve/fix it acoordingly. Can you please send me the repro file with a crisp description of the problem? Thanks, Deb.Anonymous
April 18, 2007
Deb , i will send the file on which i found the problem with other details . may i have the mail ID of the person to whom i should send the details ?
Charan
Anonymous
April 18, 2007
Charan, Ideally you'd send the file to the Microsoft Product Support contact for your company. However, if you do not have a MS PSS contact, please send them to debh@microsoft.com and briancoy@microsoft.com cheers, Deb.Anonymous
April 19, 2007
My apologies, my previous post was based on a bad piece of code. I have made many changes, and the errors disappeared. Good thing. Also the indexing result dissapeared. Not a good thing. I was thinking, having one buffer between GetText calls may result the indexer to read the last text from buffer over and over again. I'll need to check this. My Microsoft contact tells me I am having an issue with stack space. I am still working on this one.Anonymous
June 27, 2007
I'm pleased to announce that after a lot of trails and tribulations, the DWG IFilter is finally ready.Anonymous
July 18, 2007
Hi there, Thanks for all the great tips! For windows desktop search, do you know how the text in the preview pane is derived? For me it looks like the contents of the first call to getText() is shown in the preview.Anonymous
September 21, 2007
I have one doubt with IFilter ... is it possible to ask iFilter to give the textual contents in some defined order ? meaning that, for eg , if you take ppt file .. the client should able to distinguish between the text retrieved from a header and from a page. Are there any way to specify these things to iFilter so that i know which section of the document i am dealing with and handle those block of text accordingly ?Anonymous
May 13, 2008
Hi Charan Its a great new that u figured out this tricky thing - Although its a long time but still congrats :)Anonymous
June 16, 2008
Anyone knows if exists an I filter for ISO files? Thanks a lot, gaAnonymous
December 19, 2008
I have a bit of an IFilter mystery to solve. I'm writing a custom IFilter for a file format that might contain some embedded MIME stuff... How can I get a handle to the MIME IFilter so that I can pass the chunk requests through? It seems everything is setup to easily deal with OLE structured storage docs, but not MIME. If I pass the stream off with BindIFilterFromStream, I just get a message stating that the OLESTREAM format is incorrect. The data is in a stream, not a file... Thanks.