MOSS_Export phase FIM Full Sync Eats up all Memory
Keywords for search:- SharePoint 2010; MOSS_Export; Full Sync; Out of Memory; FIM;
Hello All,
I have been struggling with the FULL Sync for USer Profile Application for last 2-3 weeks and all the time the Memory on the Server was peaking out, we added RAM to the Servers thinking we were running low on the RAM for App Server we went from 8GB to 32GB RAM the max on the Windows Server 2008 R2 Standard box. I tried to find all possible causes and was really not able to find out things yes there were few things on the planning side which also went wrong and hence I decided to write this blog. Let me get to the core things:-
What you should advance plan:-
1. Make sure All the Active Directory connections and EXCLUSION Filters for those are correct. PLEASE EXECUTE ADSI QUERIES to find the exact number of accounts returned. THIS IS VERY IMPORTANT planning phase. Work with you AD team to get this done, make sure the Filters to be applied are with correct AND or OR condition as the results change vastly.
2. Make sure Hardware planning is done properly and you have planned for enough resources for the UPA Sync. I would recommend minimum of 16GB of RAM for UPA Sync server. Yes you can run the UPA service on more than one box and UPA Sync on only one box.
3. Make sure the DOP (Degree of Parallelism) on SQL Server for SharePoint value is set to 1 this is important guidance from MICROSOFT and should be followed.
4. PLEASE PLEASE PLEASE while planning for UPA start in advance way in advance and start running POCs I would say atleast a month before your PROD go live, the FULL Sync is improved in SP 2010 and it takes a little longer than in SharePoint 2007 and you might hit road blocks so please plan in advance.
5. PLEASE rely 100% on ULS logs... yes it contains everything and it is your torch light for everything... it has helped me a lot.
Now coming to my Memory Leak issue, where the Memory was spiking up to the max available on the server (32GB) in my case, all the other phases in the Full Sync job go well i.e. DS_FullImport, DS_FullSync, MOSS_FullSync all go well it is only the last stage which is MOSS_Export which is putting the Profiles in SharePoint 2010 is the one getting affected. So after spending almost 2-3 weeks on this I couldn't find the reason, yes I worked with the Microsoft Premier support and they took the Memory dump during the Memory spike up and told me that somehow the MMS store was getting pulled into the Memory and the cache was building up.
So finally l opened the ULS logs (LIVE) when the MOSS_Export was going on and found lots of these exceptions:
Exception while updating properties for ''northamerica\kanyadas': Property: SPS-ProxyAddresses, Exception System.ArgumentException: The value 'X400:c=US;a= ;p=SCHP;o=Hub-Americas;s=srvdasilcla;g=CustomNamed;' is invalid. It probably contains invalid characters or is too long. Parameter name: termLabel
at Microsoft.SharePoint.Taxonomy.Internal.CommonValidator.ValidateName(String name, String parameterName)
at Microsoft.SharePoint.Taxonomy.TermSet.GetTerms(String termLabel, Boolean trimUnavailable)
at Microsoft.Office.Server.UserProfiles.UserProfileGlobal.ValidatedTaxonomicValue(CoreProperty prop, Object value)
at Microsoft.Office.Server.UserProfiles.UserProfileGlobal.ValidatedSingleValue(Object value, ProfileSubtypeProperty prop, PropertyDataType propDataType, UserFormat userFormat, UserProfileApplicationProxy userProfileApplicationProxy, Guid partitionID, SiteContext si)
at Microsoft.Office.Server.UserProfiles.UserProfile.BulkPropertiesUpdate(Int64 importExportId, Hashtable properties, String strAccountName).
Note the highlighting (yellow) yes the SPS-ProxyAddresses is a multi-value string and we had mapped it to ProxyAddresses field (As the way it was in 2010) and because it is multivalue string it was trying to validate things with TermStore (Enterprise Keywords) before adding and that was causing the Memory cache to be build (we had 5 AD connections so 5 mappings for each Profile Property). One of my colleagues helped me understand the Importance of this field (SPS-ProxyAddreses) w.r.t. SharePoint 2010 and following was it:- SharePoint only imports proxyAddresses to extract the SIP address and not to actually use the proxyAddresses. But in our case we had already mapped our SIP Address feild to AD field EMAIL ADDRESS so we really didnt rely on it much. So we went ahead and removed the SPS-ProxyAddresses mapping that we had done to ProxyAddresses Feild and we re-ran the Full Sync and viola after 3 days of running into Full Sync MOSS Export stage started and THANK GOD the MEMORY SPIKES WERE GONE :-) :-).
My only BIG Adivce to you all is please please rely on the ULS Logs to show you the path and also please plan ahead for UPA Sync way way way ahead I would say. Check your mappings try to compare it with the out of the box UPA mappings and make sure all are in inline. This will help you save a lot of time and lot of customers would appreciate that. In the end please don't forget to read Spencer's blogs about UPA they are concise and very helpful. Also I would like to thank Spencer Harbar as all along he helped me a lot with some good advice so thank you buddy!!!.