TMG Safe search feature may not work as expected sometimes
In this post, I would like to talk about a TMG safe search problem which I dealt with recently. The problem was that the TMG safesearch feature wasn’t working properly even though the all the requirements were met. We decided to collect dynamic logs while reproducing the problem (TMG data packager + client side logs). Then I started analyzing the logs:
- The Safesearch related rule was already enabled:
SafeSearch Enabled
Action Allow
Applies Always true
Default false
Description The SafeSearch rule allows access to the "Search Engines" URL category, and filters out adult content from search results of supported search engines.
ID
Logging Enabled
Source Port Limits Disabled
Type Access Rule
From
Network Internal, Included, Array scope
Applies to all content
Protocols Specified Protocols
HTTP, Included, Array scope
HTTPS, Included, Array scope
UrlCategory Search Engines, Included, Enterprise Scope
Users All Users, Included, Array scope
- It already included "All users"
- TMG server version was the latest (SP2 + rollup2)
- Safesearch XML file was in place with the correct settings:
<Configuration>
<provider domainPattern=".google." safeSearchSuffix="&safe=strict">
<searchQuery pattern="/search?"/>
<searchQuery pattern="/images?"/>
</provider>
<provider domainPattern=".yahoo.com" safeSearchSuffix="&vm=r">
<searchQuery pattern="/search?"/>
<searchQuery pattern="/search;"/>
<searchQuery pattern="/search/images?"/>
<searchQuery pattern="/search/images;"/>
<searchQuery pattern="/search/video?"/>
<searchQuery pattern="/search/video;"/>
</provider>
<provider domainPattern="www.bing.com" safeSearchSuffix="&adlt=strict">
<searchQuery pattern="/search?"/>
</provider>
</Configuration>
- It's the same as the one provided in ISA blog:
<provider domainPattern="www.bing.com" safeSearchSuffix="&adlt=strict" >
<searchQuery pattern="/search?" />
</provider>
=> Even though all configuration was correct, TMG server was still not forwarding the correct search URL towards Bing (which would help Bing filter search results)
- We see the search requests sent by the client on the client side network trace. Some examples:
2857 12:46:22.524738 50.167033 0.000000 10.96.96.6 10.110.0.121 HTTP GET https://www.bing.com/search?q=xxx&qs=n&form=QBLH&pq=xxx&sc=0-0&sp=-1&sk= HTTP/1.1
3642 12:46:44.659253 72.301548 0.054843 10.96.96.6 10.110.0.121 HTTP GET https://www.bing.com/images/search?q=xxx&FORM=HDRSC2 HTTP/1.1
3832 12:46:48.339288 75.981583 3.680035 10.96.96.6 10.110.0.121 HTTP GET https://www.bing.com/search?q=xxx&FORM=HDRSC1 HTTP/1.1
=> Relevant session in TMG ETL Logs: (collected as a result of TMG data packager)
...
[0]197c.1cc8 10/03/2012-10:46:48.780 [1a2c7b91 1a2c7d03] [WP proxyext...] Info: WPPISAPUBLIC:Context property:WPPISAPUBLIC:HTTP URL = https://www.bing.com/search?q=xxx&FORM=HDRSC1
...
=> Safesearch filter decides that there's no need for safesearch analysis:
[0]197c.1cc8 10/03/2012-10:46:48.781 [1a2c7b91 1a2c7d03] [HTTPFLT...] Entering CHttpFilterSafeSearchEnforcer::IsSafeSearchAnalysisRequired
...
[0]197c.1cc8 10/03/2012-10:46:48.781 [1a2c7b91 1a2c7d03] [HTTPFLT...] Info:SafeSearch analysis is not required, exiting
=> Network trace collected on the external interface of TMG server:
- We can see that the request is being sent as is without any modifications towards the Bing server:
256368 06:46:49.2700830 875.6370830 10.110.7.54 212.252.126.59 HTTP HTTP:Request, GET /search, Query:q=xxx&FORM=HDRSC1 {HTTP:3723, TCP:3721, IPv4:3651}
256370 06:46:49.3080860 875.6750860 212.252.126.59 10.110.7.54 TCP TCP:Flags=...A...., SrcPort=HTTP(80), DstPort=58199, PayloadLen=0, Seq=3768470858, Ack=3665294676, Win=25515 (scale factor 0x0) = 25515 {TCP:3721, IPv4:3651}
256372 06:46:49.3600890 875.7270890 212.252.126.59 10.110.7.54 HTTP HTTP:Response, HTTP/1.1, Status: Ok, URL: /search {HTTP:3723, TCP:3721, IPv4:3651}
https://www.bing.com/search?q=sex&FORM=HDRSC1
=> Normally it should have been sent something like below so that adult content wouldn't have been returned by Bing:
https://www.bing.com/search?q=sex&FORM=HDRSC1&adlt=strict
At this point everything seemed to be correctly configured but for some reason the safesearch filter wasn’t kicking in. After some more research and with the help of an escalation engineer from TMG team, we found out that in order for safesearch filter to kick in, the request itself also should be matching the Safesearch rule which is automatically created when Safesearch is enabled.
In my customer scenario, problem was that the access request sent by the user was hitting an enterprise level access rule and hence the safesearch rule wasn’t hit and the filter wasn’t activated as a result. After my customer re-arranged the enterprise level rules so that search engine related requests don’t hit any enterprise level access rule but hits the array level “Safesearch” rule created automatically, the problem was resolved.
Hope this helps
Thanks,
Murat