Share via

URL Filtering is Here!

  • Article

Quick Introduction

Hi Everyone,

We are proud to introduce as part of Forefront TMG Beta3 one of the key elements in the Secure Web Gateway (SWG) role of Forefront TMG – URL Filtering.

URL Filtering allows controlling end-user access to Web sites, protecting the organization by denying access to known malicious sites and to sites displaying inappropriate or pornographic materials, based on predefined URL categories.

The typical use case for this feature includes:

· Enhancing your security.

· Lowering liability risks.

· Improving the productivity of your organization.

· Saving network bandwidth.

The URL Filtering administration experience is pretty straightforward. All you need to do after enabling the feature is add one or more of the predefined URL categories into Forefront TMG policy (you can find some UI snapshots further below). Once this is done, end-users browsing to a Web site included in one of those categories will be blocked and presented with a relevant notification page, which you can customize.

Additional value can be obtained from URL Filtering related reports and log entries. Have you ever wanted to understand how Web usage in your organization is distributed? And how about identifying those users who consistently violate your Web usage policy? You can do those easily now by looking at the built-in URL filtering reports.

Finally, URL Filtering categories can also be leveraged to exclude sites from being inspected by the HTTPS Traffic inspection and the Malware Inspection features. For instance, you may wish to exclude financial sites from HTTPS inspection, due to privacy considerations.

Before going into further details, note that the feature is still in Beta, so we do expect significant improvements in coverage and accuracy by the final TMG release.

URL categorization data, where does it come from?

TMG features over 80 URL categories ranging from security-oriented selections, like Phishing, Malicious and Anonymizers, through productivity-oriented categories such as Games, or Instant Messaging, and ending with liability-oriented categories like Criminal Activities and Pornography. Categories are also grouped into a higher-level hierarchy which we call Category Sets. The latter can also be used in TMG policy to simplify configuration.

As some of you may have noticed, at the RSA 2009 Conference Microsoft announced its new reputation services and its intention to provide these capabilities for our security products and solutions.  Microsoft also announced several key partnerships in the URL filtering space that will be used to support these reputation services. Forefront TMG will be the first system at Microsoft to leverage and utilize Microsoft Reputation Service (MRS).

MRS is a cloud-based object categorization system hosted in Microsoft data centers and designed to provide comprehensive reputation content to enable core trust scenarios across Microsoft solutions. In the case of Forefront TMG, in order to find out the category of a URL, TMG issues an online query to MRS. MRS maintains a database with tens of millions of unique URLs and their respective categories.

Does this mean every end-user request is sent out to the cloud? No it doesn’t. To improve bandwidth utilization and performance, we have implemented a local cache (residing on a TMG server), that stores the recently queried URLs and their respective categories. Cache entries are subject to a time-to-live value, allowing refreshing the entry periodically. This local cache is expected to serve the overwhelming majority of user requests.  The cache is persistent so it doesn't need to be refreshed after each reboot. TMG will query MRS only when a request cannot be served from the local cache.

But that's only the tip of the iceberg. Read on to find out why we think we are building something special with TMG and MRS together.

What is so special about Microsoft Reputation Service (MRS)?

The MRS team wanted to confront an inherent problem with traditional URL Filtering solutions: the problem domain is simply too large for any single vendor to provide a complete solution on its own. As a result, there are multiple vendors out there, each one specializing in a specific area of the solution.

Some vendors specialize in identifying malicious sites and spam URLs; others are rich with productivity related categories. Some specialize in covering the Internet's “long tail”; others are great with quick classification of previously unknown sites. Some use human-based classification where others use machine-based techniques. Some are great with Web2.0 style URLs… OK, I'll stop here as you get the idea by now. Even those vendors who employ several classification techniques and cover multiple categories can't deal with the huge and ever-expanding challenges of today's Web.

MRS team's idea was simple; let's leverage complementary capabilities of different vendors/sources to create a unified database that is best suited to deal with the challenges described above. And so, they have implemented a scalable architecture that allows incorporating multiple streams of data into a merged database. This way – each vendor/source brings its unique strengths to the table into a common solution.

MRS already integrates several data sources and others will be on-boarded in the following months. Some of these data sources are Microsoft internal, and others are the result of collaboration with 3rd party partners. One such agreement, announced during RSA, is an agreement with Marshal8e6.  Other agreements have not been disclosed yet. Expect some surprises...

But the real beauty is that being a Web service, and given its unique architecture, MRS can easily incorporate new DBs completely transparently to the customers. We expect the MRS unified database to expand over time and become the recognized industry leader. TMG customers will benefit naturally from this ongoing upgrade, through our Web security subscription services.

Other interesting aspects – security ,privacy, licensing

Security – Both Forefront TMG URL Filtering and MRS were designed with security in mind, following Microsoft's Security Development Lifecycle (SDL) strict standards and guidelines. Both are resilient to a variety of attacks, and the communication between the two is encrypted.

Privacy – this is a known concern when discussing cloud based services, and therefore the privacy of our customers' data is paramount. We are issuing detailed privacy statements along with the Beta 3 release to provide clarity and transparency on our privacy policies. Make sure to read those.

Licensing – URL Filtering is subscription based, and is part of the Forefront TMG Web Security Service license (together with the Malware Inspection updates).

The small (but important) things

As this is a high-level overview of the feature, we will not dive into all the small details that make for a complete, rich user experience. We will cover some of those in subsequent posts, as we go along. But here are few examples for flexibility you are likely to need/want when working with URL Filtering:

- You can locally override a URL category

- You can query for a URL's category in the TMG UI

- You can customize the block page displayed to end-users, introducing your own HTML tags into the text area.

- You can leverage URL Filtering for ad blocking

- You can use the build-in TMG scripting capabilities to allow non-TMG administrators to locally override a URL (enabling advanced help-desk scenarios)

- You can use URL Filtering related reports to figure out how your organization uses the Internet (which are the top browsed categories for instance)

- You can report classifications issues to Microsoft (this one is not available in Beta3)

A sneak peek at the UI

TMG Web Access Wizard allows you to easily introduce URL categories into your policy:

clip_image002[6]

This is how the policy may look like after completing the Web Access Wizard (viewed from the Web Access Protection node). Note that URL Categories are standard TMG network objects, so you can use the toolbox on the right to drag-drop additional categories into an existing rule, or to create new rules.

clip_image004[6]

You can query for a URL's category (available as a task in the Web Access Protection node)

clip_image006[6]

You can locally override a URL's category (available as a task in the Web Access Protection node)

clip_image008[6]

You can customize the block page presented to end users, introducing your own HTML tags (this is a per-rule setting available from the ‘Action’ tab of the rule’s properties)

clip_image010[6]

Wrap-up

Thanks for reading all the way! I hope you have found this post helpful. We are certainly interested in your comments and thoughts!! Feel free to post them below.

Author:

Dotan Elharrar, Program Manager, Forefront TMG

 

Reviewers

Yair Geva, Senior Development Lead

Vladimir Holostov, Senior Program Manager Lead

Nathan Bigman, Content Publishing Manager

Bill Jensen, Senior Product Manager

Yigal Edery, Principal Group Program Manager

David B. Cross,Product Unit Manager

Comments

  • Anonymous
    January 01, 2003
    I have seen this question a few times on both  internal and external lists so I figured I would

  • Anonymous
    January 01, 2003
    Sta procedendo il lavoro del team di sviluppo sulla nuova versione di ISA Server, che prenderà il nome

  • Anonymous
    January 01, 2003
    Introducing such a topic you'd like to congratulate you've let us know. Have good work

  • Anonymous
    January 01, 2003
    This and similar information-sharing my vocabulary expanded again. Thank you for your writing ...

  • Anonymous
    January 01, 2003
    Hi! I did a test with URL Filtering and use both proxy and secureNAT clients and it's working perfect! TMG make the name resolution of the secureNAT client request and categories the URL. But why can't TMG show the URL in the log on the same way as for the proxy client? The TMG makes the name resolution after all. Best regards, Anders

  • Anonymous
    January 01, 2003
    We will disclose more details as TMG approaches its release to market.   Regards, Dotan Elharrar

  • Anonymous
    January 01, 2003
    The comment has been removed

  • Anonymous
    January 01, 2003
    Yes. The next version of Windown Essential Business Server will include TMG URL Filtering. Regards, Dotan Elharrar

  • Anonymous
    January 01, 2003
    Will this be included in Essential Business Server ??

  • Anonymous
    January 01, 2003
    The comment has been removed

  • Anonymous
    January 01, 2003
    The comment has been removed

  • Anonymous
    January 01, 2003
    The comment has been removed

  • Anonymous
    January 01, 2003
    Hi! do you have any news about licensing for malware and url filtering? (cost per month, per user or per server ...)

  • Anonymous
    January 01, 2003
    This is very good news was well informed that the followers of the issue I am. Thank you ...

  • Anonymous
    January 01, 2003
    Thanks a ton for all the hard work

  • Anonymous
    June 09, 2010
    Hi, Can someone please explain why there are TMG categories missing from a standrad installation. So basically, in the MRS error and reporting site it has more categories such as "History" for example but in TMG not! Thanks!

  • Anonymous
    May 20, 2011
    The comment has been removed

  • Anonymous
    September 16, 2011
    when a site is set to over-ride into the unknown category, it still reports the current MRS categorization?

  • Anonymous
    September 16, 2011
    could we get an option to delay categories x amount of seconds 0-120 before display?

  • Anonymous
    September 16, 2011
    why are online reservations at www.thecal.com blocked? why is the thecal.com site set as gambling?

  • Anonymous
    September 16, 2011
    why can't we create our own custom URL categories?

  • Anonymous
    September 16, 2011
    how do we troubleshoot foxnews.com, new category is not blocked, yet the site looks horrible

  • Anonymous
    September 16, 2011
    how do we enable safe search in youtube videos and images?

  • Anonymous
    January 07, 2012
    It is a really nice instruction. You have a cool blog too.

  • Anonymous
    July 06, 2012
    Hello All! Please help me. How can export only my url category overrides? Please asnwer to my email (eduard.marchenko@miroplast.com)

  • Anonymous
    March 25, 2013
    The comment has been removed

  • Anonymous
    April 13, 2014
    how to block psiphon proxy through TMG 2010