通过

TMG URL Filtering category precedence

  • 项目

Introduction

Forefront TMG 2010 introduced URL filtering, which enables administrators to create rules that allow or block access to Web sites based on their categorization in the URL filtering database. When a request to access a Web site is received, Forefront TMG queries the remotely hosted Microsoft Reputation Service (MRS) to determine the categorization of the Web site. If the Web site has been categorized as a blocked URL category or category set, Forefront TMG blocks the request.

If a user requests access to a Web site and discovers that access to the Web site is blocked, he receives a denial notification that includes the URL category which resulted in the denied request. In addition, sites can be excluded from HTTPS and malware inspection based on their category.

The Forefront TMG URL filtering mechanism uses URL categorization provided by the MRS Web service. Some URLs have multiple categories, for instance http://finance.yahoo.com is categorized as
Financial, Online Trading and News. Forefront TMG’s policy and its rule engine are based on a single category per URL. This means that in case the MRS responds with multiple categories per URL, Forefront TMG will need to choose one of those categories as the “most relevant” URL category. In order to do that, Forefront TMG uses a pre-defined category precedence list.

Category precedence list

Multiple categories for a single requested URL are sent back by the MRS web service with no concept of prioritization or order. However, Forefront TMG uses single-URL categorization in its policy. Therefore, we need a mechanism to choose “most relevant” category from a set of URL categories provided by MRS. For that task Forefront TMG has a category precedence list, where categories are ordered by significance. The rule of thumb is that more malicious, harmful and non-productive categories have higher precedence.
The list is pre-defined and can’t be changed by administrators. The list for Forefront TMG SP1 is below.   

No.

Category

1

    "Malicious"

2

    "Pornography"

3

    "Botnet"

4

    "Phishing"

5

    "Criminal Activities"

6

    "Hate/Discrimination"

7

    "Anonymizers"

8

    "Spyware/Adware"

9

    "Illegal Drugs"

10

    "Violence"

11

    "Obscene/Tasteless"

12

    "Gambling"

13

    "Spam URLs"

14

    "Dubious"

15

    "Hacking/Computer Crime"

16

    "School Cheating Information"

17

    "P2P/File Sharing"

18

    "Personal Network Storage"

19

    "Remote Access"

20

    "Shareware/Freeware"

21

    "Nudity"

22

    "Mature Content"

23

    "Weapons"

24

    "Alcohol"

25

    "Tobacco"

26

    "Search Engines"

27

    "Financial"

28

    "Online Trading/Brokerage"

29

    "Government/Military"

30

    "Employment"

31

    "Online Communities"

32

    "Digital Postcards"

33

    "Chat"

34

    "Portal Sites"

35

    "Usenet News"

36

    "Web E-mail"

37

    "Web Phone"

38

    "Web-based Productivity Applications"

39

    "Education/Reference"

40

    "Child Friendly Materials"

41

    "Public Information"

42

    "Technical Information"

43

    "Health"

44

    "Art/Culture/Heritage"

45

    "General Entertainment"

46

    "Games"

47

    "Humor/Comics"

48

    "Recreation/Hobbies"

49

    "Special Interests"

50

    "Restaurants/Dining"

51

    "Social Opinion"

52

    "Self Defense"

53

    "Travel"

54

    "Fashion/Beauty"

55

    "Motor Vehicles"

56

    "Shopping"

57

    "Real Estate"

58

    "Legal Services & Reference"

59

    "Non-Profit/Advocacy/NGO"

60

    "Politics/Opinion"

61

    "Religion/Ideology"

62

    "Edge Content Servers/Infrastructure"

63

    "Dating/Personals"

64

    "Sports"

65

    "Free Hosting"

66

    "Internet Services"

67

    "Web Ads"

68

    "Media Sharing"

69

    "Streaming Media"

70

    "Forum/Bulletin Boards"

71

    "News"

72

    "Blogs/Wiki"

73

    "General Business"

74

    "Parked Domain"

75

    "Unknown"

When Forefront TMG receives an HTTP request, it retrieves its URL category from MRS or from internal cache. If the URL has several categories, Forefront TMG applies category precedence rules according to the precedence list. The category with the highest precedence is used by the Forefront TMG rule engine, while all other categories are disregarded.

Let’s see an example. When a user browses to http://msdn.microsoft.com, MRS categorizes that URL as General Business and Technical Information,
as can be seen from the MRS portal at http://www.microsoft.com/security/portal/mrs/

clip_image001

Since “Technical Information” has higher precedence than “General Business”, TMG will use the “Technical Information” category for that URL. The “Technical Information” category will be applied for rules, will appear in log/reports and will be presented to users in denial pages.  It will also be matched to HTTPS inspection and malware protection exemptions categories.
We can use the Forefront TMG UI Category Query tool to validate that.

clip_image003

Summary

In this blog, I showed that although MRS provides several categories for each URL, Forefront TMG rules engine decisions are based on one category only.
This “most relevant” category has the highest precedence in the pre-defined precedence list. Administrators can verify which category was chosen by Forefront  TMG using the Log or Query Category UI.

 

Author: Igor Zarivach
Reviewers: Ori Yosefi, Roman Golubchyck

Comments

  • Anonymous
    August 18, 2010
    This is one of the few times the list of categories has been released.  Even the product documentation doesn't have the list of categories in it.  Is there an official place where the list of categories is definitively listed?  We used one of these-here blog-posts in the past to obtain the list so that our executive management can approve the blocking of categories, only to find that in FTMG 2010 SP1 some of the categories which were previously present, such as "Profanity" have gone AWOL.

  • Anonymous
    August 18, 2010
    Exist some documents that provide tips or best practices firewall and web access polices.

  • Anonymous
    January 26, 2011
    I would like to appreciate the work of blog author that the person provided us with an extremely excellent information regarding the topic. Ireally learned something from this blog and started to contribute my ideas via commenting on this blog. Keep it up

  • Anonymous
    January 27, 2011
    Great article however our server does not return the category in precedence order.  The example URL comes back as General Business on TMG server.  The same can be said for www.facebook.com which we want to block as Online Communites not the returned blogs/wiki.

  • Anonymous
    February 02, 2011
    The blog provides helpful information regarding the topic and it also gives a vast knowledge as well which helps us in our studies and in practical life.

  • Anonymous
    March 03, 2011
    Is there a list with all the URL's that are in a catagory??? thanks in advanced :)

  • Anonymous
    March 24, 2011
    Anyone know how often the DB of URLS is updated? For E.G Websense releases a DB update every 24 hours and theirs is extensive and acurate, if we switch to TMG can we expect the same. Thank you for the info in the above blogg.

  • Anonymous
    April 11, 2011
    It's always amazing reading or commenting on a blog from which we get a full knowledge. Same as here I have found some really interesting information which is simply a great boost to my knowledge.

  • Anonymous
    November 19, 2012
    The comment has been removed

  • Anonymous
    August 28, 2015
    Nice article. Great read! Will definitely share 
    Thank you!