Archive

Archive for the ‘URL filtering’ Category

Further details and guidance regarding discontinuation of TMG Web Protection Services

December 15th, 2015 No comments

As discussed in the following blog, the Forefront Threat Management Gateway (TMG) Web Protection Services will be discontinued on December 31st, 2015:-

http://blogs.technet.com/b/applicationproxyblog/archive/2015/11/02/important-reminder-for-forefront-threat-management-gateway-tmg-web-protection-services-customers.aspx

We wanted to provide some additional details on what this will affect and recommendations on actions you should be taking.

The services that will be affected by this are:-

– URL Categorization
– Malware Inspection

Importantly, the Microsoft Reputation Services that supports URL Filtering will be turned off on or shortly after the 31st December 2015.

To avoid service impacting issues due to these services no longer being available, or incorrect rule processing where rules rely on URL Categorization categories, we would strongly advise customers review and amend their TMG configurations as follows:-

Review and amend any rules based on URL Categorization categories in your TMG policy

Any Allow and Deny rules that currently use URL Categories or URL Category Sets must be changed to remove the usage of URL filtering categories.

Using URL Sets or Domain Name Sets may provide limited replacement functionality or you may also want to consider a 3rd party URL filtering plug-in or upstream proxy service to provide replacement URL filtering functionality.

Note – If you have rules that are using URL filtering to allow traffic – HTTP traffic can be totally blocked after the service shutdown. Equally, if you use URL Filtering to block access to certain categories then these may be allowed after the change. There is also a possibility that performance issues will be seen if URL Filtering is left enabled after the MRS service is taken offline.

Disable URL Filtering

After amending your TMG policy ensure you then disable URL Filtering. This can be done in the TMG Management Console in the Web Access Policy node by selecting URL Filtering and unchecking the “Enable URL Filtering” check-box. This is essential to avoid TMG trying to contact the MRS services after they go offline.

clip_image002

Malware Inspection may continue to work but would not receive updated signatures

We would recommend implementing an alternative Anti-Virus solution and to disable Malware Inspection once this is in place.

As noted in the previous blog, Forefront Threat Management Gateway 2010, remains under extended support until April 14, 2020.

For details on moving from TMG to our new web publishing solutions please visit this URL:

http://blogs.technet.com/b/applicationproxyblog/archive/2015/07/02/transitioning-to-application-proxy-from-uag-and-tmg.aspx

Some Frequently Asked Questions we’ve had regarding the change are:-

Q1. Is it possible to use the MRS Cache to continue to benefit from URL Filtering after 31st December 2015?

A1. No, the MRS cache is a temporary in-memory cache of the latest lookups intended to provide internal efficiency optimizations. It does not provide a full offline cache and cannot be used for this purpose. There is no mechanism to have an offline database.

Q2. Is it possible to extend our usage of Forefront Threat Management Gateway (TMG) Web Protection Services past 31st December 2015?

A2. No, this is not possible. These dates were announced in September 2012 in order to provide sufficient time for alternative solutions to be deployed.

For the original announcement of the Forefront product roadmap changes please refer to the following URL:

http://blogs.technet.com/b/server-cloud/archive/2012/09/12/important-changes-to-forefront-product-roadmaps.aspx

Categories: EMP, malware inspection, TMG, URL filtering, URLF Tags:

Further details and guidance regarding discontinuation of TMG Web Protection Services

December 15th, 2015 No comments

As discussed in the following blog, the Forefront Threat Management Gateway (TMG) Web Protection Services will be discontinued on December 31st, 2015:-

http://blogs.technet.com/b/applicationproxyblog/archive/2015/11/02/important-reminder-for-forefront-threat-management-gateway-tmg-web-protection-services-customers.aspx

We wanted to provide some additional details on what this will affect and recommendations on actions you should be taking.

The services that will be affected by this are:-

– URL Categorization
– Malware Inspection

Importantly, the Microsoft Reputation Services that supports URL Filtering will be turned off on or shortly after the 31st December 2015.

To avoid service impacting issues due to these services no longer being available, or incorrect rule processing where rules rely on URL Categorization categories, we would strongly advise customers review and amend their TMG configurations as follows:-

Review and amend any rules based on URL Categorization categories in your TMG policy

Any Allow and Deny rules that currently use URL Categories or URL Category Sets must be changed to remove the usage of URL filtering categories.

Using URL Sets or Domain Name Sets may provide limited replacement functionality or you may also want to consider a 3rd party URL filtering plug-in or upstream proxy service to provide replacement URL filtering functionality.

Note – If you have rules that are using URL filtering to allow traffic – HTTP traffic can be totally blocked after the service shutdown. Equally, if you use URL Filtering to block access to certain categories then these may be allowed after the change. There is also a possibility that performance issues will be seen if URL Filtering is left enabled after the MRS service is taken offline.

Disable URL Filtering

After amending your TMG policy ensure you then disable URL Filtering. This can be done in the TMG Management Console in the Web Access Policy node by selecting URL Filtering and unchecking the “Enable URL Filtering” check-box. This is essential to avoid TMG trying to contact the MRS services after they go offline.

clip_image002

Malware Inspection may continue to work but would not receive updated signatures

We would recommend implementing an alternative Anti-Virus solution and to disable Malware Inspection once this is in place.

As noted in the previous blog, Forefront Threat Management Gateway 2010, remains under extended support until April 14, 2020.

For details on moving from TMG to our new web publishing solutions please visit this URL:

http://blogs.technet.com/b/applicationproxyblog/archive/2015/07/02/transitioning-to-application-proxy-from-uag-and-tmg.aspx

Some Frequently Asked Questions we’ve had regarding the change are:-

Q1. Is it possible to use the MRS Cache to continue to benefit from URL Filtering after 31st December 2015?

A1. No, the MRS cache is a temporary in-memory cache of the latest lookups intended to provide internal efficiency optimizations. It does not provide a full offline cache and cannot be used for this purpose. There is no mechanism to have an offline database.

Q2. Is it possible to extend our usage of Forefront Threat Management Gateway (TMG) Web Protection Services past 31st December 2015?

A2. No, this is not possible. These dates were announced in September 2012 in order to provide sufficient time for alternative solutions to be deployed.

For the original announcement of the Forefront product roadmap changes please refer to the following URL:

http://blogs.technet.com/b/server-cloud/archive/2012/09/12/important-changes-to-forefront-product-roadmaps.aspx

Categories: EMP, malware inspection, TMG, URL filtering, URLF Tags:

Further details and guidance regarding discontinuation of TMG Web Protection Services

December 15th, 2015 No comments

As discussed in the following blog, the Forefront Threat Management Gateway (TMG) Web Protection Services will be discontinued on December 31st, 2015:-

http://blogs.technet.com/b/applicationproxyblog/archive/2015/11/02/important-reminder-for-forefront-threat-management-gateway-tmg-web-protection-services-customers.aspx

We wanted to provide some additional details on what this will affect and recommendations on actions you should be taking.

The services that will be affected by this are:-

– URL Categorization
– Malware Inspection

Importantly, the Microsoft Reputation Services that supports URL Filtering will be turned off on or shortly after the 31st December 2015.

To avoid service impacting issues due to these services no longer being available, or incorrect rule processing where rules rely on URL Categorization categories, we would strongly advise customers review and amend their TMG configurations as follows:-

Review and amend any rules based on URL Categorization categories in your TMG policy

Any Allow and Deny rules that currently use URL Categories or URL Category Sets must be changed to remove the usage of URL filtering categories.

Using URL Sets or Domain Name Sets may provide limited replacement functionality or you may also want to consider a 3rd party URL filtering plug-in or upstream proxy service to provide replacement URL filtering functionality.

Note – If you have rules that are using URL filtering to allow traffic – HTTP traffic can be totally blocked after the service shutdown. Equally, if you use URL Filtering to block access to certain categories then these may be allowed after the change. There is also a possibility that performance issues will be seen if URL Filtering is left enabled after the MRS service is taken offline.

Disable URL Filtering

After amending your TMG policy ensure you then disable URL Filtering. This can be done in the TMG Management Console in the Web Access Policy node by selecting URL Filtering and unchecking the “Enable URL Filtering” check-box. This is essential to avoid TMG trying to contact the MRS services after they go offline.

clip_image002

Malware Inspection may continue to work but would not receive updated signatures

We would recommend implementing an alternative Anti-Virus solution and to disable Malware Inspection once this is in place.

As noted in the previous blog, Forefront Threat Management Gateway 2010, remains under extended support until April 14, 2020.

For details on moving from TMG to our new web publishing solutions please visit this URL:

http://blogs.technet.com/b/applicationproxyblog/archive/2015/07/02/transitioning-to-application-proxy-from-uag-and-tmg.aspx

Some Frequently Asked Questions we’ve had regarding the change are:-

Q1. Is it possible to use the MRS Cache to continue to benefit from URL Filtering after 31st December 2015?

A1. No, the MRS cache is a temporary in-memory cache of the latest lookups intended to provide internal efficiency optimizations. It does not provide a full offline cache and cannot be used for this purpose. There is no mechanism to have an offline database.

Q2. Is it possible to extend our usage of Forefront Threat Management Gateway (TMG) Web Protection Services past 31st December 2015?

A2. No, this is not possible. These dates were announced in September 2012 in order to provide sufficient time for alternative solutions to be deployed.

For the original announcement of the Forefront product roadmap changes please refer to the following URL:

http://blogs.technet.com/b/server-cloud/archive/2012/09/12/important-changes-to-forefront-product-roadmaps.aspx

Categories: EMP, malware inspection, TMG, URL filtering, URLF Tags:

URL Filtering and Blocked URL requests on the Dashboard

From time to time we come across the question why are the values of the Blocked URL requests at URL Filtering increasing on the dashboard, although the URL Filtering feature is disabled.

clip_image002

In this post I would like to explain, what exactly we can see there.

The first value represents the number of the denied web requests in the last 24 hour.  This value equals the value of the Sites denied in last day (Forefront TMG Web Proxy) performance counter on a standalone TMG server. If you have an array the value shows the sum of the value of the Sites denied in last day (Forefront TMG Web Proxy) performance counters on the array members.

clip_image004

The second value represents the number of incoming web requests in the last 24 hour.  This value equals the value of the Total number of request in last day (Forefront TMG Web Proxy) performance counter on a standalone TMG server. If you have an array the value shows the sum of the value of the Total number of request in last day (Forefront TMG Web Proxy) performance counters on the array members.

clip_image006

The third value shows the rate of the values above as percentage.

The Total number of request in last day (Forefront TMG Web Proxy) counter will be incremented if a HTTP request hits the Web Proxy component.

Let’s have a closer look on what increases the Sites denied in last day (Forefront TMG Web Proxy) performance counter?

Well, there are different reasons behind this:

– Client authentication fails

– A rule does not allow accessing the web site

– User override scenario (http://technet.microsoft.com/en-us/library/ff685648.aspx)

– Error occurred on checking the URL categorization of the destination website

– At renegotiation of an SSL session, SSL certificate problem

– … etc.

As you can see the Sites denied in last day (Forefront TMG Web Proxy) counter covers more scenarios than just URL filtering.

Author:

Arpad Gulyas
Sr Support Enginner
Microsoft CSS Forefront Security Edge Team

Technical Reviewer:

Lars Bentzen
Sr. Escalation Engineer
Microsoft CSS Forefront Security Edge Team

Categories: TMG, URL filtering, URLF Tags:

Common Q&A about TMG URL Filtering database

November 15th, 2010 Comments off

URL filtering is one of Forefront TMG’s most popular features. The feature makes use of a cloud service, also known as Microsoft Reputation Services (MRS) for URL categorization purposes. In this post we’d like to address some of the more frequent questions we’ve received regarding the URL filtering database and the cloud service.

What is the URL categorization cloud service?

MRS can be thought of as a web service, providing secure access to a huge, cloud-based dynamic repository of URLs and their respective categories. The database features over 70 categories ranging from security-oriented selections like Malicious sites, through productivity-oriented categories such as Online Communities, and ending with liability-oriented categories like Pornography. The database spans tens of millions of unique URLs and their respective categories. Whenever a user behind TMG tries to access a URL, TMG can look up its category by issuing an online query to the cloud service. If you’d like to learn more about URL filtering and the cloud service, check out this post.

What are the sources of the URL filtering database?

The database merges data from several providers. The data providers include internal Microsoft sources as well as 3rd party sources. Microsoft signed agreements with BrightCloud (a Webroot subsidiary) and with M86 Security to consume URL categorization data. Those sources are now integrated into the database. It is important to note different providers employ different URL categorization techniques. Some employ manual classification while others rely heavily on web crawlers performing automated classification. The highly advanced automated classification techniques ensure TMG URL filtering is as competitive as it gets when it comes to the coverage of the web and reaction speed.

How frequently is the URL filtering database updated?

The database is constantly updated in order to cope with the dynamic nature of the web. New URLs are added, obsolete ones are removed and categories can change based on the page’s content. The update frequency varies per data provider and per URL category. For example, with security-oriented categories such as Phishing or Malicious sites, the database receives updates every 20 minutes. These highly frequent updates are required to protect customers from dynamic, emerging threats. With some other URL categories, the database updates are less frequent.

How good is the coverage for URLs outside the US?

The The URL filtering database ensures a global coverage, and is meant to serve TMG customers all over the world. The coverage is regularly monitored against indicators such as Alexa’s top Million URL list. In addition, telemetry data gathered from TMG deployments around the world is used to refine the database and improve the coverage for URLs that are popular with our customers. There is also a constant dialogue between Microsoft and the other data providers to ensure a focus on geographies with heavy TMG customer presence.

Is URL categorization performed based on the top level domain only or based on the full URL?

The categorization is based on the full path. This means http://www.contoso.com and http://www.contoso.com/sports could be assigned with different categories in the database.

Can a URL have more than a single category?

Yes. You can learn more about TMG’s support for multiple URL categories in this post.

Can TMG block adult text, images and videos from search results by web search engines?

Yes. You can learn more about TMG’s safe search enforcement support in this post.

What else is Microsoft doing to improve the URL categorization service?

Microsoft is constantly seeking customer feedback about MRS and working to address our customers’ concerns. For instance one thing we are working to improve is shortening the total time it takes for a customer to report a mistake in URL categorization on the MRS feedback portal to be corrected in the database.

 

Author:

Dotan Elharrar, Senior Program Manager, Forefront TMG

Reviewers:

David Strausberg, Technical Writer, Forefront TMG

Zakie Mashiah, Group Manager, Forefront TMG

Categories: URL filtering, URLF Tags: