Archive

Archive for the ‘Troubleshooting’ Category

What do I do if my updates don’t install?

August 15th, 2013 No comments

The easiest way to get security updates automatically is to go to the Microsoft Update website and make sure that you have automatic updating turned on.

Microsoft Update provides very important updates for performance and security. If you have automatic updating turned on and it isn’t working, you will see a warning message if an update can’t be installed.

To troubleshoot Microsoft Update in Windows 7, Windows Vista, and Windows XP

If your computer is running Windows 7, Windows Vista, or Windows XP and your updates don’t install, you can use the Windows Update troubleshooter to find and fix problems. To open the troubleshooter:

  1. Click the Start button.
  2. In the search box, type Troubleshooter.
  3. In the results list, click Find and fix problems.
  4. Under System and Security, click Fix problems with Windows Update.

To troubleshoot Microsoft Update in Windows 8

Windows 8 has an automatic troubleshooter that can fix some common problems with Windows Update. To open the troubleshooter:

  1. Swipe in from the right edge of the screen, and then tap Search.
    If you’re using a mouse, point to the upper-right corner of the screen, move the mouse pointer down, and then click Search.
  2. In the search box, type Troubleshooting.
  3. Tap or click Settings, and then tap or click Troubleshooting.
  4. Under System and Security, tap or click Fix problems with Windows Update.

Learn more about how to get security updates automatically.

For more information about troubleshooting problems with Windows Update in Windows 8, see Troubleshoot problems with installing updates.

Access to remote FTP server through TMG 2010 may fail with error 550 (Access Denied)

Hi everybody!

In this article we will see how to troubleshoot an issue with accessing an FTP server behind TMG 2010.

Imagine we have the following situation: a client PC on an internal corporate network want to access a remote FTP server through TMG 2010 using an FTP client such as, for example, FileZilla.

clip_image002[7]

The way the FTP is configured (authentication, encryption, ecc…) is out of interest for this case.

On the TMG server, we’ve created an access rule allowing “Read-Only” outbound requests for the FTP protocol:

clip_image004

clip_image006

When we try to connect to our remote FTP server using, for example, FileZilla, we may face the following error:

clip_image008

FTP connection issues through ISA/TMG could be related to many different aspects.

In the following article it’s possible to find a resolution for many of the most common problems:

http://technet.microsoft.com/en-us/library/bb794745.aspx

The problem we’re focusing on in this article, however, is not included in the above troubleshooting guide and depends on a specific by-design behavior of TMG server.

Basically, in our case we see that the connection attempt is failing due to a “550-Access Denied” error after having performed a MLSD command.

What is MLSD exactly ?

Here we can find a description of what MLSD is used for:

http://tools.ietf.org/html/draft-ietf-ftpext-mlst-16#section-7

As we can see from the above:

The MLST and MLSD commands are intended to standardize the file and directory information returned by the Server-FTP process. These commands differ from the LIST command in that the format of the replies is strictly defined although extensible.

In the default configuration of the TMG FTP Access filter in “Read-Only Mode”, the filter will only allow a specific subset of FTP commands. The MLSD command is not included in this set of “Read-Only” commands. FTP clients using LIST command will not experience this problem, since LIST is an allowed command.

Its easy to resolve the problem by allowing write-permissions in the FTP-Filter advanced properties of our access rule:

clip_image010

Now, granting write rights is not always a good choice, and most of the times this is not allowed nor suggested.

Nevertheless, a workaround exists for this situation: in fact, it’s possible to add the MLDS command in the “allowed-commands list” of the “Read-only” TMG FTP filter.

The following MSDN article explains how to configure add-ins:

http://msdn.microsoft.com/en-us/library/dd435753.aspx

Specifically:

FTP Access Filter

FTP Access Filter is an application filter that is installed with Forefront TMG. It enables FTP protocols. When running in read-only mode, FTP Access Filter blocks all commands in the control channel except the following commands: ABOR, ACCT, CDUP, CWD /0, FEAT, HELP, LANG, LIST, MODE, NLST, NOOP, PASS, PASV, PORT, PWD /0, QUIT, REIN, REST, RETR, SITE, STRU, SYST, TYPE, USER, XDUP, XCWD, XPWD, SMNT. This should block any writing to the server side. The default list of allowed commands can be replaced by a customized list that is written to the collection of vendor parameters sets (FPCVendorParametersSets) associated with the filter. The Firewall service must restarted for the new settings to take effect.

The above article provides a script example through which it is possible to customize FTP filter list. This way, it will be possible to keep the filter configured in Read-Only mode, and also allow the FileZilla connection to work as expected.

Hope this can be useful!

Let’s see you back with the next topic!!

Author:
Daniele Gaiulli

Support Engineer – EMEA Forefront Edge

Reviewer:
Philipp Sand

Support Escalation Engineer – EMEA Forefront Edge

Access to remote FTP server through TMG 2010 may fail with error 550 (Access Denied)

Hi everybody!

In this article we will see how to troubleshoot an issue with accessing an FTP server behind TMG 2010.

Imagine we have the following situation: a client PC on an internal corporate network want to access a remote FTP server through TMG 2010 using an FTP client such as, for example, FileZilla.

clip_image002[7]

The way the FTP is configured (authentication, encryption, ecc…) is out of interest for this case.

On the TMG server, we’ve created an access rule allowing “Read-Only” outbound requests for the FTP protocol:

clip_image004

clip_image006

When we try to connect to our remote FTP server using, for example, FileZilla, we may face the following error:

clip_image008

FTP connection issues through ISA/TMG could be related to many different aspects.

In the following article it’s possible to find a resolution for many of the most common problems:

http://technet.microsoft.com/en-us/library/bb794745.aspx

The problem we’re focusing on in this article, however, is not included in the above troubleshooting guide and depends on a specific by-design behavior of TMG server.

Basically, in our case we see that the connection attempt is failing due to a “550-Access Denied” error after having performed a MLSD command.

What is MLSD exactly ?

Here we can find a description of what MLSD is used for:

http://tools.ietf.org/html/draft-ietf-ftpext-mlst-16#section-7

As we can see from the above:

The MLST and MLSD commands are intended to standardize the file and directory information returned by the Server-FTP process. These commands differ from the LIST command in that the format of the replies is strictly defined although extensible.

In the default configuration of the TMG FTP Access filter in “Read-Only Mode”, the filter will only allow a specific subset of FTP commands. The MLSD command is not included in this set of “Read-Only” commands. FTP clients using LIST command will not experience this problem, since LIST is an allowed command.

Its easy to resolve the problem by allowing write-permissions in the FTP-Filter advanced properties of our access rule:

clip_image010

Now, granting write rights is not always a good choice, and most of the times this is not allowed nor suggested.

Nevertheless, a workaround exists for this situation: in fact, it’s possible to add the MLDS command in the “allowed-commands list” of the “Read-only” TMG FTP filter.

The following MSDN article explains how to configure add-ins:

http://msdn.microsoft.com/en-us/library/dd435753.aspx

Specifically:

FTP Access Filter

FTP Access Filter is an application filter that is installed with Forefront TMG. It enables FTP protocols. When running in read-only mode, FTP Access Filter blocks all commands in the control channel except the following commands: ABOR, ACCT, CDUP, CWD /0, FEAT, HELP, LANG, LIST, MODE, NLST, NOOP, PASS, PASV, PORT, PWD /0, QUIT, REIN, REST, RETR, SITE, STRU, SYST, TYPE, USER, XDUP, XCWD, XPWD, SMNT. This should block any writing to the server side. The default list of allowed commands can be replaced by a customized list that is written to the collection of vendor parameters sets (FPCVendorParametersSets) associated with the filter. The Firewall service must restarted for the new settings to take effect.

The above article provides a script example through which it is possible to customize FTP filter list. This way, it will be possible to keep the filter configured in Read-Only mode, and also allow the FileZilla connection to work as expected.

Hope this can be useful!

Let’s see you back with the next topic!!

Author:
Daniele Gaiulli

Support Engineer – EMEA Forefront Edge

Reviewer:
Philipp Sand

Support Escalation Engineer – EMEA Forefront Edge

Access to remote FTP server through TMG 2010 may fail with error 550 (Access Denied)

Hi everybody!

In this article we will see how to troubleshoot an issue with accessing an FTP server behind TMG 2010.

Imagine we have the following situation: a client PC on an internal corporate network want to access a remote FTP server through TMG 2010 using an FTP client such as, for example, FileZilla.

clip_image002[7]

The way the FTP is configured (authentication, encryption, ecc…) is out of interest for this case.

On the TMG server, we’ve created an access rule allowing “Read-Only” outbound requests for the FTP protocol:

clip_image004

clip_image006

When we try to connect to our remote FTP server using, for example, FileZilla, we may face the following error:

clip_image008

FTP connection issues through ISA/TMG could be related to many different aspects.

In the following article it’s possible to find a resolution for many of the most common problems:

http://technet.microsoft.com/en-us/library/bb794745.aspx

The problem we’re focusing on in this article, however, is not included in the above troubleshooting guide and depends on a specific by-design behavior of TMG server.

Basically, in our case we see that the connection attempt is failing due to a “550-Access Denied” error after having performed a MLSD command.

What is MLSD exactly ?

Here we can find a description of what MLSD is used for:

http://tools.ietf.org/html/draft-ietf-ftpext-mlst-16#section-7

As we can see from the above:

The MLST and MLSD commands are intended to standardize the file and directory information returned by the Server-FTP process. These commands differ from the LIST command in that the format of the replies is strictly defined although extensible.

In the default configuration of the TMG FTP Access filter in “Read-Only Mode”, the filter will only allow a specific subset of FTP commands. The MLSD command is not included in this set of “Read-Only” commands. FTP clients using LIST command will not experience this problem, since LIST is an allowed command.

Its easy to resolve the problem by allowing write-permissions in the FTP-Filter advanced properties of our access rule:

clip_image010

Now, granting write rights is not always a good choice, and most of the times this is not allowed nor suggested.

Nevertheless, a workaround exists for this situation: in fact, it’s possible to add the MLDS command in the “allowed-commands list” of the “Read-only” TMG FTP filter.

The following MSDN article explains how to configure add-ins:

http://msdn.microsoft.com/en-us/library/dd435753.aspx

Specifically:

FTP Access Filter

FTP Access Filter is an application filter that is installed with Forefront TMG. It enables FTP protocols. When running in read-only mode, FTP Access Filter blocks all commands in the control channel except the following commands: ABOR, ACCT, CDUP, CWD /0, FEAT, HELP, LANG, LIST, MODE, NLST, NOOP, PASS, PASV, PORT, PWD /0, QUIT, REIN, REST, RETR, SITE, STRU, SYST, TYPE, USER, XDUP, XCWD, XPWD, SMNT. This should block any writing to the server side. The default list of allowed commands can be replaced by a customized list that is written to the collection of vendor parameters sets (FPCVendorParametersSets) associated with the filter. The Firewall service must restarted for the new settings to take effect.

The above article provides a script example through which it is possible to customize FTP filter list. This way, it will be possible to keep the filter configured in Read-Only mode, and also allow the FileZilla connection to work as expected.

Hope this can be useful!

Let’s see you back with the next topic!!

Author:
Daniele Gaiulli

Support Engineer – EMEA Forefront Edge

Reviewer:
Philipp Sand

Support Escalation Engineer – EMEA Forefront Edge

Errors When Using the FEP 2010 Definition Update Automation Tool

by Michael Cureton

We’ve become aware of two issues when using the Definition Update Automation Tool. This blog article presents workarounds for the issues.

Definition Update Automation Tool fails to add new definition updates to the deployment package

 

Symptoms

The FEP 2010 Definition Update Automation Tool may fail to add new definition updates to your deployment package. Reviewing the %ProgramData%\SoftwareUpdateAutomation.log file shows the following exception:

SmsAdminUISnapIn Error: 1 : Unexpected exception: System.ArgumentException: An item with the same key has already been added.
  at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)
  at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
  at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
  at Microsoft.Forefront.EndpointProtection.SoftwareUpdateAutomation.SccmUtilities.CalculateCleanupDelta(ConnectionManagerBase connection, ICollection`1 freshUpdateFilesObjectList, IResultObject destinationPackageObject)
  at Microsoft.Forefront.EndpointProtection.SoftwareUpdateAutomation.SoftwareUpdater.Update(SoftwareUpdateAutomationArguments arguments)
  at Microsoft.Forefront.EndpointProtection.SoftwareUpdateAutomation.SoftwareUpdater.Main(String[] args)

 

Cause

More than one FEP 2010 definition update is being detected as active by the tool.

More Information

The FEP 2010 Definition Update Automation tool queries WMI (SELECT * FROM SMS_SoftwareUpdate WHERE ArticleID=2461484 AND IsSuperseded=0 AND IsEnabled=1) to get the single active FEP 2010 definition update. The exception happens as a result of more than one update being returned. The tool may detect more than one update as being active when one of the two conditions is TRUE:

  1. One or more FEP 2010 definition updates has been expired but not superseded, OR
  2. One or more FEP 2010 definition updates has been orphaned.

To confirm if you’re experiencing condition #1 or #2, run the below WMI query:

SELECT * FROM SMS_SoftwareUpdate WHERE ArticleID=2461484 AND IsSuperseded=0 AND IsEnabled=1 AND IsExpired=0

If the query only returns one row, then you are experiencing condition #1. If two or more rows are returned, you are experiencing condition #2.

Workarounds

Condition #1

If you are experiencing condition #1, you can prevent the symptom by simply adding the /UpdateFilter flag to the command line for the tool (SoftwareUpdateAutomation.exe) with the appropriate values to filter out expired definition updates that are not superseded.

For example:

SoftwareUpdateAutomation.exe /AssignmentName <AssignmentName> /PackageName <DeploymentPkgName> /UpdateFilter “ArticleID=2461484 AND IsSuperseded=0 AND IsEnabled=1 AND IsExpired=0”

Condition #2

If you are experiencing condition #2, you will need to manually decline the orphaned updates via the WSUS administration console. For each update returned from the WMI query that you used to confirm that you have condition #2, double-click on the LocalizedDisplayName property and note the definition version. The update with the highest definition version will be the active one. The update(s) with the lower definition versions have been orphaned.

For example, using the list below, 1.107.713.0 would be the active update and the other two updates are orphaned and would need to be declined manually in WSUS.

Definition Update for Microsoft Forefront Endpoint Protection 2010 – KB2461484 (Definition 1.103.1405.0)
Definition Update for Microsoft Forefront Endpoint Protection 2010 – KB2461484 (Definition 1.105.2231.0)
Definition Update for Microsoft Forefront Endpoint Protection 2010 – KB2461484 (Definition 1.107.713.0)

After you have determined the orphaned update(s) title (and version), load the WSUS snap-in and drill down to the Updates node. On the action pane, click New Update View. Select “Updates are in a specific classification” and “Updates are for a specific product”. In step 2, click any classification and ensure that only Definition Updates is checked. Next click any product and ensure that only Forefront Endpoint Protection 2010 is checked. In step 3, specify a name for the view and click OK.

Locate the created view in the WSUS console. Change the Approval value to “Any Except Declined” and the Status to “Any” and hit Refresh. Click the Title column so that the results are sorted using the version. Find the orphaned update(s) that you identified by version and select the Decline action for each. Once this is complete, you’ll need to wait for the next scheduled Software Update Point (SUP) sync to complete, at which time the updates that you declined will be marked as expired in the ConfigMgr database.

NOTE: Running a manual SUP sync will NOT expire the declined updates. Only a scheduled sync will perform this operation.

Once the sync is complete, you can run the WMI query used to determine condition to confirm that only one row is now returned. You will also need to run the tool going forward using the condition #1 workaround with the /UpdateFilter flag.

Definition Update Automation Tool does not refresh distribution points

 

Symptoms

The FEP 2010 Definition Update Automation Tool does not refresh distribution points (DPs) by default. Even though the help output for the tool states that /RefreshDP is set by default, it is not.

 

Workarounds

Add /RefreshDP to the command line for the tool (SoftwareUpdateAutomation.exe). For example:

SoftwareUpdateAutomation.exe /AssignmentName <AssignmentName> /PackageName <DeploymentPkgName> /RefreshDP

Important Security Update for Windows Server: Active Directory Certificate Services Web Enrollment!

June 14th, 2011 No comments

An important security update, described in MS11-051 (http://go.microsoft.com/fwlink/?LinkId=217101) was released today. The update fixes a cross-site scripting vulnerability in the sample web enrollment ASP pages that are part of Active Directory Certificate Services Web Enrollment in Windows Server 2003, Windows Server 2008, and Windows Server 2008 R2.

Important: Back up any sample web enrollment sample pages you modified (%windir%\system32\Certsrv) before applying MS11-051. After you apply the security update, you can integrate any changes you made to the original sample files into the new secure ASP sample pages. For more information, see Microsoft Knowledge Base Article 2518295 (http://support.microsoft.com/kb/2518295).

Forefront Endpoint Protection (FEP) 2010: FEP Reports may not display properly

From Angela Latimer, CSS

If you are using Forefront Endpoint Protection (FEP) 2010, you may have tried running one of the three default FEP reports and noticed that not all areas or sub-reports display properly. You may see an error in processing the reporting data or retrieving the data, similar to the error displayed below:

Error while trying to run the Antimalware Activity Report:

clip_image002

We found this error was due to the installed version of Microsoft SQL Server not being up-to-date with the latest Cumulative Update package. Cumulative Update packages contain hot fixes that address issues in the currently installed version of Microsoft SQL Server which may be versions ranging from Release to Manufacturing (RTM), Service Pack (SP), or Feature Release (R).

In digging into the details of the error related to FEP reports not displaying properly, we found the following errors in the System Center Configuration Manager Console and/or in the %drive%:\Program Files (x86)\Microsoft Configuration Manager\Logs\SRSRP.log file, reporting Error ID 7403 related to the health of SRS Reporting Point thread:

STATMSG: ID=7403 SEV=E LEV=M SOURCE=”SMS Server” COMP=”SMS_SRS_REPORTING_POINT” SYS= SITE= PID=2880 TID=5572 GMTDATE=Wed Oct 21 17:57:26.302 2009 ISTR0=”HACM01″ ISTR1=”” ISTR2=”” ISTR3=”” ISTR4=”” ISTR5=”” ISTR6=”” ISTR7=”” ISTR8=”” ISTR9=”” NUMATTRS=0 SMS_SRS_REPORTING_POINT 10/21/2009 10:57:26 AM 5572 (0x15C4)  
Failures reported during periodic health check by the SRS Server . Will retry check in 57 minutes SMS_SRS_REPORTING_POINT 10/21/2009 10:57:26 AM 5572 (0x15C4)

In the two environments we discovered this issue, Microsoft SQL Server 2008 and SQL Server 2008 R2 were running, but had NOT had the Cumulative Update package installed. As soon as this update was installed, the FEP reports began displaying properly.

At the time of this blog, these are the most current Cumulative Update Packages for Microsoft SQL Server 2008 and 2008 R2. However, you should do a Bing search to ensure you are always installing the latest version.



Using the MscSupport tool to collect data for troubleshooting

February 1st, 2011 Comments off

 

The MscSupport tool is a tool designed to collect support data to troubleshoot Forefront Endpoint Protection. You can download the tool from the Forefront Endpoint Protection 2010 Tools download page (http://www.microsoft.com/downloads/en/details.aspx?FamilyID=04f7d456-24a2-4061-a2ed-82fe93a03fd5).

When to use the MscSupport tool

It is a troubleshooting tool, so you only need to run the tool when you have a problem with Forefront Endpoint Protection.
On the other hand, you don’t need to run the tool with every occasion. Typically you need to collect the MscSupport data in the following scenarios:

  • Remote online troubleshooting is difficult
  • The cause of the problem is not clear
  • You have a Support case with Microsoft

What data does the tool collect

The data collected depends on the system you run the tool on. The tool collects additional information when it is run on the server hosting the FEP2010 server roles.

The Support files are files that contain FEP2010 specific information. This information can be gathered when you run the below command (located in C:\Program Files\Microsoft Security Client\Antimalware) in a Command Prompt:

Mpcmdrun -GetFiles

The following data is collected:

  • Any trace files from Microsoft Antimalware Service
  • The Windows Update history log
  • All Microsoft Antimalware Service events from the System event log
  • All relevant Microsoft Antimalware Service registry locations
  • The log file of this tool
  • The log file of the signature update helper tool

Microsoft is committed to protecting your privacy. Please read the Microsoft Privacy Statement<http://go.microsoft.com/fwlink/?LinkId=81184> for more information.

How to run the MscSupport tool

The tool must be executed with Administrator privileges on the system you want to collect the data from, otherwise the data collected by the tool may not be complete.

The data the tool collects will be placed in a cabinet file and is located in %SystemDrive%\MscSupportData

  1. Open Windows Explorer and navigate to the location where you stored the tool
  2. Right-click MscSupportTool.exe and click Run as administrator
  3. The tool will start to collect the support data

    clip_image001

  4. When data gathering is complete, you can close or open the folder that contains the CAB file

    clip_image002

Kurt Sarens, Senior Support Engineer

Categories: FEP, log files, mscsupport, Troubleshooting Tags:

ISA Firewall Service Process (wspsrv.exe) high CPU utilization issue

January 26th, 2011 Comments off

1. Introduction

 

When dealing with ISA high CPU utilization where wspsrve.exe is the one consuming more resources, the first impression is that ISA is the culprit for that. There are some scenarios where this statement is true, such as this one that I documented last year. But there are other scenarios where this is not true, including scenarios where wspsrv.exe is crashing – not always is because of ISA itself. I tried to demystify this on this post that I wrote back in 2009.

 

This post is about a scenario where ISA Server was facing a 100% CPU utilization and wspsrv.exe process was consuming most part of that (around 93%). This was happening at least twice a day and the only way to get CPU get to a normal utilization was by restarting Firewall Service. Sounds like an ISA issue right? Not really and you will see.

 

2. Gathering Data

 

Use the same data gathering approach as the one listed in this post with an addition of getting a full memory dump using notmyfault tool, see KB972110 for more info.

 

3. Analyzing the Data

 

By reviewing perfmon data from the time that the issue was happen it was possible to see that wspsrv.exe was 88% average of CPU utilization, but notice that within this process there is no thread spending too much CPU cycles as shown below:

image

In order to better understand this issue we started to review the manual dump of wspsrv.exe process. By reviewing the user mode dump we were unable to determine what the root cause of the issue was. Most of the threads were waiting for something in Kernel mode as shown below:

 

> First step was to review the top threads that had calls in Kernel mode:

 

0:000> !runaway 7

 User Mode Time

… [hiding the values as they are not relevant here]

 Kernel Mode Time

  Thread       Time

  69:1310      0 days 0:00:49.500

  53:129c      0 days 0:00:46.812

 104:113c      0 days 0:00:42.390

  45:127c      0 days 0:00:40.875

  78:688       0 days 0:00:40.734

  94:4d0       0 days 0:00:38.890

 162:18a8      0 days 0:00:38.500

 166:18c8      0 days 0:00:38.093

 108:1418      0 days 0:00:37.062

 

> By taking a closer look into the threads we see that some of them are either performing RPC or DNS such as the one below:

 

0:000> ~104kb

ChildEBP RetAddr  Args to Child             

WARNING: Stack unwind information not available. Following frames may be wrong.

28cada54 77c7feb0 28cada90 28cada74 77c80845 ntdll!KiFastSystemCallRet

28cada60 77c80845 28cada90 76ed42d8 28cade84 rpcrt4!I_RpcSendReceive+0×23

28cada74 77ce415a 28cadabc 24b86ff4 00000000 rpcrt4!NdrSendReceive+0×28

28cade64 76ed5049 76ed42d8 76ed421c 28cade84 rpcrt4!NdrClientCall2+0x1a8

28cade7c 76ed4f69 00000000 29e2f078 00000001 dnsapi!NetInfo_Copy+0x5c2

28caded8 76ed6f5d 29e2f078 00000001 00000000 dnsapi!NetInfo_Copy+0x4e2

28cadf0c 76ee9d0c 00000003 28cae1d8 00000001 dnsapi!DnsValidateName_W+0x31f

28cadf34 62ea2cc4 28cae1d8 00000001 00000000 dnsapi!DnsQuery_A+0×20

28cadf78 62ea30a2 28cae1d8 00000001 28cadfb8 msphlpr!ProxyDnsCacheInit+0x52f

28cadf98 62ea217c 28cae1d8 28cadfb8 62ea1f0f msphlpr!ProxyDnsCacheInit+0x90d

28cae2dc 64760c2a 021d7710 28caeedc 00000001 msphlpr!ProxyGetHostByName+0x26d

28cae764 64761d11 28caeed4 00000400 28caee74 W3Filter!DllUnregisterServer+0x42ca6

28cae9d4 647638dc 28caeed4 00000400 28caee74 W3Filter!DllUnregisterServer+0x43d8d

28caf534 647295c7 00000000 00000000 28cafe14 W3Filter!DllUnregisterServer+0×45958

28caf554 647262ef 021d7008 00000000 000003e5 W3Filter!DllUnregisterServer+0xb643

28caf574 647277f8 28cafe14 00000000 00000001 W3Filter!DllUnregisterServer+0x836b

28caf590 647281c5 28cafe14 28038ce0 28038728 W3Filter!DllUnregisterServer+0×9874

28cafe1c 64728ad0 000000a7 00000000 0000000c W3Filter!DllUnregisterServer+0xa241

28cafe90 0046051a 28038728 00000001 00000000 W3Filter!DllUnregisterServer+0xab4c

28cafefc 0046032d 000000a7 00000000 00000001 wspsrv!SSL_CREDENTIALS::operator=+0x2a870

28caff20 00460bea 2685d018 004602b1 28caff50 wspsrv!SSL_CREDENTIALS::operator=+0x2a683

28caff30 00460266 000000a7 00000000 00000001 wspsrv!SSL_CREDENTIALS::operator=+0x2af40

28caff50 0044b0ff 2685d0bc 000000a7 00000000 wspsrv!SSL_CREDENTIALS::operator=+0x2a5bc

28caff7c 0044c08e 2685d0bc 000000a7 00000000 wspsrv!SSL_CREDENTIALS::operator=+0×15455

28caffb8 77e6482f 0000004f 00000000 00000000 wspsrv!SSL_CREDENTIALS::operator=+0x163e4

28caffec 00000000 0044bf26 0000004f 00000000 kernel32!GetModuleHandleA+0xdf

 

It was time to review the kernel dump file and verify if we had more information there:

 

> In this case we first started reviewing if there was anything locked in kernel space:

 

1: kd> !locks

**** DUMP OF ALL RESOURCE OBJECTS ****

KD: Scanning for held locks……………………………………………………

 

Resource @ 0x8a7f3008    Shared 1 owning threads

    Contention Count = 856651

    NumberOfExclusiveWaiters = 5

     Threads: 84149020-01<*>

     Threads Waiting On Exclusive Access:

              84166a80       8961d620       84166750       89649240      

              841777a0      

 

 

> Notice that we have 5 exclusive waiters wanting on a thread. Let’s see which thread is that:

 

1: kd> !thread 84149020

THREAD 84149020  Cid 1074.18a8  Teb: 7ff12000 Win32Thread: 00000000 READY

IRP List:

    fe5eae28: (0006,01d8) Flags: 00000884  Mdl: 00000000

    88bb1a68: (0006,01d8) Flags: 00000000  Mdl: 89128478

    fee50ba8: (0006,01d8) Flags: 00000000  Mdl: 880eb6c8

    f9d62af8: (0006,01d8) Flags: 00000000  Mdl: 887adee8

    fa284008: (0006,01d8) Flags: 00000000  Mdl: 87fb7bb8

    feb2dd80: (0006,01d8) Flags: 00000000  Mdl: 88bbdc48

    ff8c95f0: (0006,01d8) Flags: 00000000  Mdl: 88bbe488

    fac4b820: (0006,01d8) Flags: 00000000  Mdl: 8847d798

    fea1d008: (0006,01d8) Flags: 00000000  Mdl: 89252968

    ff097008: (0006,01d8) Flags: 00000000  Mdl: 8a5ee320

    fe64b008: (0006,01d8) Flags: 00000000  Mdl: 88a114a0

    feba1b30: (0006,01d8) Flags: 00000000  Mdl: 8975beb8

    ff499550: (0006,01d8) Flags: 00000000  Mdl: 87cd53a8

    ff5cf9f0: (0006,01d8) Flags: 00000000  Mdl: 8856dd18

    fc26ec38: (0006,01d8) Flags: 00000000  Mdl: 883df0a8

Not impersonating

DeviceMap                 e1750820

Owning Process            89f4ead8       Image:         wspsrv.exe

Wait Start TickCount      4518588        Ticks: 0

Context Switch Count      3718083            

UserTime                  00:01:15.546

KernelTime                00:01:02.906

Win32 Start Address 0x0044bf26

Start Address 0x77e617ec

Stack Init b7b39000 Current b7b38898 Base b7b39000 Limit b7b36000 Call 0

Priority 8 BasePriority 8 PriorityDecrement 0

ChildEBP RetAddr  Args to Child             

b7b388b0 8083d5b1 84149020 841490c8 00000000 nt!KiSwapContext+0×26 (FPO: [Uses EBP] [0,0,4])

b7b388dc 8083df9e 84149020 8a7f3008 00000000 nt!KiSwapThread+0x2e5 (FPO: [Non-Fpo])

b7b38924 8082e057 88f99718 0000001b 00000000 nt!KeWaitForSingleObject+0×346 (FPO: [Non-Fpo])

b7b3895c 80824ba8 89a0a400 00000000 8082c38e nt!ExpWaitForResource+0×30 (FPO: [Non-Fpo])

b7b3897c 8085110e 8a7f3008 00000001 b7b389d4 nt!ExAcquireResourceSharedLite+0xf5 (FPO: [Non-Fpo])

b7b3898c b9fb5c70 8a7f3008 89a0a400 00000000 nt!ExEnterCriticalRegionAndAcquireResourceShared+0×19 (FPO: [Non-Fpo])

b7b389d4 b9fb5b17 b7b38a18 b7b389ec 00000028 afd!AfdGetTransportInfo+0x1b (FPO: [Non-Fpo])

b7b389f0 b9fb5d9f b7b38a34 b7b38a18 00000000 afd!AfdAllocateEndpoint+0x1e (FPO: [Non-Fpo])

b7b38a38 b9fbf7a4 89b1e438 8a9a9f38 b7b38a5c afd!AfdCreate+0x13b (FPO: [Non-Fpo])

b7b38a48 80840153 8a9bc030 fe5eae28 fe5eae28 afd!AfdDispatch+0×79 (FPO: [Non-Fpo])

b7b38a5c 8092e87e b7b38c04 8a9bc018 00000000 nt!IofCallDriver+0×45 (FPO: [Non-Fpo])

b7b38b44 8092c3ea 8a9bc030 00000000 89062228 nt!IopParseDevice+0xa35 (FPO: [Non-Fpo])

b7b38bc4 8092d80d 00000000 b7b38c04 00000042 nt!ObpLookupObjectName+0x5b0 (FPO: [Non-Fpo])

b7b38c18 8092c6cd 00000000 00000000 dffa7c01 nt!ObOpenObjectByName+0xea (FPO: [Non-Fpo])

b7b38c94 80931d92 29b2f958 c0140000 29b2f90c nt!IopCreateFile+0×447 (FPO: [Non-Fpo])

b7b38cf0 8092fb28 29b2f958 c0140000 29b2f90c nt!IoCreateFile+0xa3 (FPO: [Non-Fpo])

b7b38d30 80833bef 29b2f958 c0140000 29b2f90c nt!NtCreateFile+0×30 (FPO: [Non-Fpo])

b7b38d30 7c82860c 29b2f958 c0140000 29b2f90c nt!KiFastCallEntry+0xfc (FPO: [0,0] TrapFrame @ b7b38d64)

WARNING: Frame IP not in any known module. Following frames may be wrong.

29b2f9ec 00000000 00000000 00000000 00000000 0x7c82860c

 

> This stack has a very good amount of IRPs (I/O Request Packet), let’s see if there is any pending IRP:

 

1: kd> !irp 88bb1a68

Irp is active with 4 stacks 2 is current (= 0x88bb1afc)

 Mdl=89128478: No System Buffer: Thread 84149020:  Irp stack trace. 

     cmd  flg cl Device   File     Completion-Context

 [  0, 0]   0  0 00000000 00000000 00000000-00000000   

 

                     Args: 00000000 00000000 00000000 00000000

>[  f, 8]   0 e1 8a94ddf0 870404b0 b9e964d0-870f66c8 Success Error Cancel pending

              \Driver\Tcpip*** ERROR: Module load completed but symbols could not be loaded for idmtdi.sys

       idmtdi

                     Args: 0000127b 00010020 00000000 00000000

 [  f, 8]   0 e0 8a85d020 870404b0 b9fc1b72-833955a0 Success Error Cancel

              \Driver\IDMTDI      afd!AfdRestartBufferReceiveWithUserIrp

                     Args: 0000127b 00010020 00000000 00000000

 [  e, 5]   5  0 8a9bc030 8583a190 00000000-00000000   

              \Driver\AFD

                     Args: 0000127b 00000000 20000020 00000000

 

 

> Notice that there is a third party transport driver interface (idmtdi.sys) loaded within this IRP and it is pending.

 

4. Conclusion

As you could see not always the user mode dump will give us the definitive answer for a problem, mainly when most of the threads on user space are waiting for something to happen on kernel space. In this particular case, when reading the kernel dump, we used the !locks command, which displays all locks held on resources by threads. A lock can be shared or exclusive, which means no other threads can gain access to that resource. In this case we saw that we had five threads waiting for exclusive access. The threads that were waiting for exclusive access were part of the wspsrv.exe.

What we can conclude here is that we have a condition where wspsrv.exe was allocating more and more threads without releasing resources (releasing threads) because it was getting stuck waiting for an answer from kernel driver. Some threads were working on I/O, mainly socket and LRPC call (almost socket) that run in Kernel mode. The third party driver is causing every AFD call to take more time and more CPU cycles, the fact that ISA is extremely using sockets amplify the problem. The third party kernel driver was uninstalled from ISA Server and the issue was resolved.

Another takeaway from this particular scenario is that not always that wsprsv.exe is using lot of CPU resources is because it’s an ISA issue. Keep that in mind.

 

Author

Gabriel Koren

Sr. Support Escalation Engineer

Microsoft CSS Forefront Edge Team

 

Technical Reviewer

Franck Heilmann

Sr. Escalation Engineer

Microsoft CSS Forefront Edge Team

 

Categories: ISA, Performance, Troubleshooting Tags:

Random authentication prompts while accessing internet through ISA Server followed by ISA Server becoming unresponsive

January 13th, 2011 Comments off

Introduction

Consider a scenario where users behind ISA Server (internal network) start to receive random prompts for authentication while trying to access internet using ISA Server as proxy. The authentication prompt persists even after entering the credentials. To resolve the issue it is necessary to restart Firewall Service.

Although you probably heard or read about this scenario many times, the goal of this post is to give you a compiled version of the action plan and what to look for while analyzing the data.

Data Collection

Start by following the plan from this post (basics section), along with that make sure that binding order is also correct i.e. internal NIC is higher in order then the external. Wrong binding order can cause issues such as the one mentioned here. In addition to the data gathering specified previously, also collect the following data:

1. Use ISA Data Packager while doing repro of the issue.
2. Enable netLogon logging on the ISA server nodes, using command nltest /dbflag:0x2080ffff in the command prompt as per KB109626.
2. Set the Performance counters as specified in this post.

Data analysis

When start reviewing the perfmon data you want to check the counter ISA Server Firewall Packet Engine\Backlogged Packets. You will notice a trend similar to the perfmon screenshot showed in this post. This can happen due name resolution issue as explained in this TechNet Article.

Next data to analyze is the netlogon.log, which also can be done using the same approach as the following post. In other words, look for the following pattern:

08/21 12:00:00 [DOMAIN] Contoso: Domain thread started 08/21 12:00:00 [DOMAIN] Contoso: Domain thread started doing API timeout 08/21 12:00:00 [SESSION] Contoso: Contoso: NlTimeoutApiClientSession: Unbind from server \\ab-cd.Contoso.local (TCP) 0.

From above data it appeared we can conclude that the Domain Controller to which ISA server had the secure channel established with, did not responded in time manner, which triggered the NlTimeoutApiClientSession in the netlogon logging. After that ISA Server resets the secure channel and tries to make secure channel with another DC.

Resolution for this Particular Case

In this particular case the clients were using WPAD (automatic detection), which by default returns the IP address of the ISA Server rather than the name. This forced the client to use NTLM authentication rather than Kerberos (supported in IE7 or higher).

Note: The advantages to use Kerberos instead of NTLM are documented in this article.

In order to force WPAD to use FQDN instead of IP address we ran the script described in this post. After running the script, all the web proxy clients using WPAD started getting FQDN of the ISA server nodes and use Kerberos for authentication, which enhance the authentication traffic and decrease the number of authentication request.

Author
Suraj Singh
Security Support Engineer
Microsoft CSS Forefront Security Edge Team

Technical Reviewer
Yuri Diogenes
Sr Security Support Escalation Engineer
Microsoft CSS Forefront Security Edge Team

Windows Update fails for some workstations behind TMG when using WPAD

November 30th, 2010 Comments off

Introduction
This post is about a recent scenario where TMG Administrator was receiving complains that some workstations that were using TMG as proxy were failing to run Windows Update. The interesting part of this issue was that only some workstations were having such problem and only if they were using “Automatic Detection” settings (which use WPAD). But all other workstations were using the same setting and were working just fine.

Data Gathering
In order to troubleshoot this I started TMG Data Packager in repro mode using Web Proxy and Web Publishing template and performed the following steps in the client workstation that was having the issue:

  1. Clear the wpad cache by executing the following command in an elevated command window: del \wpad*.dat /s
  2. Restart Windows Update service
  3. Wait for the error to happen on Windows Update.

Data Analysis
I started the data review by looking to the TMG Logging and notice that when it failed the following URL was sent it back to the client:

http://www.update.microsoft.com/microsoftupdate/v6/errorinformation.aspx?error=-2145107946&ln=en-us&IsMu=true&wgaerrorcode=0&wgaend=http://www.update.microsoft.com/microsoftupdate/v6/default.aspx

Notice that this URL returns the error -2145107946 (in decimal), which corresponds to 0×80244016 (in Hex), which means the following:

WU_E_PT_HTTP_STATUS_BAD_REQUEST wuerror.h
# Same as HTTP status 400 – the server could not process the
# request due to invalid syntax.

Having that info, it was time to review the netmon trace to understand why the request was invalid and after reading the trace, it was possible to understand why it was invalid.

Conclusion
By using Netmon it was possible to see the moment that client downloads the WPAD file and tries to access the TMG. But, in it can’t access for some reason (in this case it was a networking routing issue) and switches to another TMG. I wasn’t aware that this environment had another TMG, so I opened the WPAD file and found the following entry called BackupRoute:

BackupRoute="FFTMGEEN2.contoso.com";
UseDirectForLocal=true;
ConvertUrlToLowerCase=false;

This was a backup TMG that was supposed to be used only if the main one was down. So the problem here was:

  1. Both TMG servers were listening for CERN proxy requests on the default port 8080.
  2. They were also listening for WPAD requests on port 80.
  3. The CONNECT requests sent from the client to the FFTMGEEN2.contoso.com was done on port 80 (wpad listener).This caused the failure (error 0×80244016) since the only valid request to this listener is GET /wpad.dat or GET /array.dll?GetRouting.Script.

Solution
In order to fix this it was necessary to change the entry on the Alternate Forefront TMG field within TMG console to be as shown below:

image

Once this change was done, the WPAD file changed to have the following entry on the backup route:

image

Have a happy Windows Update process behind TMG !!

Author
Yuri Diogenes
Senior Support Escalation Engineer
Microsoft CSS Forefront (ISA/TMG) Team

Categories: TMG, Troubleshooting, windows update Tags:

“No network adapters could be identified” error when choosing a network template in TMG

November 19th, 2010 Comments off

Introduction
Some of our customers have experienced the problem described below when doing the initial network configuration of a fresh TMG installation. I wanted to share here the cause and solution to this issue.

Consider the following scenario
You have installed Forefront TMG 2010, but when running the Getting Started wizard, you get the error “No network adapters could be identified. The wizard cannot continue” when choosing the Network Template, see screenshot below:

image

Cause
We’ve seen this issue on servers where Operating System hardening has been applied prior to TMG installation. As a result, some services required by TMG to operate properly have been wrongly disabled causing the problem described above.

Solution
The solution and only supported way to perform hardening of a TMG machine is to execute the Security Configuration Wizard (SCW) tool and use the TMG security configuration template (XML file) matching your deployment in order to harden properly the server.

Note: By default, the SCW does not include support for the TMG 2010 role nor TMG Enterprise Management Server (EMS) role. To support these roles, download and install TMGRolesForSCW.exe included in the TMG 2010 Tools and Software Development Kit (SDK), available here.

Author
Eric Detoc
Escalation Engineer – Microsoft CSS Forefront Security Edge Team

Technical Reviewer
Franck Heilmann
Escalation Engineer – Microsoft CSS Forefront Security Edge Team

Categories: hardening, SCW, TMG, Troubleshooting Tags:

Unable to join to TMG EMS Array with error: 0xC0040431

November 17th, 2010 Comments off

Introduction

Consider a scenario where TMG Admin reinstalled TMG Enterprise Edition after a hardware failure and decided to rejoin the array member to the EMS. However, when TMG Admin tried to rejoin it the following error occurred:

image

Troubleshooting

The first basic step in this type of scenario is to review the event view, since it can give you some clues about what is going on. Most of TMG events will be found on Application Event log. So, on Event View we found the following error:

 

Log Name: Application
Source: Microsoft Forefront TMG Control
Event ID: 11003
Level: Error
Computer: srvtmg.contoso.com

Description: Microsoft Forefront TMG Control encountered a failure. The failure occurred during creation of logging module because the configuration property msFPCLogFileDirectory of key SOFTWARE\Microsoft\Fpc\Storage\EffecTree2\Array-Root\Arrays\{7855C30A-6FAF-44D1-908E-264466A3EAE8}\Logs\Proxy-WSP is not valid. Use the source location 5.846.7.0.9027.400 to report the failure. The error description is: The system cannot find the path specified.

Since we are joining to an EMS all setting including log configuration will be retrieved now from it. So, after checking the Log configuration on EMS for that particular Array where TMG Server was trying to get joined, we found that setting:

image

The EMS had a configuration for all logs should get stored on volume L:\. We investigates that the target server trying to rejoin did have a volume dedicated for log but it had the letter D: assigned to it. After getting the volume changed to letter L: the joining process worked fine. This change on the drive letter happened while TMG Admin was rebuilding the server after the hardware failure.

Conclusion

This post showed how TMG administrators need to be aware that EMS configuration will try to enforce the settings, in this particular scenario for a logging path that did not exist in the joining array member. It is a good practice to always have the core settings matching in between the EMS and the array members. Additionally document your recovery process making sure that all the requirements are in place to not get surprises down the road.

Author
Daniel Mauser
Sr. Support Escalation Engineer
LATAM Team

Technical Reviewer
Yuri Diogenes
Sr. Support Escalation Engineer
Forefront Edge Team

Categories: array, ems, join, TMG, Troubleshooting Tags:

What is NAP traffic?

January 6th, 2009 No comments

Here is a question posed by a member of the NAP community:


·         What new traffic will there be on the network when I deploy NAP?


A NAP deployment can have the following additional sets of network traffic:


·         Traffic between the NAP client and the NAP enforcement point. The nature of this traffic depends on the NAP enforcement method.


o       For IPsec enforcement, the NAP client communicates to the HRA using HTTP or HTTPS to indicate its identity and health state and to receive the system health evaluation results and the health certificate.


o       For 802.1X enforcement, the NAP health evaluation is done over PEAP-TLV, resulting in a small amount of additional EAPOL traffic to send the health state and health evaluation results between the NAP client and the switch or wireless access point.


o       For VPN enforcement, the NAP health evaluation is done over PEAP-TLV, resulting in small amount of additional PPP traffic to send the health state and health evaluation results between the NAP client and the VPN server.


o       For DHCP enforcement, the NAP health evaluation is done using the same DHCP messages that are already being used for DHCP address allocation, resulting in larger payloads for some DCHP messages, but not additional messages.


o       For TS Gateway enforcement, the NAP health evaluation is done over the Remote Procedure Call (RPC) over HTTP protocol that is used for connections to a TS Gateway server, resulting in a small amount of additional traffic to send the health state from the TS Gateway client and the TS Gateway server.


·         Traffic between the NAP enforcement point and the NAP health policy server. This is RADIUS traffic, consisting of one or multiple exchanges of RADIUS request and response messages. RADIUS traffic is UDP-based and adds minimal additional traffic on your network.


·         Traffic between the NAP enforcement point and other servers. The most obvious example is the traffic between the Health Registration Authority (HRA) and an Active Directory domain controller and a certification authority (CA) to authenticate the NAP client and obtain a health certificate.


·         Traffic between the NAP health policy server and health requirement servers. This traffic depends on the SHVs running on the NAP health policy server. The Windows Security Health Validator (WSHV) does not require communication with health requirement servers.


 

Joe Davies
Senior Program Manager

Categories: design, Troubleshooting Tags:

What is NAP traffic?

January 6th, 2009 No comments

Here is a question posed by a member of the NAP community:


·         What new traffic will there be on the network when I deploy NAP?


A NAP deployment can have the following additional sets of network traffic:


·         Traffic between the NAP client and the NAP enforcement point. The nature of this traffic depends on the NAP enforcement method.


o       For IPsec enforcement, the NAP client communicates to the HRA using HTTP or HTTPS to indicate its identity and health state and to receive the system health evaluation results and the health certificate.


o       For 802.1X enforcement, the NAP health evaluation is done over PEAP-TLV, resulting in a small amount of additional EAPOL traffic to send the health state and health evaluation results between the NAP client and the switch or wireless access point.


o       For VPN enforcement, the NAP health evaluation is done over PEAP-TLV, resulting in small amount of additional PPP traffic to send the health state and health evaluation results between the NAP client and the VPN server.


o       For DHCP enforcement, the NAP health evaluation is done using the same DHCP messages that are already being used for DHCP address allocation, resulting in larger payloads for some DCHP messages, but not additional messages.


o       For TS Gateway enforcement, the NAP health evaluation is done over the Remote Procedure Call (RPC) over HTTP protocol that is used for connections to a TS Gateway server, resulting in a small amount of additional traffic to send the health state from the TS Gateway client and the TS Gateway server.


·         Traffic between the NAP enforcement point and the NAP health policy server. This is RADIUS traffic, consisting of one or multiple exchanges of RADIUS request and response messages. RADIUS traffic is UDP-based and adds minimal additional traffic on your network.


·         Traffic between the NAP enforcement point and other servers. The most obvious example is the traffic between the Health Registration Authority (HRA) and an Active Directory domain controller and a certification authority (CA) to authenticate the NAP client and obtain a health certificate.


·         Traffic between the NAP health policy server and health requirement servers. This traffic depends on the SHVs running on the NAP health policy server. The Windows Security Health Validator (WSHV) does not require communication with health requirement servers.


 

Joe Davies
Senior Program Manager

Categories: design, Troubleshooting Tags:

What is NAP traffic?

January 6th, 2009 Comments off

Here is a question posed by a member of the NAP community:


·         What new traffic will there be on the network when I deploy NAP?


A NAP deployment can have the following additional sets of network traffic:


·         Traffic between the NAP client and the NAP enforcement point. The nature of this traffic depends on the NAP enforcement method.


o       For IPsec enforcement, the NAP client communicates to the HRA using HTTP or HTTPS to indicate its identity and health state and to receive the system health evaluation results and the health certificate.


o       For 802.1X enforcement, the NAP health evaluation is done over PEAP-TLV, resulting in a small amount of additional EAPOL traffic to send the health state and health evaluation results between the NAP client and the switch or wireless access point.


o       For VPN enforcement, the NAP health evaluation is done over PEAP-TLV, resulting in small amount of additional PPP traffic to send the health state and health evaluation results between the NAP client and the VPN server.


o       For DHCP enforcement, the NAP health evaluation is done using the same DHCP messages that are already being used for DHCP address allocation, resulting in larger payloads for some DCHP messages, but not additional messages.


o       For TS Gateway enforcement, the NAP health evaluation is done over the Remote Procedure Call (RPC) over HTTP protocol that is used for connections to a TS Gateway server, resulting in a small amount of additional traffic to send the health state from the TS Gateway client and the TS Gateway server.


·         Traffic between the NAP enforcement point and the NAP health policy server. This is RADIUS traffic, consisting of one or multiple exchanges of RADIUS request and response messages. RADIUS traffic is UDP-based and adds minimal additional traffic on your network.


·         Traffic between the NAP enforcement point and other servers. The most obvious example is the traffic between the Health Registration Authority (HRA) and an Active Directory domain controller and a certification authority (CA) to authenticate the NAP client and obtain a health certificate.


·         Traffic between the NAP health policy server and health requirement servers. This traffic depends on the SHVs running on the NAP health policy server. The Windows Security Health Validator (WSHV) does not require communication with health requirement servers.


 

Joe Davies
Senior Program Manager

Categories: design, Troubleshooting Tags:

Network Access Protection Troubleshooting Guide is live!

December 22nd, 2008 Comments off

Hey citizens of New NAP City!


The Network Access Protection Troubleshooting Guide, authored by our very own technical writer and NAP Forum hero Greg Lindsay, is now live! This completes the product documentation set for NAP in Windows Server 2008.


http://technet.microsoft.com/en-us/library/dd348515.aspx


The NAP Troubleshooting Guide provides provides task-oriented information to help you identify and resolve NAP-related problems quickly for the Windows Server  2008, Windows Vista, and Windows XP with Service Pack 3 operating systems. This guide can assist you when performing root-cause analysis of incidents and problems with the components of a NAP infrastructure.


This guide contains the following sections:


·         Introduction to Troubleshooting NAP


·         A-Z List of Problem Topics for NAP


·         Things to Check Before Troubleshooting NAP


·         Quick Fixes for NAP


·         Tools for Troubleshooting NAP


·         Troubleshooting NAP Problems


You can provide feedback on individual pages of the NAP Troubleshooting Guide by clicking “Click to Rate and Give Feedback” just above the content pane.


A big thank you to Greg for his authoring efforts and to NAP product team reviewers.


Enjoy!


 

Joe Davies

Categories: operations, Resources, Troubleshooting Tags:

Network Access Protection Troubleshooting Guide is live!

December 22nd, 2008 No comments

Hey citizens of New NAP City!


The Network Access Protection Troubleshooting Guide, authored by our very own technical writer and NAP Forum hero Greg Lindsay, is now live! This completes the product documentation set for NAP in Windows Server 2008.


http://technet.microsoft.com/en-us/library/dd348515.aspx


The NAP Troubleshooting Guide provides provides task-oriented information to help you identify and resolve NAP-related problems quickly for the Windows Server  2008, Windows Vista, and Windows XP with Service Pack 3 operating systems. This guide can assist you when performing root-cause analysis of incidents and problems with the components of a NAP infrastructure.


This guide contains the following sections:


·         Introduction to Troubleshooting NAP


·         A-Z List of Problem Topics for NAP


·         Things to Check Before Troubleshooting NAP


·         Quick Fixes for NAP


·         Tools for Troubleshooting NAP


·         Troubleshooting NAP Problems


You can provide feedback on individual pages of the NAP Troubleshooting Guide by clicking “Click to Rate and Give Feedback” just above the content pane.


A big thank you to Greg for his authoring efforts and to NAP product team reviewers.


Enjoy!


 

Joe Davies

Categories: operations, Resources, Troubleshooting Tags:

Network Access Protection Troubleshooting Guide is live!

December 22nd, 2008 No comments

Hey citizens of New NAP City!


The Network Access Protection Troubleshooting Guide, authored by our very own technical writer and NAP Forum hero Greg Lindsay, is now live! This completes the product documentation set for NAP in Windows Server 2008.


http://technet.microsoft.com/en-us/library/dd348515.aspx


The NAP Troubleshooting Guide provides provides task-oriented information to help you identify and resolve NAP-related problems quickly for the Windows Server  2008, Windows Vista, and Windows XP with Service Pack 3 operating systems. This guide can assist you when performing root-cause analysis of incidents and problems with the components of a NAP infrastructure.


This guide contains the following sections:


·         Introduction to Troubleshooting NAP


·         A-Z List of Problem Topics for NAP


·         Things to Check Before Troubleshooting NAP


·         Quick Fixes for NAP


·         Tools for Troubleshooting NAP


·         Troubleshooting NAP Problems


You can provide feedback on individual pages of the NAP Troubleshooting Guide by clicking “Click to Rate and Give Feedback” just above the content pane.


A big thank you to Greg for his authoring efforts and to NAP product team reviewers.


Enjoy!


 

Joe Davies

Categories: operations, Resources, Troubleshooting Tags:

Interesting problem when adding an ADFS Proxy

June 4th, 2008 No comments

I am working on a blog post (step-by-step) for the Proxy component and I ran into a problem yesterday that ran me around pretty good.  We have seen this issue or variations of it on some support cases recently, so I thought the actual problem itself would make a good post.


The problem is caused by permissions to the private key on the Client Authentication Certificate needed.  In my initial attempt to setup and document the Proxy component, I made a request to my Standalone CA for a client authentication certificate.  After approving the request, the only option from the certificate web page was to “install this certificate”.  Next, when I viewed the certificate snap-in on the proxy server, I noticed that the certificate was installed to the user store and not the computer store.  I simply did a copy paste operation from user to computer.  This appeared to work for me because when I double clicked the certificate, it looked fine.  I saw the “You have a private key” on the general tab and I assumed all was well.


When I went to test – I received a failure.  The first thing I did was run the ADFS Diagnostic tool.  I ran it on the FS-A, then copied the file to the FS-A Proxy.  I passed all tests and the tool was not finding the failure! 


Here are the Event Log and Debug Logs from my FS-A and FS-A Proxy when I attempted to access the application with the Proxy in place:


From the FS-A


 


Event Viewer:


 


Event Type:           Error


Event Source:       ADFS Federation Service


Event Category:    None


Event ID:                664


Date:                      6/3/2008


Time:                      5:13:09 PM


User:                      N/A


Computer:             ADFSACCOUNT


Description:


The Federation Service failed a privileged Web method call because Secure Sockets Layer (SSL) client authentication information was not available.


 


This event can occur if the client does not provide a client certificate or if Internet Information Services (IIS) rejects the client’s certificate because it does not chain to a trusted root certification authority in the Federation Service.


 


User Action


If this is a valid call from the Federation Service Proxy, ensure that the root of the Federation Service Proxy client certificate is trusted by the Federation Service.


 


Debug logs:


 


2008-06-03T22:13:09 [INFO] Processing HTTP POST: https://adfsaccount.adatum.com/adfs/fs/FederationServerService.asmx


2008-06-03T22:13:09 [VERBOSE] Received message that is not SignIn Request or Response.


2008-06-03T22:13:09 [ERROR] MethodInvocationCheck: Client cert is not present


2008-06-03T22:13:09 [EVENTLOG] Error ProxyWebMethodAccessDeniedNoCert ()


2008-06-03T22:13:09 [ERROR] MethodInvocationCheck: Denying access


 


 


 


From the FS-A Proxy


 


Event Viewer:


 


Event Type:           Error


Event Source:       ADFS


Event Category:    None


Event ID:                605


Date:                      6/3/2008


Time:                      5:13:09 PM


User:                      N/A


Computer:             FSA-PROXY


Description:


The Federation Service Proxy encountered an exception when it called a Federation Service Web method.


Federation Server URL: https://adfsaccount.adatum.com/adfs/fs/FederationServerService.asmx


Web method: GetProxyTrustConfiguration


Proxy certificate thumbprint: ECF1FE79E51231DF48098E1044233FCBDABF04CC


 


This may cause a user request to fail.


 


User Action


The exception details may give an indication of the precise problem.


Check network connectivity between the Federation Service Proxy and the Federation Service.


Ensure that the Federation Service is running.


Ensure that the Federation Service Proxy client authentication certificate has been added to the list of proxy authentication certificates in the Federation Service trust policy.


Ensure that the Federation Service Proxy client authentication certificate chains to a root that is trusted by the Federation Service.


Ensure that the Federation Service Internet Information Services (IIS) Secure Sockets Layer (SSL) server certificate chains to a root that is trusted by the Federation Service Proxy.


Ensure that the Federation Service Uniform Resource Locator (URL) that is configured in the Federation Service Proxy web.config uses the name that is the subject of the Federation Service IIS SSL server certificate.


 


Additional Data


Exception details:


System.Web.Services.Protocols.SoapException: Server was unable to process request. —> Attempted to perform an unauthorized operation.


   at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)


   at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)


   at System.Web.Security.SingleSignOn.FederationServerSoapProxy.GetProxyTrustConfiguration(VersionInformation proxyVersion, VersionInformation& fsVersion, ProxyInformation& proxyInformation, TrustConfigurationData[]& trustConfig)


   at System.Web.Security.SingleSignOn.LSPersistentState.GetPolicy(VersionInformation& fsVersion, ProxyInformation& proxyInformation, TrustConfigurationData[]& data)


 




 


Debug logs:


 


2008-06-03T22:13:09 [VERBOSE] Processing HTTP GET: https://adfsaccount.adatum.com/adfs/ls/?wa=wsignin1.0&wtrealm=urn:federation:treyresearch&wct=2008-06-03T22:13:09Z&wctx=https://adfsweb.treyresearch.net:8081/claimapp/\https://adfsweb.treyresearch.net:8081/claimapp/default.aspx


2008-06-03T22:13:09 [VERBOSE] Received SignIn Request.


2008-06-03T22:13:09 [ERROR] Exception from GetProxyTrustConfiguration: System.Web.Services.Protocols.SoapException: Server was unable to process request. —> Attempted to perform an unauthorized operation.


   at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)


   at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)


   at System.Web.Security.SingleSignOn.FederationServerSoapProxy.GetProxyTrustConfiguration(VersionInformation proxyVersion, VersionInformation& fsVersion, ProxyInformation& proxyInformation, TrustConfigurationData[]& trustConfig)


   at System.Web.Security.SingleSignOn.LSPersistentState.GetPolicy(VersionInformation& fsVersion, ProxyInformation& proxyInformation, TrustConfigurationData[]& data)


2008-06-03T22:13:09 [EVENTLOG] Error ExceptionFromFedServer (https://adfsaccount.adatum.com/adfs/fs/FederationServerService.asmx, GetProxyTrustConfiguration, ECF1FE79E51231DF48098E1044233FCBDABF04CC, System.Web.Services.Protocols.SoapException: Server was unable to process request. —> Attempted to perform an unauthorized operation.


   at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)


   at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)


   at System.Web.Security.SingleSignOn.FederationServerSoapProxy.GetProxyTrustConfiguration(VersionInformation proxyVersion, VersionInformation& fsVersion, ProxyInformation& proxyInformation, TrustConfigurationData[]& trustConfig)


   at System.Web.Security.SingleSignOn.LSPersistentState.GetPolicy(VersionInformation& fsVersion, ProxyInformation& proxyInformation, TrustConfigurationData[]& data))


 


As you can see, there is a problem with the client auth certificate somewhere.  I did a fair amount of double checking my steps – but everything looked correct and seemed to be checking out.  The doubt was starting to creep in – I started to wonder how much I knew about this stuff!  Then I remembered an issue that came up a few weeks ago. 


The diagnostic tool does check for the existence and proper permissions of the private key and will flag it – but it does so in the user context.  ADFS is operating under the machine context.  So when I look at the certificate or run some certutil commands against it – it all checks out because I’m in my user security context.  If I launch a CMD prompt with AT scheduler and run the same commands or run the Diagnostic tool – I find the error.  The local computer does not have permissions to the private key of the client authentication certificate.


I was able to re-issue the certificate and mark the private keys as exportable, then do an export/import operation from the user store to computer store and everything worked as expected.


Since Client Authentication certificates are commonly used for user operations vs. computer operations – it is easy to see how others could hit this very same problem.  Hopefully the errors and debug log entries will make this blog post discoverable for others hitting this.