Archive

Archive for August, 2009

How to control routing from your own routing agent

With the release of Microsoft Exchange Server 2007 Service Pack 1 you can programmatically override the default routing for message recipients on a per-recipient basis. This can be done by using the SetRoutingOverride method on the recipient for which you want to override the default routing.


Let’s assume that you have an Exchange 2007 server with just one send connector “Internet Connector”” using DNS for the address space *. With this configuration all messages leaving the organization will use this connector. Let’s further assume that you want to route messages from certain senders to a smarthost instead of using DNS. This can be done as follows:


– Create an additional send connector “Smarthost Connector” pointing to the smarthost
– Specify a non-existing domain (e.g. nexthopdomain.com) as address space of the new connector
– Write, install and enable a routing agent which registers for the OnResolvedMessage event and overwrites the default routing for the recipient.


The sample agent of this article shows you how to route messages from administrator@contoso.com over the new connector, the routing for all other sender’s won’t be changed. 


1.) Create a C# project (dll/library type) 


2.) Copy the following DLLs from the C:\Program Files\Microsoft\Exchange Server\Public directory of an Exchange 2007 server to the debug directory of your new C# project:
     a. Microsoft.Exchange.Data.Common.dll
     b. Microsoft.Exchange.Data.Transport.dll


3.) Add references to the two DLLs to the C# project using the Visual Studio solution explorer


4.) Add the following code to your project:


using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;


using Microsoft.Exchange.Data.Transport;
using Microsoft.Exchange.Data.Transport.Email;
using Microsoft.Exchange.Data.Transport.Smtp;
using Microsoft.Exchange.Data.Transport.Routing;
using Microsoft.Exchange.Data.Common;



namespace RoutingAgentOverride
{
    public class SampleRoutingAgentFactory : RoutingAgentFactory
    {
        public override RoutingAgent CreateAgent(SmtpServer server)
        {
            RoutingAgent myAgent = new ownRoutingAgent();


            return myAgent;
        }
    }
}
public class ownRoutingAgent : RoutingAgent
{
    public ownRoutingAgent()
    {
        //subscribe to different events
        base.OnResolvedMessage += new ResolvedMessageEventHandler(ownRoutingAgent_OnResolvedMessage);
    }


    void ownRoutingAgent_OnResolvedMessage(ResolvedMessageEventSource source, QueuedMessageEventArgs e)
    {
        try
        {
            // For testing purposes we do not only check the sender address but the subject line as well
            // If the subject contains the substring “REDIR” then the default routing is overwritten.
            //
            // Instead of hard-coding the sender you could also perform an LDAP-query, read the information
            // from a text file, etc.
            //
            if (e.MailItem.FromAddress.ToString() == “administrator@contoso.com
                && e.MailItem.Message.Subject.Contains(“REDIR”))
            {


                // Here we set the address space we want to use for the next hop. Note that this doesn’t change the recipient address.
                // Setting the routing domain to “nexthopdomain.com” only means that the routing engine chooses a suitable connector
                // for nexthopdomain.com instead of using the recpient’s domain.


                RoutingDomain myRoutingOverride = new RoutingDomain(“nexthopdomain.com”); 
                                                                                         
                foreach (EnvelopeRecipient recp in e.MailItem.Recipients)
                {
                    recp.SetRoutingOverride(myRoutingOverride);


                }


            }
        }


        catch // (Exception except)
        {


        }
    }


}


5.) Compile the DLL


6.) Copy the DLL to the HT server


7.) Install the transport agent using the Exchange Management Shell:
     Install-TransportAgent “OwnTestAgent” -TransportAgentFactory “RoutingAgentOverride.SampleRoutingAgentFactory” -AssemblyPath “Path to DLL”


8.) Enable the transport agent using the Exchange Management Shell:
     Enable-TransportAgent “OwnTestAgent”


9.) IMPORTANT: Exit Powershell
10. IMPORTANT: Restart the MSExchangeTransport service


11.) Verify that the agent was successfully enabled / registered by running Get-Transportagent


Tips:



  • To live debug the dll you need to attach to edgetransport.exe.

  • To recompile after a change


    • detach from edgetransport

    • stop the transport service (otherwise Visual studio won’t be able to overwrite the file)
      (only if you registered the dll from the \debug folder)

  • Sometimes messages remain in the outbox until the mail submission service is restarted

Released : Windows SDK for Windows 7

August 24th, 2009 No comments

Hi


The RTM release of SDK samples for Windows 7 is now available for download. The samples can be downloaded in ISO format  or  Web Setup format. This release contains some updates to the NAP SDK samples over the Win7 SDK samples RC release.


Thanks


Ravi


 

Categories: Uncategorized Tags:

Released : Windows SDK for Windows 7

August 24th, 2009 No comments

Hi


The RTM release of SDK samples for Windows 7 is now available for download. The samples can be downloaded in ISO format  or  Web Setup format. This release contains some updates to the NAP SDK samples over the Win7 SDK samples RC release.


Thanks


Ravi


 

Categories: Uncategorized Tags:

Released : Windows SDK for Windows 7

August 24th, 2009 No comments

Hi


The RTM release of SDK samples for Windows 7 is now available for download. The samples can be downloaded in ISO format  or  Web Setup format. This release contains some updates to the NAP SDK samples over the Win7 SDK samples RC release.


Thanks


Ravi


 

Categories: Uncategorized Tags:

Released : Windows SDK for Windows 7

August 24th, 2009 No comments

Hi


The RTM release of SDK samples for Windows 7 is now available for download. The samples can be downloaded in ISO format  or  Web Setup format. This release contains some updates to the NAP SDK samples over the Win7 SDK samples RC release.


Thanks


Ravi


 

Categories: Uncategorized Tags:

Released : Windows SDK for Windows 7

August 24th, 2009 Comments off

Hi


The RTM release of SDK samples for Windows 7 is now available for download. The samples can be downloaded in ISO format  or  Web Setup format. This release contains some updates to the NAP SDK samples over the Win7 SDK samples RC release.


Thanks


Ravi


 

Categories: Uncategorized Tags:

Released : Windows SDK for Windows 7

August 24th, 2009 No comments

Hi

The RTM release of SDK samples for Windows 7 is now available for download. The samples can be downloaded in ISO format  or  Web Setup format. This release contains some updates to the NAP SDK samples over the Win7 SDK samples RC release.

Thanks

Ravi

 

Categories: Uncategorized Tags:

Windows Home Server category grows with the announcement of the Lenovo IdeaCentre D400

August 22nd, 2009 No comments

As reported by COMPUTERWORLD and other sources (see below) earlier this week, Lenovo will be introducing the IdeaCentre D400 to the US next month.  Previously launched in China, Lenovo’s offering will be a great addition to the category of home servers powered by Windows Home Server software.

Announced in conjunction with upcoming Q Series of compact home entertainment PCs, the IdeaCentre D400 home server has an aesthetic design that would suit the living room as well as the home office.  (See Lenovo’s press release for details.)

lenovoserver-9136792

Photo: Lenovo IdeaCentre D400 home server

Eugene Saburi, General Manager, Microsoft Solutions Marketing had this to say. "With the IdeaCentre D400 home server coinciding with the availability of Windows 7 on Lenovo laptops and netbooks, consumers will have affordable solution that enables them to unleash the power of their home network.”

With the Acer Aspire easystore and the lineup of HP MediaSmart Servers already on the market, the Lenovo IdeaCentre will provide even more choice for consumers.  We are still waiting on confirmation of where the IdeaCentre D400 home server will be sold.  Let’s hope pre-orders are offered soon!

More articles on the Lenovo announcement:

Lenovo unveils Windows Home Server, tiny home theater PCs
Seattle Times
The company also rolled out several new tiny PCs with home theater capabilities — a sample of its Windows 7 wave lineup, out in time for back-to-school …

Lenovo Q100/Q700 nettops, D400 home server debut
Electronista
Lenovo this afternoon put an end to speculation and launched its IdeaCentre Q series of mini desktops as well as its first Windows Home Server. ..

Lenovo to Introduce IdeaCentre D400 Windows Home Server
Notebooks.com
By Xavier | August 17, 2009 Lenovo’s getting ready to announce the IdeaCentre D400 Windows Home Server. The company posted a teaser image in a twitter post …

Lenovo Reveal Their First Home Server – The IdeaCenter D400
By Terry Walsh
Lenovo release a teaser shot of their debut Windows Home Server, but we’ve uncovered a whole host of pics courtesy of the Interwebs. Check ’em out!

– Steven

Windows Home Server Software Development Kit (SDK) adds Japanese Language Support

August 22nd, 2009 No comments

The momentum behind Windows Home Server add-ins continues to grow!  The Windows Home Server team is very pleased to announce the availability of the Japanese Software Development Kit (SDK).  The SDK provides developers with guidance, for example application programming interface (API) information, on how to create programs that integrate with the Windows Home Server Console. 

Japanese developers now have access to the localized version of the SDK on the Windows Home Server section of the MSDN website.

image

Coinciding with the release of the Japanese SDK, the Windows Home Server Japanese Evaluation software is now available for download

For users who have not stayed abreast of the Windows Home Server add-ins community, you may want to visit Terry Walsh’s site www.whsplus.com.  You’ll find a pretty comprehensive list of Windows Home Server add-ins, reviews of the latest add-ins, as well as a list of user-ranked top add-ins. 

image

– Steven

Categories: community, developers, Evaluation, Platform, SDK Tags:

CDO & usage of the meetingitem respond method

We recently had to troubleshoot an issue with CDO responses to meeting updates.


It turns out that MS09-003 contained a code change from KB 913579.


913579 An update to a recurring meeting request that is sent from Exchange Server 2003 does not occur on the receiver side
http://support.microsoft.com/default.aspx?scid=kb;EN-US;913579


All Exchange 2003 Hot-fixes are cumulative!


The purpose of the code change was a stricter enforcing of RFC compliance for iCalendar messages.


The property written to urn:schemas:calendar:sequence is now written to the dispidApptSequence named property used by Outlook if it is not present.
This occurs during local delivery and content conversion of the message.


This caused problems in the logic for the CDO Application in that, when calling the “respond” function on the Meeting Update message, the dispidApptSequence  number was not being copied to the  meeting response, and the urn:schemas:calendar:sequence was already set with a value of “0”.
Outlook does not use the urn:schemas:calendar:version property.


The Exchange Server sees the messages with only the 1 property set, and overwrites the one used by Outlook with “0”.


This means that when calling the respond method on meeting updates / requests, the sequence number must be read from the request and written to the response, otherwise Outlook will not be able to match the response to the original meeting.
(The proposed solution is to read the dispidApptSequence from the original message and write this to the meeting update response before sending.)


This requirement to keep the sequence number updated is actually listed in our MAPI protocol documentation, however it is not automatically enforced by CDO.


See: http://msdn.microsoft.com/en-us/library/ee218725.aspx


Here is the sample VB-Script used to verify the code change:


(This code is provided “AS IS” with no warranties, and confers no rights. Use of included code samples are subject to the terms specified in the Terms of Use.)


<script>
Dim sess,spMeetingResponse ,spAppointment ,spMeetingItem ,pMessages ,pFolder
const PidLidAppointmentSequence = “0x8201”
const CdoPropSetID1 = “0220060000000000C000000000000046”
 
Set sess = CreateObject(“MAPI.Session”)
sess.Logon
 
Set pFolder = sess.Inbox
Set pMessages = pFolder.Messages
Set spMeetingItem = pMessages.GetLast()
Set spAppointment = spMeetingItem.GetAssociatedAppointment()
 
spAppointment.Update()
 
iSeqNum=spMeetingItem.Fields.Item(“{” & CdoPropSetID1 & “}” & PidLidAppointmentSequence  ).Value
 
 
Set spMeetingResponse = spMeetingItem.Respond(3)
bstrSubject = spMeetingResponse.Subject
spMeetingResponse.Subject = “Accepted: ” & bstrSubject
spMeetingResponse.Fields.Add PidLidAppointmentSequence,3,iSeqNum,CdoPropSetID1
spMeetingResponse.Send
 
sess.Logoff
 
Set spMeetingResponse = Nothing
Set spAppointment = Nothing
Set spMeetingItem = NOthing
Set pMessages = nothing
Set pFolder = NOthing
Set sess = Nothing
MsgBox “Done.”
</script>

Categories: CDO, recurring meeting Tags:

Exchange 2007 SPAM & SPF

If you are trying to configure Exchange 2007 SPF and are seeing “TempError” for mails spoofed for your own domains, it is becuase Exchange 2007 uses the same query code for SPF as it does for other SMTP DNS queries.


This leads to it thinking that if it matches it’s own IP to an MX record query, this indicates a loop, and removes these entries from it’s results, leading to the TempError in the SPF query.


Example SMTP SPAM message header:


—————————————————————————————————-


Received: from 10.10.10.11 (10.10.10.11) by exchange.contoso.com


 (10.10.10.12 with Microsoft SMTP Server id 8.1.375.2; Thu, 13 Aug 2009


 13:03:34 +0200


From: “? VIAGRA ? Official Site” <ontheroad@contoso.com>


To: <Jane@contoso.com>


Subject: Dear admin@contoso.com72% 0FF on Pfizer !


MIME-Version: 1.0


Content-Type: text/html; charset=”ISO-8859-1″


Content-Transfer-Encoding: 7bit


Message-ID: <12345678-abcd-abcd-1234-1234567890ab@exchange.contoso.com>


Return-Path: ontheroad@contoso.com


Date: Thu, 13 Aug 2009 13:03:34 +0200


X-MS-Exchange-Organization-PRD: contoso.com


X-MS-Exchange-Organization-SenderIdResult: TempError


Received-SPF: TempError (exchange.contoso.com: error in processing during


 lookup of ontheroad@contoso.com: )


X-MS-Exchange-Organization-Antispam-Report: SenderOnRecipientSafeList


X-MS-Exchange-Organization-SCL: -1


—————————————————————————————————- 


If you have safe lists (in this example, for a mobile user ontheroad@contoso.com), this can override other SPAM  protection mechanisms, meaning that someone spoofing an address which you have listed in your safe list will get through to your mailboxes.


The best way to prevent this is not to use MX records in conjunction with Exchange 2007 and SPF entries.


Rather replace the MX records with the actual SMTP server IP Addresses which are responsible for your external delivery.

Proxying CAS HTTP Cross Forest availability requests

Some of you may want to configure Cross Forest availability with your CAS Servers, but don’t necessarily want to open additional paths through your firewalls or networks for the CAS Servers to be able to talk directly with one another.


 


In this case, if you have an HTTP Web Proxy (obviously we like to use ISA Server), you can configure the CAS Server to use this to proxy it’s SOAP requests for Free Busy and Autodiscover to the other forest.


 


Locate the following folders (actual path may vary if you changed the default install path of Exchange):


 


..\Program Files\Microsoft\Exchange Server\ClientAccess\exchweb\ews


..\Program Files\Microsoft\Exchange Server\ClientAccess\Autodiscover


 


You will find a web.config file in each of these, to which you will need to add the following section (to clarify, it is best placed directly after /<system.web>):


 


    <system.net>
        <defaultProxy>
            <proxy
                usesystemdefault = “false”
                proxyaddress = “http://proxyserver/
                bypassonlocal = “true”
            />
            <bypasslist>
                <add address=”http://[a-z]+/.contoso/.com$” />
            </bypasslist>
        </defaultProxy>
    </system.net>


 

For more information, check out the following MSDN article: http://msdn.microsoft.com/en-us/library/5w91x7a7.aspx

 


Important parameters are:


 


Usesystemdefault -> set to false to be able to specify our own proxy settings


Proxyaddress -> Your HTTP Proxy (ISA Server)


Bypassonlocal -> This setting is important if you have more than 1 CAS Server internally, and if your Proxy Server can not proxy internal HTTP requests.


Bypasslist -> The list of addresses which should be bypassed as local addresses.


 


Following the changes in the web.config files, you should perform an IISRESET.


 


You CAS Server will now use your HTTP Proxy for HTTP requests such as Cross Forest Free busy lookups, Cross Forest Autodiscover etc.


If you did not configure bypass lists, it will be using the HTTP Proxy for all requests.


 


Please note that this is only for HTTPWebRequests (http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.aspx) that the CAS Server initiates (such as those SOAP Free Busy requests to other servers) rather than responses to OWA Clients.

Microsoft Security Advisory (973472): Vulnerability in Microsoft Office Web Components Control Could Allow Remote Code Execution – Version: 2.0

Revision Note: V2.0 (August 11, 2009): Advisory updated to reflect publication of security bulletin.
Summary: Microsoft has completed the investigation of a privately reported vulnerability in Microsoft Office Web Components. We have issued MS09-043 to address this issue. For more information about this issue, including download links for an available security update, please review MS09-043. The vulnerability addressed is the Office Web Components HTML Script Vulnerability – CVE-2009-1136.

Categories: Uncategorized Tags:

Microsoft Security Advisory (973472): Vulnerability in Microsoft Office Web Components Control Could Allow Remote Code Execution – 8/11/2009

August 11th, 2009 Comments off

Revision Note: V2.0 (August 11, 2009): Advisory updated to reflect publication of security bulletin. Advisory Summary:Microsoft has completed the investigation of a privately reported vulnerability in Microsoft Office Web Components. We have issued MS09-043 to address this issue. For more information about this issue, including download links for an available security update, please review MS09-043. The vulnerability addressed is the Office Web Components HTML Script Vulnerability – CVE-2009-1136.

Categories: Uncategorized Tags:

Error “404 Not Found” when using PFDavAdmin against Exchange 2007

 

Sometimes when using PFDavAdmin to view folder properties with Exchange 2007 you may see the following error in the tool:


 


The remote server returned an error: (404) Not Found


 


If you look at the IIS logs on the target machine, you will see the error was “404.11”.


This is due to enhanced security settings that can be set in IIS, and are set by default in IIS7 (Windows 2008).


 


The solution is described in the following article:


 


942076  Error message when you visit a Web site that is hosted on IIS 7.0: “HTTP Error 404.11 – URL_DOUBLE_ESCAPED”


http://support.microsoft.com/default.aspx?scid=kb;EN-US;942076


 


I found that the change in settings only worked after I had used APPCMD in the following way (on my W2K8 MBX server):


Under the following folder:


 


%windir%\system32\inetsrv


 


Run:


 


Appcmd set config “Default Web Site” /section:system.webServer/Security/requestFiltering -allowDoubleEscaping:True


 


If you are still having problems, seeing different errors in the IIS logs, or see different symptoms, you probably have a different issue.

Auditing system impact on performance

August 10th, 2009 No comments

UPDATE 2010-06-06 (EricF) – Fixed Vista+ architecture image; link was broken on migration to new blog platform

I get questions from time to time, such as my recent offline question from Steve, about what performance impact auditing has on the system as a whole.

To answer this you need to understand a couple of things:

  1. Auditable activity is implemented as instrumentation (e.g. a function call to the auditing system) inside the code that does something auditable.
  2. The auditing system in Windows has two sets of programmatic interfaces for introducing an event, one in kernel mode and one in user mode – so the component generating audit does not need to switch between kernel and user modes.
  3. Although audit policy is stored in user mode, we cache a copy of the relevant policy for kernel-mode components, in kernel mode.  This means that no mode switch is necessary to check audit policy to decide whether to generate an event.
  4. There are user-mode and kernel-mode queues for audit events.  The call to generate an audit event actually just queues the event, assuming the queue is not full.  So from the perspective of the component generating audit, audit has an “asynchronous” flavor under light-medium loads.  Under heavy loads when the queues fill, audit blocks the component raising the audit until the event can be queued, showing its true synchronous behavior.
  5. Dequeuing audit events always occurs on a separate thread than enqueueing so that raising audit events and writing them to the log don’t affect each other’s perf under light to moderate load.
  6. The pre-Vista auditing system in the kernel delivers events to LSA.  The Vista+ auditing system in the kernel delivers most events directly to ETW, the kernel mode event trace engine, which means that most of the kernel audit (including the potentially perf impacting object access events) doesn’t require a mode switch at all.
  7. The LSA formats events and then delivers them to the event log.  In WS03, events are batched in the RPC call to eventlog.  In Vista+, delivery is done by means of ETW in almost all cases.
  8. ETW queues events and spools them to the Windows eventlog service as fast as the service will accept them.
  9. The eventlog service writes the events to the log file as they arrive.

I have uploaded graphics of the Windows XP/Windows Server 2003 auditing architecture, and the Windows Vista/WS08/Windows 7 architecture, to make this process more clear:

Pre-Vista Windows Auditing Architecture

 

Windows Vista+ Auditing Architecture

 

So now back to the original question- what is the impact of auditing on performance?

At low auditing loads, auditing generally has no discernable impact on perf.  If you were hardcode with a profiler and iterated an auditable activity a million times I am sure you’d be able to measure it, but for reasonable audit policies you won’t notice a significant difference.

At high auditing loads, auditing has a significant performance impact.  This is more true of pre-Vista multiprocessor systems than of systems with the new eventlog system.

For example, a multi-processor domain controller (say a 32-processor box) running Windows Server 2003, might run into problems under extreme load.  Why is that?  Because ultimately the limiting factor on event rate is how fast you can write the events to disk.  Pre-Vista eventlog has a single thread writing events to disk.  So even though you might have 32 threads servicing authentication requests (an auditable activity), each of them is queueing to a single audit queue which is ultimately despooling to eventlog via RPC on a single thread, and eventlog is only writing to the security log with a single thread.  What we observe in practice in this case is that a single processor on the system goes to 85-100% utilization, and the other processors drop to a very low utilization as the authentication threads are blocked waiting for the audit function call to return.  This call won’t return until the queue is not full, and the queue is waiting on RPC which is waiting on eventlog…  so eventlog governs the rate.

In Windows Server 2003, we added a particular optimization only for the security event log, which batches events in the RPC call to eventlog.  This means that you can get more event throughput in the security log than in other logs on the system.  It didn’t eliminate the bottleneck, but it pushed back the limit, so WS03 on typical hardware should be able to log several thousand events per second to the security event log.  Previous versions were only able to log about 1000 events per second.

Note that the change in performance characteristics occurs all at once.  So the impact tends to be trivial until the queues fill, at which point the impact is severe.  It does not scale linearly, there’s a discrete behavior change.  What this means realistically is that if you ever encounter a performance problem with auditing, then you probably just need to turn it down a little and you won’t have a problem any more. 

In Vista and subsequent releases, audit queues events via ETW.  ETW was designed for high-performance kernel tracing, and in the auditing team we tested it to over 10,000 (10.000 for you folks in Germany 🙂 events per second before we decided that we had hit our scale targets.  We never tested exactly how high it would go, but we were satisfied that the eventlog service was no longer a bottleneck in realistic scenarios.

There are some edge cases where you might run into performance problems by trying to audit too much in a critical path.  For instance, it is a really really bad idea to put SACLs on your entire registry.  If you monitor registry activity with a tool like Process Monitor, you will notice that when a system is not idle, there are often hundreds or thousands of registry accesses per second.  If you impose an auditing tax on each of those activities you will notice a degradation in performance.  Not to mention that the resultant mountain of events is probably not very valuable.  Of course you can tune SACLs as I have mentioned before, but I doubt that it’s useful to take the time to tune SACLs for the entire registry.

One last point is that the eventlog is writing the events somewhere.  Wherever it is writing events, it is consuming disk I/Os and competing with anything else writing to the same volume.  If you have a disk performance problem on that disk, it can result in an auditing performance problem, as everything else will back up if the eventlog can’t write events to disk fast enough.  So one thing you can do is ensure that the disk where your log is placed has enough I/Os.

In summary audit has very minimal impact unless you do a whole lot of it, in which case it can have severe impact on your system.  The change happens suddenly, not gradually, so you can do a lot of auditing with no problem.  If you run into a problem, turn it down just a little (or little by little) and at some point the behavior will change such that you won’t have any significant perf impact anymore.

Categories: Tips Tags:

Auditing system impact on performance

August 10th, 2009 No comments

UPDATE 2010-06-06 (EricF) – Fixed Vista+ architecture image; link was broken on migration to new blog platform

I get questions from time to time, such as my recent offline question from Steve, about what performance impact auditing has on the system as a whole.

To answer this you need to understand a couple of things:

  1. Auditable activity is implemented as instrumentation (e.g. a function call to the auditing system) inside the code that does something auditable.
  2. The auditing system in Windows has two sets of programmatic interfaces for introducing an event, one in kernel mode and one in user mode – so the component generating audit does not need to switch between kernel and user modes.
  3. Although audit policy is stored in user mode, we cache a copy of the relevant policy for kernel-mode components, in kernel mode.  This means that no mode switch is necessary to check audit policy to decide whether to generate an event.
  4. There are user-mode and kernel-mode queues for audit events.  The call to generate an audit event actually just queues the event, assuming the queue is not full.  So from the perspective of the component generating audit, audit has an “asynchronous” flavor under light-medium loads.  Under heavy loads when the queues fill, audit blocks the component raising the audit until the event can be queued, showing its true synchronous behavior.
  5. Dequeuing audit events always occurs on a separate thread than enqueueing so that raising audit events and writing them to the log don’t affect each other’s perf under light to moderate load.
  6. The pre-Vista auditing system in the kernel delivers events to LSA.  The Vista+ auditing system in the kernel delivers most events directly to ETW, the kernel mode event trace engine, which means that most of the kernel audit (including the potentially perf impacting object access events) doesn’t require a mode switch at all.
  7. The LSA formats events and then delivers them to the event log.  In WS03, events are batched in the RPC call to eventlog.  In Vista+, delivery is done by means of ETW in almost all cases.
  8. ETW queues events and spools them to the Windows eventlog service as fast as the service will accept them.
  9. The eventlog service writes the events to the log file as they arrive.

I have uploaded graphics of the Windows XP/Windows Server 2003 auditing architecture, and the Windows Vista/WS08/Windows 7 architecture, to make this process more clear:

Pre-Vista Windows Auditing Architecture

 

Windows Vista+ Auditing Architecture

 

So now back to the original question- what is the impact of auditing on performance?

At low auditing loads, auditing generally has no discernable impact on perf.  If you were hardcode with a profiler and iterated an auditable activity a million times I am sure you’d be able to measure it, but for reasonable audit policies you won’t notice a significant difference.

At high auditing loads, auditing has a significant performance impact.  This is more true of pre-Vista multiprocessor systems than of systems with the new eventlog system.

For example, a multi-processor domain controller (say a 32-processor box) running Windows Server 2003, might run into problems under extreme load.  Why is that?  Because ultimately the limiting factor on event rate is how fast you can write the events to disk.  Pre-Vista eventlog has a single thread writing events to disk.  So even though you might have 32 threads servicing authentication requests (an auditable activity), each of them is queueing to a single audit queue which is ultimately despooling to eventlog via RPC on a single thread, and eventlog is only writing to the security log with a single thread.  What we observe in practice in this case is that a single processor on the system goes to 85-100% utilization, and the other processors drop to a very low utilization as the authentication threads are blocked waiting for the audit function call to return.  This call won’t return until the queue is not full, and the queue is waiting on RPC which is waiting on eventlog…  so eventlog governs the rate.

In Windows Server 2003, we added a particular optimization only for the security event log, which batches events in the RPC call to eventlog.  This means that you can get more event throughput in the security log than in other logs on the system.  It didn’t eliminate the bottleneck, but it pushed back the limit, so WS03 on typical hardware should be able to log several thousand events per second to the security event log.  Previous versions were only able to log about 1000 events per second.

Note that the change in performance characteristics occurs all at once.  So the impact tends to be trivial until the queues fill, at which point the impact is severe.  It does not scale linearly, there’s a discrete behavior change.  What this means realistically is that if you ever encounter a performance problem with auditing, then you probably just need to turn it down a little and you won’t have a problem any more. 

In Vista and subsequent releases, audit queues events via ETW.  ETW was designed for high-performance kernel tracing, and in the auditing team we tested it to over 10,000 (10.000 for you folks in Germany 🙂 events per second before we decided that we had hit our scale targets.  We never tested exactly how high it would go, but we were satisfied that the eventlog service was no longer a bottleneck in realistic scenarios.

There are some edge cases where you might run into performance problems by trying to audit too much in a critical path.  For instance, it is a really really bad idea to put SACLs on your entire registry.  If you monitor registry activity with a tool like Process Monitor, you will notice that when a system is not idle, there are often hundreds or thousands of registry accesses per second.  If you impose an auditing tax on each of those activities you will notice a degradation in performance.  Not to mention that the resultant mountain of events is probably not very valuable.  Of course you can tune SACLs as I have mentioned before, but I doubt that it’s useful to take the time to tune SACLs for the entire registry.

One last point is that the eventlog is writing the events somewhere.  Wherever it is writing events, it is consuming disk I/Os and competing with anything else writing to the same volume.  If you have a disk performance problem on that disk, it can result in an auditing performance problem, as everything else will back up if the eventlog can’t write events to disk fast enough.  So one thing you can do is ensure that the disk where your log is placed has enough I/Os.

In summary audit has very minimal impact unless you do a whole lot of it, in which case it can have severe impact on your system.  The change happens suddenly, not gradually, so you can do a lot of auditing with no problem.  If you run into a problem, turn it down just a little (or little by little) and at some point the behavior will change such that you won’t have any significant perf impact anymore.

Categories: Tips Tags:

Auditing system impact on performance

August 10th, 2009 Comments off

UPDATE 2010-06-06 (EricF) – Fixed Vista+ architecture image; link was broken on migration to new blog platform

I get questions from time to time, such as my recent offline question from Steve, about what performance impact auditing has on the system as a whole.

To answer this you need to understand a couple of things:

  1. Auditable activity is implemented as instrumentation (e.g. a function call to the auditing system) inside the code that does something auditable.
  2. The auditing system in Windows has two sets of programmatic interfaces for introducing an event, one in kernel mode and one in user mode – so the component generating audit does not need to switch between kernel and user modes.
  3. Although audit policy is stored in user mode, we cache a copy of the relevant policy for kernel-mode components, in kernel mode.  This means that no mode switch is necessary to check audit policy to decide whether to generate an event.
  4. There are user-mode and kernel-mode queues for audit events.  The call to generate an audit event actually just queues the event, assuming the queue is not full.  So from the perspective of the component generating audit, audit has an “asynchronous” flavor under light-medium loads.  Under heavy loads when the queues fill, audit blocks the component raising the audit until the event can be queued, showing its true synchronous behavior.
  5. Dequeuing audit events always occurs on a separate thread than enqueueing so that raising audit events and writing them to the log don’t affect each other’s perf under light to moderate load.
  6. The pre-Vista auditing system in the kernel delivers events to LSA.  The Vista+ auditing system in the kernel delivers most events directly to ETW, the kernel mode event trace engine, which means that most of the kernel audit (including the potentially perf impacting object access events) doesn’t require a mode switch at all.
  7. The LSA formats events and then delivers them to the event log.  In WS03, events are batched in the RPC call to eventlog.  In Vista+, delivery is done by means of ETW in almost all cases.
  8. ETW queues events and spools them to the Windows eventlog service as fast as the service will accept them.
  9. The eventlog service writes the events to the log file as they arrive.

I have uploaded graphics of the Windows XP/Windows Server 2003 auditing architecture, and the Windows Vista/WS08/Windows 7 architecture, to make this process more clear:

Pre-Vista Windows Auditing Architecture

 

Windows Vista+ Auditing Architecture

 

So now back to the original question- what is the impact of auditing on performance?

At low auditing loads, auditing generally has no discernable impact on perf.  If you were hardcode with a profiler and iterated an auditable activity a million times I am sure you’d be able to measure it, but for reasonable audit policies you won’t notice a significant difference.

At high auditing loads, auditing has a significant performance impact.  This is more true of pre-Vista multiprocessor systems than of systems with the new eventlog system.

For example, a multi-processor domain controller (say a 32-processor box) running Windows Server 2003, might run into problems under extreme load.  Why is that?  Because ultimately the limiting factor on event rate is how fast you can write the events to disk.  Pre-Vista eventlog has a single thread writing events to disk.  So even though you might have 32 threads servicing authentication requests (an auditable activity), each of them is queueing to a single audit queue which is ultimately despooling to eventlog via RPC on a single thread, and eventlog is only writing to the security log with a single thread.  What we observe in practice in this case is that a single processor on the system goes to 85-100% utilization, and the other processors drop to a very low utilization as the authentication threads are blocked waiting for the audit function call to return.  This call won’t return until the queue is not full, and the queue is waiting on RPC which is waiting on eventlog…  so eventlog governs the rate.

In Windows Server 2003, we added a particular optimization only for the security event log, which batches events in the RPC call to eventlog.  This means that you can get more event throughput in the security log than in other logs on the system.  It didn’t eliminate the bottleneck, but it pushed back the limit, so WS03 on typical hardware should be able to log several thousand events per second to the security event log.  Previous versions were only able to log about 1000 events per second.

Note that the change in performance characteristics occurs all at once.  So the impact tends to be trivial until the queues fill, at which point the impact is severe.  It does not scale linearly, there’s a discrete behavior change.  What this means realistically is that if you ever encounter a performance problem with auditing, then you probably just need to turn it down a little and you won’t have a problem any more. 

In Vista and subsequent releases, audit queues events via ETW.  ETW was designed for high-performance kernel tracing, and in the auditing team we tested it to over 10,000 (10.000 for you folks in Germany 🙂 events per second before we decided that we had hit our scale targets.  We never tested exactly how high it would go, but we were satisfied that the eventlog service was no longer a bottleneck in realistic scenarios.

There are some edge cases where you might run into performance problems by trying to audit too much in a critical path.  For instance, it is a really really bad idea to put SACLs on your entire registry.  If you monitor registry activity with a tool like Process Monitor, you will notice that when a system is not idle, there are often hundreds or thousands of registry accesses per second.  If you impose an auditing tax on each of those activities you will notice a degradation in performance.  Not to mention that the resultant mountain of events is probably not very valuable.  Of course you can tune SACLs as I have mentioned before, but I doubt that it’s useful to take the time to tune SACLs for the entire registry.

One last point is that the eventlog is writing the events somewhere.  Wherever it is writing events, it is consuming disk I/Os and competing with anything else writing to the same volume.  If you have a disk performance problem on that disk, it can result in an auditing performance problem, as everything else will back up if the eventlog can’t write events to disk fast enough.  So one thing you can do is ensure that the disk where your log is placed has enough I/Os.

In summary audit has very minimal impact unless you do a whole lot of it, in which case it can have severe impact on your system.  The change happens suddenly, not gradually, so you can do a lot of auditing with no problem.  If you run into a problem, turn it down just a little (or little by little) and at some point the behavior will change such that you won’t have any significant perf impact anymore.

Categories: Tips Tags:

Auditing system impact on performance

August 10th, 2009 No comments

UPDATE 2010-06-06 (EricF) – Fixed Vista+ architecture image; link was broken on migration to new blog platform

I get questions from time to time, such as my recent offline question from Steve, about what performance impact auditing has on the system as a whole.

To answer this you need to understand a couple of things:

  1. Auditable activity is implemented as instrumentation (e.g. a function call to the auditing system) inside the code that does something auditable.
  2. The auditing system in Windows has two sets of programmatic interfaces for introducing an event, one in kernel mode and one in user mode – so the component generating audit does not need to switch between kernel and user modes.
  3. Although audit policy is stored in user mode, we cache a copy of the relevant policy for kernel-mode components, in kernel mode.  This means that no mode switch is necessary to check audit policy to decide whether to generate an event.
  4. There are user-mode and kernel-mode queues for audit events.  The call to generate an audit event actually just queues the event, assuming the queue is not full.  So from the perspective of the component generating audit, audit has an “asynchronous” flavor under light-medium loads.  Under heavy loads when the queues fill, audit blocks the component raising the audit until the event can be queued, showing its true synchronous behavior.
  5. Dequeuing audit events always occurs on a separate thread than enqueueing so that raising audit events and writing them to the log don’t affect each other’s perf under light to moderate load.
  6. The pre-Vista auditing system in the kernel delivers events to LSA.  The Vista+ auditing system in the kernel delivers most events directly to ETW, the kernel mode event trace engine, which means that most of the kernel audit (including the potentially perf impacting object access events) doesn’t require a mode switch at all.
  7. The LSA formats events and then delivers them to the event log.  In WS03, events are batched in the RPC call to eventlog.  In Vista+, delivery is done by means of ETW in almost all cases.
  8. ETW queues events and spools them to the Windows eventlog service as fast as the service will accept them.
  9. The eventlog service writes the events to the log file as they arrive.

I have uploaded graphics of the Windows XP/Windows Server 2003 auditing architecture, and the Windows Vista/WS08/Windows 7 architecture, to make this process more clear:

Pre-Vista Windows Auditing Architecture

 

Windows Vista+ Auditing Architecture

 

So now back to the original question- what is the impact of auditing on performance?

At low auditing loads, auditing generally has no discernable impact on perf.  If you were hardcode with a profiler and iterated an auditable activity a million times I am sure you’d be able to measure it, but for reasonable audit policies you won’t notice a significant difference.

At high auditing loads, auditing has a significant performance impact.  This is more true of pre-Vista multiprocessor systems than of systems with the new eventlog system.

For example, a multi-processor domain controller (say a 32-processor box) running Windows Server 2003, might run into problems under extreme load.  Why is that?  Because ultimately the limiting factor on event rate is how fast you can write the events to disk.  Pre-Vista eventlog has a single thread writing events to disk.  So even though you might have 32 threads servicing authentication requests (an auditable activity), each of them is queueing to a single audit queue which is ultimately despooling to eventlog via RPC on a single thread, and eventlog is only writing to the security log with a single thread.  What we observe in practice in this case is that a single processor on the system goes to 85-100% utilization, and the other processors drop to a very low utilization as the authentication threads are blocked waiting for the audit function call to return.  This call won’t return until the queue is not full, and the queue is waiting on RPC which is waiting on eventlog…  so eventlog governs the rate.

In Windows Server 2003, we added a particular optimization only for the security event log, which batches events in the RPC call to eventlog.  This means that you can get more event throughput in the security log than in other logs on the system.  It didn’t eliminate the bottleneck, but it pushed back the limit, so WS03 on typical hardware should be able to log several thousand events per second to the security event log.  Previous versions were only able to log about 1000 events per second.

Note that the change in performance characteristics occurs all at once.  So the impact tends to be trivial until the queues fill, at which point the impact is severe.  It does not scale linearly, there’s a discrete behavior change.  What this means realistically is that if you ever encounter a performance problem with auditing, then you probably just need to turn it down a little and you won’t have a problem any more. 

In Vista and subsequent releases, audit queues events via ETW.  ETW was designed for high-performance kernel tracing, and in the auditing team we tested it to over 10,000 (10.000 for you folks in Germany :-) events per second before we decided that we had hit our scale targets.  We never tested exactly how high it would go, but we were satisfied that the eventlog service was no longer a bottleneck in realistic scenarios.

There are some edge cases where you might run into performance problems by trying to audit too much in a critical path.  For instance, it is a really really bad idea to put SACLs on your entire registry.  If you monitor registry activity with a tool like Process Monitor, you will notice that when a system is not idle, there are often hundreds or thousands of registry accesses per second.  If you impose an auditing tax on each of those activities you will notice a degradation in performance.  Not to mention that the resultant mountain of events is probably not very valuable.  Of course you can tune SACLs as I have mentioned before, but I doubt that it’s useful to take the time to tune SACLs for the entire registry.

One last point is that the eventlog is writing the events somewhere.  Wherever it is writing events, it is consuming disk I/Os and competing with anything else writing to the same volume.  If you have a disk performance problem on that disk, it can result in an auditing performance problem, as everything else will back up if the eventlog can’t write events to disk fast enough.  So one thing you can do is ensure that the disk where your log is placed has enough I/Os.

In summary audit has very minimal impact unless you do a whole lot of it, in which case it can have severe impact on your system.  The change happens suddenly, not gradually, so you can do a lot of auditing with no problem.  If you run into a problem, turn it down just a little (or little by little) and at some point the behavior will change such that you won’t have any significant perf impact anymore.

Categories: Tips Tags: