Archive

Archive for the ‘ACS’ Category

ACS Event Retention Mechanism

July 18th, 2008 Comments off

I get a lot of questions about how ACS event retention works.  So here you go, I’m blogging it so I can just answer with a link 🙂


There are two DWORD registry values which affect backlog transmission.  Both are on the collector machine under HKLM\System\CurrentControlSet\Services\AdtServer\Parameters.


EventRetentionPeriod, if present, is expressed in hours (I forget the default).  It takes precedence over MaximumEventAge, which is in days (default=1).  Both of these values control the backlog of events that will be sent from agents to the collector on agent connect, but as mentioned, EventRetentionPeriod wins any conflict.  MaximumEventAge used to control database retention in early beta builds but does not anymore, since the database moved to a partitioning mechanism.  You might encounter MaximumEventAge if you are migrating from ACS beta to Operations Manager 2007 ACS.


Grooming is now governed entirely by the grooming algorithm.  The grooming algorithm is simple: partitions will be deleted by the next grooming job as soon as they are eligible for deletion.


Eligible for deletion means:



  • dtPartition.Status == 2 AND

  • dtPartition.LastCreationTime < (now() – (partitionDuration * numPartitions))

Think of (partitionDuration * numPartitions) as the retention period before data is groomed from the database. 



  • partitionDuration = dtConfig[5]

  • numPartitions = dtConfig[6]

Note that dtPartition[<partitionId>].LastCreationTime defaults to 12:00am 1/1/2000 (collector local time).  After successful execution of the close partition script, this field’s value is set to max(dtEvent_<partitionId>.CreationTime) for the partition in question.  There is an implication here that if you update status to 2 without updating LastCreationTime, then the partition is immediately eligible for grooming assuming your clock is accurate.


The partition switch offset (time of day to switch partitions) value in dtConfig has no effect on grooming, other than that grooming will not occur during a partition switch.


Grooming runs at startup and immediately after checkpointing.  The default checkpoint interval is 198 seconds but this interval can be configured  by the DWORD registry value CheckPointInterval on the collector, in the same location as the other registry values.  A successful checkpoint logs an event in the database, event ID 0 with a source of “_acs” (you might have seen these on an “idle” ACS and wondered how they got there…)

Categories: ACS Tags:

ACS Event Retention Mechanism

July 18th, 2008 No comments

I get a lot of questions about how ACS event retention works.  So here you go, I’m blogging it so I can just answer with a link 🙂


There are two DWORD registry values which affect backlog transmission.  Both are on the collector machine under HKLM\System\CurrentControlSet\Services\AdtServer\Parameters.


EventRetentionPeriod, if present, is expressed in hours (I forget the default).  It takes precedence over MaximumEventAge, which is in days (default=1).  Both of these values control the backlog of events that will be sent from agents to the collector on agent connect, but as mentioned, EventRetentionPeriod wins any conflict.  MaximumEventAge used to control database retention in early beta builds but does not anymore, since the database moved to a partitioning mechanism.  You might encounter MaximumEventAge if you are migrating from ACS beta to Operations Manager 2007 ACS.


Grooming is now governed entirely by the grooming algorithm.  The grooming algorithm is simple: partitions will be deleted by the next grooming job as soon as they are eligible for deletion.


Eligible for deletion means:



  • dtPartition.Status == 2 AND

  • dtPartition.LastCreationTime < (now() – (partitionDuration * numPartitions))

Think of (partitionDuration * numPartitions) as the retention period before data is groomed from the database. 



  • partitionDuration = dtConfig[5]

  • numPartitions = dtConfig[6]

Note that dtPartition[<partitionId>].LastCreationTime defaults to 12:00am 1/1/2000 (collector local time).  After successful execution of the close partition script, this field’s value is set to max(dtEvent_<partitionId>.CreationTime) for the partition in question.  There is an implication here that if you update status to 2 without updating LastCreationTime, then the partition is immediately eligible for grooming assuming your clock is accurate.


The partition switch offset (time of day to switch partitions) value in dtConfig has no effect on grooming, other than that grooming will not occur during a partition switch.


Grooming runs at startup and immediately after checkpointing.  The default checkpoint interval is 198 seconds but this interval can be configured  by the DWORD registry value CheckPointInterval on the collector, in the same location as the other registry values.  A successful checkpoint logs an event in the database, event ID 0 with a source of “_acs” (you might have seen these on an “idle” ACS and wondered how they got there…)

Categories: ACS Tags:

ACS Event Retention Mechanism

July 18th, 2008 No comments

I get a lot of questions about how ACS event retention works.  So here you go, I’m blogging it so I can just answer with a link 🙂


There are two DWORD registry values which affect backlog transmission.  Both are on the collector machine under HKLMSystemCurrentControlSetServicesAdtServerParameters.


EventRetentionPeriod, if present, is expressed in hours (I forget the default).  It takes precedence over MaximumEventAge, which is in days (default=1).  Both of these values control the backlog of events that will be sent from agents to the collector on agent connect, but as mentioned, EventRetentionPeriod wins any conflict.  MaximumEventAge used to control database retention in early beta builds but does not anymore, since the database moved to a partitioning mechanism.  You might encounter MaximumEventAge if you are migrating from ACS beta to Operations Manager 2007 ACS.


Grooming is now governed entirely by the grooming algorithm.  The grooming algorithm is simple: partitions will be deleted by the next grooming job as soon as they are eligible for deletion.


Eligible for deletion means:



  • dtPartition.Status == 2 AND

  • dtPartition.LastCreationTime < (now() – (partitionDuration * numPartitions))

Think of (partitionDuration * numPartitions) as the retention period before data is groomed from the database. 



  • partitionDuration = dtConfig[5]

  • numPartitions = dtConfig[6]

Note that dtPartition[<partitionId>].LastCreationTime defaults to 12:00am 1/1/2000 (collector local time).  After successful execution of the close partition script, this field’s value is set to max(dtEvent_<partitionId>.CreationTime) for the partition in question.  There is an implication here that if you update status to 2 without updating LastCreationTime, then the partition is immediately eligible for grooming assuming your clock is accurate.


The partition switch offset (time of day to switch partitions) value in dtConfig has no effect on grooming, other than that grooming will not occur during a partition switch.


Grooming runs at startup and immediately after checkpointing.  The default checkpoint interval is 198 seconds but this interval can be configured  by the DWORD registry value CheckPointInterval on the collector, in the same location as the other registry values.  A successful checkpoint logs an event in the database, event ID 0 with a source of “_acs” (you might have seen these on an “idle” ACS and wondered how they got there…)

Categories: ACS Tags:

ACS’ first bug from being too performant

July 16th, 2008 Comments off

We got several reports recently of a bug in ACS that certain DS Access events, primarily for dnsNode and dnsZone objects, don’t properly get looked up.


Some background: the event log in Windows prefers to log invariants such as message IDs, parameter message IDs, SIDs (security IDs which represent users and groups, etc.), and GUIDs (globally unique IDs which represent objects in Active Directory), rather than the actual names of the objects.  At view time the viewing application is expected to look up the name associated with the invariant and display it to the user.


The reasons that Windows does this are (1) that it enables localization, so that English speakers can see “Administrator” and French speakers can see “Administrateur”, and (2) that it provides rename safefy- many objects are rename-able, such as domain accounts and other AD objects.


Anyway in ACS we had to solve the problem of how to store mountains of log data in a database, make it queryable in meaningful ways, preserve original format, present to users in a recognizable/understandable format, etc.


The way we chose to solve our several problems was to take strings that contained an invariant and append the translated name or string.


For example:
%{e0fa1e8c-9b45-11d0-afdd-00c04fd930c9}
would be translated to:
%{e0fa1e8c-9b45-11d0-afdd-00c04fd930c9}=”dnsNode”


and
%%7685
becomes:
%%7685=”Write Property”


As I mentioned, though, we ran into a problem recently.  Some of our customers were monitoring AD objects with ACS and noticed that ACS was not translating the GUIDs for certain objects.  When they manually looked up the GUIDs they noticed that they were for AD-integrated DNS objects.


After investigation, we found that AD was logging certain audit events for the objects, before all the attributes of the objects had been populated- DNS was populating the objects in multiple operations per object and each operation causes a separate event.  So ACS, which operates as close to real-time as we could get, was actually noticing the first event and asking AD “what’s this?” before DNS had finished updating AD with things like the object’s name.  The difference in time was literally only milliseconds.

Anyway I didn’t really feel it was an ACS bug and wanted to file a bug against Windows DNS Server.  However the Operations Manager team has prototyped a configurable behavior for the ACS agent that lets it wait a very short time (configurable number of ms) and retry, when it fails to look up an AD object because the object doesn’t exist.  This might be released as a public patch and/or in a future Service Pack.


I thought you might appreciate stories of the kinds of weirdness we run into.

Categories: ACS, News Tags:

ACS’ first bug from being too performant

July 16th, 2008 No comments

We got several reports recently of a bug in ACS that certain DS Access events, primarily for dnsNode and dnsZone objects, don’t properly get looked up.


Some background: the event log in Windows prefers to log invariants such as message IDs, parameter message IDs, SIDs (security IDs which represent users and groups, etc.), and GUIDs (globally unique IDs which represent objects in Active Directory), rather than the actual names of the objects.  At view time the viewing application is expected to look up the name associated with the invariant and display it to the user.


The reasons that Windows does this are (1) that it enables localization, so that English speakers can see “Administrator” and French speakers can see “Administrateur”, and (2) that it provides rename safefy- many objects are rename-able, such as domain accounts and other AD objects.


Anyway in ACS we had to solve the problem of how to store mountains of log data in a database, make it queryable in meaningful ways, preserve original format, present to users in a recognizable/understandable format, etc.


The way we chose to solve our several problems was to take strings that contained an invariant and append the translated name or string.


For example:
%{e0fa1e8c-9b45-11d0-afdd-00c04fd930c9}
would be translated to:
%{e0fa1e8c-9b45-11d0-afdd-00c04fd930c9}=”dnsNode”


and
%%7685
becomes:
%%7685=”Write Property”


As I mentioned, though, we ran into a problem recently.  Some of our customers were monitoring AD objects with ACS and noticed that ACS was not translating the GUIDs for certain objects.  When they manually looked up the GUIDs they noticed that they were for AD-integrated DNS objects.


After investigation, we found that AD was logging certain audit events for the objects, before all the attributes of the objects had been populated- DNS was populating the objects in multiple operations per object and each operation causes a separate event.  So ACS, which operates as close to real-time as we could get, was actually noticing the first event and asking AD “what’s this?” before DNS had finished updating AD with things like the object’s name.  The difference in time was literally only milliseconds.

Anyway I didn’t really feel it was an ACS bug and wanted to file a bug against Windows DNS Server.  However the Operations Manager team has prototyped a configurable behavior for the ACS agent that lets it wait a very short time (configurable number of ms) and retry, when it fails to look up an AD object because the object doesn’t exist.  This might be released as a public patch and/or in a future Service Pack.


I thought you might appreciate stories of the kinds of weirdness we run into.

Categories: ACS, News Tags:

ACS’ first bug from being too performant

July 16th, 2008 No comments

We got several reports recently of a bug in ACS that certain DS Access events, primarily for dnsNode and dnsZone objects, don’t properly get looked up.


Some background: the event log in Windows prefers to log invariants such as message IDs, parameter message IDs, SIDs (security IDs which represent users and groups, etc.), and GUIDs (globally unique IDs which represent objects in Active Directory), rather than the actual names of the objects.  At view time the viewing application is expected to look up the name associated with the invariant and display it to the user.


The reasons that Windows does this are (1) that it enables localization, so that English speakers can see “Administrator” and French speakers can see “Administrateur”, and (2) that it provides rename safefy- many objects are rename-able, such as domain accounts and other AD objects.


Anyway in ACS we had to solve the problem of how to store mountains of log data in a database, make it queryable in meaningful ways, preserve original format, present to users in a recognizable/understandable format, etc.


The way we chose to solve our several problems was to take strings that contained an invariant and append the translated name or string.


For example:
%{e0fa1e8c-9b45-11d0-afdd-00c04fd930c9}
would be translated to:
%{e0fa1e8c-9b45-11d0-afdd-00c04fd930c9}=”dnsNode”


and
%%7685
becomes:
%%7685=”Write Property”


As I mentioned, though, we ran into a problem recently.  Some of our customers were monitoring AD objects with ACS and noticed that ACS was not translating the GUIDs for certain objects.  When they manually looked up the GUIDs they noticed that they were for AD-integrated DNS objects.


After investigation, we found that AD was logging certain audit events for the objects, before all the attributes of the objects had been populated- DNS was populating the objects in multiple operations per object and each operation causes a separate event.  So ACS, which operates as close to real-time as we could get, was actually noticing the first event and asking AD “what’s this?” before DNS had finished updating AD with things like the object’s name.  The difference in time was literally only milliseconds.

Anyway I didn’t really feel it was an ACS bug and wanted to file a bug against Windows DNS Server.  However the Operations Manager team has prototyped a configurable behavior for the ACS agent that lets it wait a very short time (configurable number of ms) and retry, when it fails to look up an AD object because the object doesn’t exist.  This might be released as a public patch and/or in a future Service Pack.


I thought you might appreciate stories of the kinds of weirdness we run into.

Categories: ACS, News Tags:

ACS Event Transformation Demystified

February 28th, 2008 Comments off

I’ve decided to start dumping my knowledge of ACS for posterity’s sake.  My first installment is here, and it’s an excerpt from an external email I put together which describes how event transformation works on ACS.


 


Transformation is performed on the agent (using instructions provided at connect time by the collector) and on the collector.  Transformation instructions are all stored on the collector in a file called EventSchema.xml which is in the AdtServer directory (%windir%\system32\security\adtserver).  This file is pointed to in the collector’s registry and is read during startup of the collector service; failure to successfully read and parse this file at startup is a fatal error for the collector (the debug log will complain about parsing).


 


The collector reads EventSchema.xml and builds in-memory binary tables of event transformation instructions and event string types by OS version/event log/event source.


 


The collector (as explained elsewhere) also reads AcsConfig.xml to get its persistent state and configuration for all known agents, to know what logs/sources to collect for each agent/agent group, etc.  This is all read into in-memory state for each agent.


 


At connect time, the agent sends version information- what the OS and agent version and service pack are, etc.  The collector first looks in its in-memory agent state to see what configuration applies to the agent.  Then it looks in its transformation tables and extracts the appropriate version-specific transformation instructions for the events that the collector is configured to collect from that agent.  Then it packages these instructions and sends them to the agent.


 


The agent starts reading events, transforming them according to its instructions from the collector, and sending the transformed events to the collector.  The collector finishes the transformation, services real-time subscriptions and loads the events into the database as appropriate.


 


If the agent encounters an event that is it configured to send (by log/source) but does not have transformation instructions for, then it simply builds a copy the event string for string and sends the copy of the event to the collector as an “unschematized” event.  The collector will handle this event without problems but will not extract non-header user fields (no primary/client/target user fields) and will not add string type information.


 


I’ll take Windows Server 2003 (build 3790), Event Log: Security, Event Source: Security, Event ID: 644 as an example.


 

Here’s the WS03 schema for 644 (excerpt from %systemroot%\system32\security\adtserver\EventSchema.xml in the path “Schema\Log[@Name=’Security’\Source[@Name=’Security’]\Version[@MinBuild=’3790’]\Event[@SourceId=’644’]”).


 


                        <Event SourceId=644 SourceName=SE_AUDITID_ACCOUNT_AUTO_LOCKED>


                              <Call Name=AppendString Param1=1 Param2=0 />


                              <Call Name=AppendString Param1=3 Param2=0 />


                              <Call Name=AppendString Param1=2 Param2=0 />


                              <Call Name=AppendString Param1=4 Param2=0 />


                              <Call Name=AppendString Param1=5 Param2=0 />


                              <Call Name=AppendString Param1=6 Param2=0 />


                              <Call Name=AppendSidFromNames Param1=4 Param2=5 />


                              <Call Name=AppendNamesFromSid Param1=3 Param2=0 />


                              <Param TypeName=typeUserDn />


                              <Param TypeName=typeComputerName />


                              <Param TypeName=typeTargetSid />


                              <Param TypeName=typeClientUser />


                              <Param TypeName=typeClientDomain />


                              <Param TypeName=typeClientLogonId />


                              <Param TypeName=typeClientSid />


                              <Param TypeName=typeTargetUser />


                              <Param TypeName=typeTargetDomain />


                        </Event>


 


The instructions are all applied in order.  “Call” instructions are executed agent-side; “Param” instructions are executed server-side.


 


These instructions can be translated as:


 


·         Take string 1 from the original event and make it string 1 in the new event.  It is of type “typeUserDn”.


·         Take string 3 from the original event and make it string 2 in the new event.  It is of type “typeComputerName”.  Note that we are doing reordering here by appending original string #3 before original string #2.  Nifty, eh?


·         Take string 2 from the original event and make it string 3 in the new event.  It is of type “typeTargetSid”.


·         Take string 4 from the original event and make it string 4 in the new event.  It is of type “typeClientUser”.


·         Take string 5 from the original event and make it string 5 in the new event.  It is of type “typeClientDomain”.


·         Take string 6 from the original event and make it string 6 in the new event.  It is of type “typeClientLogonId”.


·         Take string 4 from the original event and treat is as a user name, and take string 5 from the original event and treat it as a domain name, look up the associated SID and make it string 7 in the new event.  The new string is of type “typeClientSid”.


·         Take string 3 from the new event, treat it as a SID, look up the user/domain name associated with it and append the user name as string 8 to the new event and the domain name as string 9 to the new event.  String 8 is of type “typeTargetUser” and String 9 is of type “typeTargetDomain”.


 


See the reordering?  Now here is an instance of the event with the original event data.  If you’re not familiar with the XML, it’s the XML output of Crimson, the new eventlog service introduced in Vista/WS08, but this is a WS03 [pre-Crimson] machine; we’re looking at a saved event log (evt) file.


 


<Event xmlns=”http://schemas.microsoft.com/win/2004/08/events/event“>


  <System>


    <Provider Name=”Security” />


    <EventID Qualifiers=”0″>644</EventID>


    <Level>0</Level>


    <Task>7</Task>


    <Keywords>0xa0000000000000</Keywords>


    <TimeCreated SystemTime=”2007-12-17T15:50:14.000Z” />


    <EventRecordID>28003981</EventRecordID>


    <Channel>C:\Users\ericf\AppData\Local\Temp\SERVER34_SecEvts.evt</Channel>


    <Computer>SERVER34</Computer>


    <Security UserID=”S-1-5-18″ />


  </System>


  <EventData>


    <Data>user09</Data>                                                                                             // String 1 – user name


    <Data>SERVER34</Data>                                                                                       // String 2 – looks like a machine name, confirmed by string 4


    <Data>%{S-1-5-21-5998314728-109421381-169156293-611111}</Data>            // String 3 – definitely a SID


    <Data>SERVER34$</Data>                                                                                     // String 4 – definitely an account name (machine account)


    <Data>CONTOSO</Data>                                                                                       // String 5 – looks like a domain name


    <Data>(0x0,0x3E7)</Data>                                                                                     // String 6 – definitely a logon ID


    <Data>-</Data>                                                                                                       // String 7 – empty null string at the end of the event (ignored by ACS)


  </EventData>


</Event>


 


When the event arrives at the collector, type information is applied, and then the user fields (typePrimary*, typeClient*, typeTarget*) are extracted from the string data section and the strings that are left are re-numbered starting at 1 (no reordering occurs).


 


Here’s a chart of what the event looks like at the various points in the system.  The changes at each step are shown in red.


 































































































































Original Event in Event Log


Client-Side Transformation at Agent


Server-Side Normalization (WMI/SQL output)


Field


Content Description (implicit)


Field


Content Description (implicit)


Field


Content Description (explicit)


 


 


Client User


 


Client User


typeClientUser


 


 


Client Domain


 


Client Domain


typeClientDomain


 


 


Client Sid


 


Client Sid


typeClientSid


 


 


Client Login Id


 


Client Login Id


typeClientLogonId


 


 


Target User


 


Target User


typeTargetUser


 


 


Target Domain


 


Target Domain


typeTargetDomain


 


 


Target Sid


 


Target Sid


typeTargetSid


String01


typeUserDn


String01


typeUserDn


String01


typeUserDn


String02


typeTargetSid


String02


typeComputerName


String02


typeComputerName


String03


typeComputerName


String03


typeTargetSid


String03


 


String04


typeClientUser


String04


typeClientUser


String04


 


String05


typeClientDomain


String05


typeClientDomain


String05


 


String06


typeClientLogonId


String06


typeClientLogonId


String06


 


String07


 


String07


typeClientSid


String07


 


String08


 


String08


typeTargetUser


String08


 


String09


 


String09


typeTargetDomain


String09


 


 


To finish off a description of transformation, there are 7 transformation functions, each of which can optionally take 2 integers as parameters.  Note that there is no “destination event” field specifier; all references are only to the original event.  That’s because when constructing the destination event, any data added to the event is always appended- it is constructed from beginning to end- so the implicit destination field is “at the end of the event as it is now”.


 












































Function


Parameter 1


Parameter 2


Description


AppendString


Reference to a string parameter in the source event in the event log


Unused


Appends the referenced string to the event which will be sent to the collector


AppendStringFromTable


Reference to a constant string in the statically defined <Strings> table (1-based) in the relevant Source\Version element in EventSchema.xml


Unused


Appends the referenced constant string to the event which will be sent to the collector


AppendProcessNameFromPid


Reference to a string parameter in the source event in the event log (source string is expected to be a numeric process ID)


Unused


Looks up the process image path name for the referenced PID and appends it to the event which will be sent to the collector


AppendTimeFromDatetime


Unused


Unused


Not Implemented/No Action


AppendSidFromNames


Reference to a string parameter in the source event in the event log (source string is expected to be a user name)


Reference to a string parameter in the source event in the event log (source string is expected to be a domain name)


Looks up the SID for the account represented by the specified user and domain names, and appends the SID to the event which will be sent to the collector


AppendNamesFromSid


Reference to a string parameter in the source event in the event log (source string is expected to be a security ID)


Unused


Looks up the user name and domain name for the account represented by the specified SID, and appends the user name and the domain name as separate strings to the event which will be sent to the collector


AppendNumber


Unused


Unused


Not Implemented/No Action


 


Out of range params cause the transformation instruction to be ignored and skipped.  Non-integer params or other XML formatting/malformation problem (including non-UTF8 formatting) cause an EventSchema.xml parsing error at collector startup which in turn causes collector startup failure.


 


So that’s ACS transformation in a nutshell.  I hope this helps you guys understand ACS functionality a little better.


 


Shortly I will finish my write-up on AcsConfig.xml but that is a simple file and not too hard to figure out if you are into experimentation.


 


Here are some cool things that you can try with the event schema file if you are adventurous:


 


1.       Drop fields.  We have modified eventschema.xml successfully to cause it not to collect certain fields (e.g. logon GUIDs) of certain events:


                              <Call Name=AppendString Param1=1 Param2=0 />


                              <Call Name=AppendString Param1=2 Param2=0 />


                              <Call Name=AppendString Param1=3 Param2=0 />









// try deleting a line here


// or, to preserve ordering of subsequent strings


// try replacing “AppendString” with “AppendStringFromTable (param1=1)”


                              <Call Name=AppendString Param1=4 Param2=0 />


                              <Call Name=AppendString Param1=5 Param2=0 />


                              <Call Name=AppendString Param1=6 Param2=0 />




2. Add an event source.  Some caveats are:


·         You must have a unique, well-formed GUID for the new source


·         You have to get events of the new source into the log (try “AuthzReportSecurityEvent” from MSDN)


·         You have to modify AcsConfig.xml to tell the agent(s) to collect the new source


 


 


NB I have used the C/C++ comment syntax throughout this post but note that ACS does not support either C/C++ nor XML style comments in the XML config files it uses

Categories: ACS, HowTo, SEM, Tools Tags:

ACS Event Transformation Demystified

February 28th, 2008 No comments

I’ve decided to start dumping my knowledge of ACS for posterity’s sake.  My first installment is here, and it’s an excerpt from an external email I put together which describes how event transformation works on ACS.


 


Transformation is performed on the agent (using instructions provided at connect time by the collector) and on the collector.  Transformation instructions are all stored on the collector in a file called EventSchema.xml which is in the AdtServer directory (%windir%\system32\security\adtserver).  This file is pointed to in the collector’s registry and is read during startup of the collector service; failure to successfully read and parse this file at startup is a fatal error for the collector (the debug log will complain about parsing).


 


The collector reads EventSchema.xml and builds in-memory binary tables of event transformation instructions and event string types by OS version/event log/event source.


 


The collector (as explained elsewhere) also reads AcsConfig.xml to get its persistent state and configuration for all known agents, to know what logs/sources to collect for each agent/agent group, etc.  This is all read into in-memory state for each agent.


 


At connect time, the agent sends version information- what the OS and agent version and service pack are, etc.  The collector first looks in its in-memory agent state to see what configuration applies to the agent.  Then it looks in its transformation tables and extracts the appropriate version-specific transformation instructions for the events that the collector is configured to collect from that agent.  Then it packages these instructions and sends them to the agent.


 


The agent starts reading events, transforming them according to its instructions from the collector, and sending the transformed events to the collector.  The collector finishes the transformation, services real-time subscriptions and loads the events into the database as appropriate.


 


If the agent encounters an event that is it configured to send (by log/source) but does not have transformation instructions for, then it simply builds a copy the event string for string and sends the copy of the event to the collector as an “unschematized” event.  The collector will handle this event without problems but will not extract non-header user fields (no primary/client/target user fields) and will not add string type information.


 


I’ll take Windows Server 2003 (build 3790), Event Log: Security, Event Source: Security, Event ID: 644 as an example.


 

Here’s the WS03 schema for 644 (excerpt from %systemroot%\system32\security\adtserver\EventSchema.xml in the path “Schema\Log[@Name=’Security’\Source[@Name=’Security’]\Version[@MinBuild=’3790’]\Event[@SourceId=’644’]”).


 


                        <Event SourceId=644 SourceName=SE_AUDITID_ACCOUNT_AUTO_LOCKED>


                              <Call Name=AppendString Param1=1 Param2=0 />


                              <Call Name=AppendString Param1=3 Param2=0 />


                              <Call Name=AppendString Param1=2 Param2=0 />


                              <Call Name=AppendString Param1=4 Param2=0 />


                              <Call Name=AppendString Param1=5 Param2=0 />


                              <Call Name=AppendString Param1=6 Param2=0 />


                              <Call Name=AppendSidFromNames Param1=4 Param2=5 />


                              <Call Name=AppendNamesFromSid Param1=3 Param2=0 />


                              <Param TypeName=typeUserDn />


                              <Param TypeName=typeComputerName />


                              <Param TypeName=typeTargetSid />


                              <Param TypeName=typeClientUser />


                              <Param TypeName=typeClientDomain />


                              <Param TypeName=typeClientLogonId />


                              <Param TypeName=typeClientSid />


                              <Param TypeName=typeTargetUser />


                              <Param TypeName=typeTargetDomain />


                        </Event>


 


The instructions are all applied in order.  “Call” instructions are executed agent-side; “Param” instructions are executed server-side.


 


These instructions can be translated as:


 


·         Take string 1 from the original event and make it string 1 in the new event.  It is of type “typeUserDn”.


·         Take string 3 from the original event and make it string 2 in the new event.  It is of type “typeComputerName”.  Note that we are doing reordering here by appending original string #3 before original string #2.  Nifty, eh?


·         Take string 2 from the original event and make it string 3 in the new event.  It is of type “typeTargetSid”.


·         Take string 4 from the original event and make it string 4 in the new event.  It is of type “typeClientUser”.


·         Take string 5 from the original event and make it string 5 in the new event.  It is of type “typeClientDomain”.


·         Take string 6 from the original event and make it string 6 in the new event.  It is of type “typeClientLogonId”.


·         Take string 4 from the original event and treat is as a user name, and take string 5 from the original event and treat it as a domain name, look up the associated SID and make it string 7 in the new event.  The new string is of type “typeClientSid”.


·         Take string 3 from the new event, treat it as a SID, look up the user/domain name associated with it and append the user name as string 8 to the new event and the domain name as string 9 to the new event.  String 8 is of type “typeTargetUser” and String 9 is of type “typeTargetDomain”.


 


See the reordering?  Now here is an instance of the event with the original event data.  If you’re not familiar with the XML, it’s the XML output of Crimson, the new eventlog service introduced in Vista/WS08, but this is a WS03 [pre-Crimson] machine; we’re looking at a saved event log (evt) file.


 


<Event xmlns=”http://schemas.microsoft.com/win/2004/08/events/event“>


  <System>


    <Provider Name=”Security” />


    <EventID Qualifiers=”0″>644</EventID>


    <Level>0</Level>


    <Task>7</Task>


    <Keywords>0xa0000000000000</Keywords>


    <TimeCreated SystemTime=”2007-12-17T15:50:14.000Z” />


    <EventRecordID>28003981</EventRecordID>


    <Channel>C:\Users\ericf\AppData\Local\Temp\SERVER34_SecEvts.evt</Channel>


    <Computer>SERVER34</Computer>


    <Security UserID=”S-1-5-18″ />


  </System>


  <EventData>


    <Data>user09</Data>                                                                                             // String 1 – user name


    <Data>SERVER34</Data>                                                                                       // String 2 – looks like a machine name, confirmed by string 4


    <Data>%{S-1-5-21-5998314728-109421381-169156293-611111}</Data>            // String 3 – definitely a SID


    <Data>SERVER34$</Data>                                                                                     // String 4 – definitely an account name (machine account)


    <Data>CONTOSO</Data>                                                                                       // String 5 – looks like a domain name


    <Data>(0x0,0x3E7)</Data>                                                                                     // String 6 – definitely a logon ID


    <Data>-</Data>                                                                                                       // String 7 – empty null string at the end of the event (ignored by ACS)


  </EventData>


</Event>


 


When the event arrives at the collector, type information is applied, and then the user fields (typePrimary*, typeClient*, typeTarget*) are extracted from the string data section and the strings that are left are re-numbered starting at 1 (no reordering occurs).


 


Here’s a chart of what the event looks like at the various points in the system.  The changes at each step are shown in red.


 































































































































Original Event in Event Log


Client-Side Transformation at Agent


Server-Side Normalization (WMI/SQL output)


Field


Content Description (implicit)


Field


Content Description (implicit)


Field


Content Description (explicit)


 


 


Client User


 


Client User


typeClientUser


 


 


Client Domain


 


Client Domain


typeClientDomain


 


 


Client Sid


 


Client Sid


typeClientSid


 


 


Client Login Id


 


Client Login Id


typeClientLogonId


 


 


Target User


 


Target User


typeTargetUser


 


 


Target Domain


 


Target Domain


typeTargetDomain


 


 


Target Sid


 


Target Sid


typeTargetSid


String01


typeUserDn


String01


typeUserDn


String01


typeUserDn


String02


typeTargetSid


String02


typeComputerName


String02


typeComputerName


String03


typeComputerName


String03


typeTargetSid


String03


 


String04


typeClientUser


String04


typeClientUser


String04


 


String05


typeClientDomain


String05


typeClientDomain


String05


 


String06


typeClientLogonId


String06


typeClientLogonId


String06


 


String07


 


String07


typeClientSid


String07


 


String08


 


String08


typeTargetUser


String08


 


String09


 


String09


typeTargetDomain


String09


 


 


To finish off a description of transformation, there are 7 transformation functions, each of which can optionally take 2 integers as parameters.  Note that there is no “destination event” field specifier; all references are only to the original event.  That’s because when constructing the destination event, any data added to the event is always appended- it is constructed from beginning to end- so the implicit destination field is “at the end of the event as it is now”.


 












































Function


Parameter 1


Parameter 2


Description


AppendString


Reference to a string parameter in the source event in the event log


Unused


Appends the referenced string to the event which will be sent to the collector


AppendStringFromTable


Reference to a constant string in the statically defined <Strings> table (1-based) in the relevant Source\Version element in EventSchema.xml


Unused


Appends the referenced constant string to the event which will be sent to the collector


AppendProcessNameFromPid


Reference to a string parameter in the source event in the event log (source string is expected to be a numeric process ID)


Unused


Looks up the process image path name for the referenced PID and appends it to the event which will be sent to the collector


AppendTimeFromDatetime


Unused


Unused


Not Implemented/No Action


AppendSidFromNames


Reference to a string parameter in the source event in the event log (source string is expected to be a user name)


Reference to a string parameter in the source event in the event log (source string is expected to be a domain name)


Looks up the SID for the account represented by the specified user and domain names, and appends the SID to the event which will be sent to the collector


AppendNamesFromSid


Reference to a string parameter in the source event in the event log (source string is expected to be a security ID)


Unused


Looks up the user name and domain name for the account represented by the specified SID, and appends the user name and the domain name as separate strings to the event which will be sent to the collector


AppendNumber


Unused


Unused


Not Implemented/No Action


 


Out of range params cause the transformation instruction to be ignored and skipped.  Non-integer params or other XML formatting/malformation problem (including non-UTF8 formatting) cause an EventSchema.xml parsing error at collector startup which in turn causes collector startup failure.


 


So that’s ACS transformation in a nutshell.  I hope this helps you guys understand ACS functionality a little better.


 


Shortly I will finish my write-up on AcsConfig.xml but that is a simple file and not too hard to figure out if you are into experimentation.


 


Here are some cool things that you can try with the event schema file if you are adventurous:


 


1.       Drop fields.  We have modified eventschema.xml successfully to cause it not to collect certain fields (e.g. logon GUIDs) of certain events:


                              <Call Name=AppendString Param1=1 Param2=0 />


                              <Call Name=AppendString Param1=2 Param2=0 />


                              <Call Name=AppendString Param1=3 Param2=0 />









// try deleting a line here


// or, to preserve ordering of subsequent strings


// try replacing “AppendString” with “AppendStringFromTable (param1=1)”


                              <Call Name=AppendString Param1=4 Param2=0 />


                              <Call Name=AppendString Param1=5 Param2=0 />


                              <Call Name=AppendString Param1=6 Param2=0 />




2. Add an event source.  Some caveats are:


·         You must have a unique, well-formed GUID for the new source


·         You have to get events of the new source into the log (try “AuthzReportSecurityEvent” from MSDN)


·         You have to modify AcsConfig.xml to tell the agent(s) to collect the new source


 


 


NB I have used the C/C++ comment syntax throughout this post but note that ACS does not support either C/C++ nor XML style comments in the XML config files it uses

Categories: ACS, HowTo, SEM, Tools Tags:

ACS Event Transformation Demystified

February 28th, 2008 No comments

I’ve decided to start dumping my knowledge of ACS for posterity’s sake.  My first installment is here, and it’s an excerpt from an external email I put together which describes how event transformation works on ACS.


 


Transformation is performed on the agent (using instructions provided at connect time by the collector) and on the collector.  Transformation instructions are all stored on the collector in a file called EventSchema.xml which is in the AdtServer directory (%windir%system32securityadtserver).  This file is pointed to in the collector’s registry and is read during startup of the collector service; failure to successfully read and parse this file at startup is a fatal error for the collector (the debug log will complain about parsing).


 


The collector reads EventSchema.xml and builds in-memory binary tables of event transformation instructions and event string types by OS version/event log/event source.


 


The collector (as explained elsewhere) also reads AcsConfig.xml to get its persistent state and configuration for all known agents, to know what logs/sources to collect for each agent/agent group, etc.  This is all read into in-memory state for each agent.


 


At connect time, the agent sends version information- what the OS and agent version and service pack are, etc.  The collector first looks in its in-memory agent state to see what configuration applies to the agent.  Then it looks in its transformation tables and extracts the appropriate version-specific transformation instructions for the events that the collector is configured to collect from that agent.  Then it packages these instructions and sends them to the agent.


 


The agent starts reading events, transforming them according to its instructions from the collector, and sending the transformed events to the collector.  The collector finishes the transformation, services real-time subscriptions and loads the events into the database as appropriate.


 


If the agent encounters an event that is it configured to send (by log/source) but does not have transformation instructions for, then it simply builds a copy the event string for string and sends the copy of the event to the collector as an “unschematized” event.  The collector will handle this event without problems but will not extract non-header user fields (no primary/client/target user fields) and will not add string type information.


 


I’ll take Windows Server 2003 (build 3790), Event Log: Security, Event Source: Security, Event ID: 644 as an example.


 

Here’s the WS03 schema for 644 (excerpt from %systemroot%system32securityadtserverEventSchema.xml in the path “SchemaLog[@Name=’Security’Source[@Name=’Security’]Version[@MinBuild=’3790’]Event[@SourceId=’644’]”).


 


                        <Event SourceId=644 SourceName=SE_AUDITID_ACCOUNT_AUTO_LOCKED>


                              <Call Name=AppendString Param1=1 Param2=0 />


                              <Call Name=AppendString Param1=3 Param2=0 />


                              <Call Name=AppendString Param1=2 Param2=0 />


                              <Call Name=AppendString Param1=4 Param2=0 />


                              <Call Name=AppendString Param1=5 Param2=0 />


                              <Call Name=AppendString Param1=6 Param2=0 />


                              <Call Name=AppendSidFromNames Param1=4 Param2=5 />


                              <Call Name=AppendNamesFromSid Param1=3 Param2=0 />


                              <Param TypeName=typeUserDn />


                              <Param TypeName=typeComputerName />


                              <Param TypeName=typeTargetSid />


                              <Param TypeName=typeClientUser />


                              <Param TypeName=typeClientDomain />


                              <Param TypeName=typeClientLogonId />


                              <Param TypeName=typeClientSid />


                              <Param TypeName=typeTargetUser />


                              <Param TypeName=typeTargetDomain />


                        </Event>


 


The instructions are all applied in order.  “Call” instructions are executed agent-side; “Param” instructions are executed server-side.


 


These instructions can be translated as:


 


·         Take string 1 from the original event and make it string 1 in the new event.  It is of type “typeUserDn”.


·         Take string 3 from the original event and make it string 2 in the new event.  It is of type “typeComputerName”.  Note that we are doing reordering here by appending original string #3 before original string #2.  Nifty, eh?


·         Take string 2 from the original event and make it string 3 in the new event.  It is of type “typeTargetSid”.


·         Take string 4 from the original event and make it string 4 in the new event.  It is of type “typeClientUser”.


·         Take string 5 from the original event and make it string 5 in the new event.  It is of type “typeClientDomain”.


·         Take string 6 from the original event and make it string 6 in the new event.  It is of type “typeClientLogonId”.


·         Take string 4 from the original event and treat is as a user name, and take string 5 from the original event and treat it as a domain name, look up the associated SID and make it string 7 in the new event.  The new string is of type “typeClientSid”.


·         Take string 3 from the new event, treat it as a SID, look up the user/domain name associated with it and append the user name as string 8 to the new event and the domain name as string 9 to the new event.  String 8 is of type “typeTargetUser” and String 9 is of type “typeTargetDomain”.


 


See the reordering?  Now here is an instance of the event with the original event data.  If you’re not familiar with the XML, it’s the XML output of Crimson, the new eventlog service introduced in Vista/WS08, but this is a WS03 [pre-Crimson] machine; we’re looking at a saved event log (evt) file.


 


<Event xmlns=”http://schemas.microsoft.com/win/2004/08/events/event“>


  <System>


    <Provider Name=”Security” />


    <EventID Qualifiers=”0″>644</EventID>


    <Level>0</Level>


    <Task>7</Task>


    <Keywords>0xa0000000000000</Keywords>


    <TimeCreated SystemTime=”2007-12-17T15:50:14.000Z” />


    <EventRecordID>28003981</EventRecordID>


    <Channel>C:UsersericfAppDataLocalTempSERVER34_SecEvts.evt</Channel>


    <Computer>SERVER34</Computer>


    <Security UserID=”S-1-5-18″ />


  </System>


  <EventData>


    <Data>user09</Data>                                                                                             // String 1 – user name


    <Data>SERVER34</Data>                                                                                       // String 2 – looks like a machine name, confirmed by string 4


    <Data>%{S-1-5-21-5998314728-109421381-169156293-611111}</Data>            // String 3 – definitely a SID


    <Data>SERVER34$</Data>                                                                                     // String 4 – definitely an account name (machine account)


    <Data>CONTOSO</Data>                                                                                       // String 5 – looks like a domain name


    <Data>(0x0,0x3E7)</Data>                                                                                     // String 6 – definitely a logon ID


    <Data>-</Data>                                                                                                       // String 7 – empty null string at the end of the event (ignored by ACS)


  </EventData>


</Event>


 


When the event arrives at the collector, type information is applied, and then the user fields (typePrimary*, typeClient*, typeTarget*) are extracted from the string data section and the strings that are left are re-numbered starting at 1 (no reordering occurs).


 


Here’s a chart of what the event looks like at the various points in the system.  The changes at each step are shown in red.


 































































































































Original Event in Event Log


Client-Side Transformation at Agent


Server-Side Normalization (WMI/SQL output)


Field


Content Description (implicit)


Field


Content Description (implicit)


Field


Content Description (explicit)


 


 


Client User


 


Client User


typeClientUser


 


 


Client Domain


 


Client Domain


typeClientDomain


 


 


Client Sid


 


Client Sid


typeClientSid


 


 


Client Login Id


 


Client Login Id


typeClientLogonId


 


 


Target User


 


Target User


typeTargetUser


 


 


Target Domain


 


Target Domain


typeTargetDomain


 


 


Target Sid


 


Target Sid


typeTargetSid


String01


typeUserDn


String01


typeUserDn


String01


typeUserDn


String02


typeTargetSid


String02


typeComputerName


String02


typeComputerName


String03


typeComputerName


String03


typeTargetSid


String03


 


String04


typeClientUser


String04


typeClientUser


String04


 


String05


typeClientDomain


String05


typeClientDomain


String05


 


String06


typeClientLogonId


String06


typeClientLogonId


String06


 


String07


 


String07


typeClientSid


String07


 


String08


 


String08


typeTargetUser


String08


 


String09


 


String09


typeTargetDomain


String09


 


 


To finish off a description of transformation, there are 7 transformation functions, each of which can optionally take 2 integers as parameters.  Note that there is no “destination event” field specifier; all references are only to the original event.  That’s because when constructing the destination event, any data added to the event is always appended- it is constructed from beginning to end- so the implicit destination field is “at the end of the event as it is now”.


 












































Function


Parameter 1


Parameter 2


Description


AppendString


Reference to a string parameter in the source event in the event log


Unused


Appends the referenced string to the event which will be sent to the collector


AppendStringFromTable


Reference to a constant string in the statically defined <Strings> table (1-based) in the relevant SourceVersion element in EventSchema.xml


Unused


Appends the referenced constant string to the event which will be sent to the collector


AppendProcessNameFromPid


Reference to a string parameter in the source event in the event log (source string is expected to be a numeric process ID)


Unused


Looks up the process image path name for the referenced PID and appends it to the event which will be sent to the collector


AppendTimeFromDatetime


Unused


Unused


Not Implemented/No Action


AppendSidFromNames


Reference to a string parameter in the source event in the event log (source string is expected to be a user name)


Reference to a string parameter in the source event in the event log (source string is expected to be a domain name)


Looks up the SID for the account represented by the specified user and domain names, and appends the SID to the event which will be sent to the collector


AppendNamesFromSid


Reference to a string parameter in the source event in the event log (source string is expected to be a security ID)


Unused


Looks up the user name and domain name for the account represented by the specified SID, and appends the user name and the domain name as separate strings to the event which will be sent to the collector


AppendNumber


Unused


Unused


Not Implemented/No Action


 


Out of range params cause the transformation instruction to be ignored and skipped.  Non-integer params or other XML formatting/malformation problem (including non-UTF8 formatting) cause an EventSchema.xml parsing error at collector startup which in turn causes collector startup failure.


 


So that’s ACS transformation in a nutshell.  I hope this helps you guys understand ACS functionality a little better.


 


Shortly I will finish my write-up on AcsConfig.xml but that is a simple file and not too hard to figure out if you are into experimentation.


 


Here are some cool things that you can try with the event schema file if you are adventurous:


 


1.       Drop fields.  We have modified eventschema.xml successfully to cause it not to collect certain fields (e.g. logon GUIDs) of certain events:


                              <Call Name=AppendString Param1=1 Param2=0 />


                              <Call Name=AppendString Param1=2 Param2=0 />


                              <Call Name=AppendString Param1=3 Param2=0 />









// try deleting a line here


// or, to preserve ordering of subsequent strings


// try replacing “AppendString” with “AppendStringFromTable (param1=1)”


                              <Call Name=AppendString Param1=4 Param2=0 />


                              <Call Name=AppendString Param1=5 Param2=0 />


                              <Call Name=AppendString Param1=6 Param2=0 />




2. Add an event source.  Some caveats are:


·         You must have a unique, well-formed GUID for the new source


·         You have to get events of the new source into the log (try “AuthzReportSecurityEvent” from MSDN)


·         You have to modify AcsConfig.xml to tell the agent(s) to collect the new source


 


 


NB I have used the C/C++ comment syntax throughout this post but note that ACS does not support either C/C++ nor XML style comments in the XML config files it uses

Categories: ACS, HowTo, SEM, Tools Tags: