Swedish Windows Security User Group

I get a lot of questions about how ACS event retention works.  So here you go, I’m blogging it so I can just answer with a link 🙂

There are two DWORD registry values which affect backlog transmission.  Both are on the collector machine under HKLMSystemCurrentControlSetServicesAdtServerParameters.

EventRetentionPeriod, if present, is expressed in hours (I forget the default).  It takes precedence over MaximumEventAge, which is in days (default=1).  Both of these values control the backlog of events that will be sent from agents to the collector on agent connect, but as mentioned, EventRetentionPeriod wins any conflict.  MaximumEventAge used to control database retention in early beta builds but does not anymore, since the database moved to a partitioning mechanism.  You might encounter MaximumEventAge if you are migrating from ACS beta to Operations Manager 2007 ACS.

Grooming is now governed entirely by the grooming algorithm.  The grooming algorithm is simple: partitions will be deleted by the next grooming job as soon as they are eligible for deletion.

Eligible for deletion means:

• dtPartition.Status == 2 AND


• dtPartition.LastCreationTime < (now() – (partitionDuration * numPartitions))

Think of (partitionDuration * numPartitions) as the retention period before data is groomed from the database. 

• partitionDuration = dtConfig[5]


• numPartitions = dtConfig[6]

Note that dtPartition[<partitionId>].LastCreationTime defaults to 12:00am 1/1/2000 (collector local time).  After successful execution of the close partition script, this field’s value is set to max(dtEvent_<partitionId>.CreationTime) for the partition in question.  There is an implication here that if you update status to 2 without updating LastCreationTime, then the partition is immediately eligible for grooming assuming your clock is accurate.

The partition switch offset (time of day to switch partitions) value in dtConfig has no effect on grooming, other than that grooming will not occur during a partition switch.

Grooming runs at startup and immediately after checkpointing.  The default checkpoint interval is 198 seconds but this interval can be configured  by the DWORD registry value CheckPointInterval on the collector, in the same location as the other registry values.  A successful checkpoint logs an event in the database, event ID 0 with a source of “_acs” (you might have seen these on an “idle” ACS and wondered how they got there…)

I get a lot of questions about how ACS event retention works.  So here you go, I’m blogging it so I can just answer with a link 🙂

There are two DWORD registry values which affect backlog transmission.  Both are on the collector machine under HKLM\System\CurrentControlSet\Services\AdtServer\Parameters.

EventRetentionPeriod, if present, is expressed in hours (I forget the default).  It takes precedence over MaximumEventAge, which is in days (default=1).  Both of these values control the backlog of events that will be sent from agents to the collector on agent connect, but as mentioned, EventRetentionPeriod wins any conflict.  MaximumEventAge used to control database retention in early beta builds but does not anymore, since the database moved to a partitioning mechanism.  You might encounter MaximumEventAge if you are migrating from ACS beta to Operations Manager 2007 ACS.

Grooming is now governed entirely by the grooming algorithm.  The grooming algorithm is simple: partitions will be deleted by the next grooming job as soon as they are eligible for deletion.

Eligible for deletion means:

• dtPartition.Status == 2 AND


• dtPartition.LastCreationTime < (now() – (partitionDuration * numPartitions))

Think of (partitionDuration * numPartitions) as the retention period before data is groomed from the database. 

• partitionDuration = dtConfig[5]


• numPartitions = dtConfig[6]

Note that dtPartition[<partitionId>].LastCreationTime defaults to 12:00am 1/1/2000 (collector local time).  After successful execution of the close partition script, this field’s value is set to max(dtEvent_<partitionId>.CreationTime) for the partition in question.  There is an implication here that if you update status to 2 without updating LastCreationTime, then the partition is immediately eligible for grooming assuming your clock is accurate.

The partition switch offset (time of day to switch partitions) value in dtConfig has no effect on grooming, other than that grooming will not occur during a partition switch.

Grooming runs at startup and immediately after checkpointing.  The default checkpoint interval is 198 seconds but this interval can be configured  by the DWORD registry value CheckPointInterval on the collector, in the same location as the other registry values.  A successful checkpoint logs an event in the database, event ID 0 with a source of “_acs” (you might have seen these on an “idle” ACS and wondered how they got there…)

I get a lot of questions about how ACS event retention works.  So here you go, I’m blogging it so I can just answer with a link 🙂

There are two DWORD registry values which affect backlog transmission.  Both are on the collector machine under HKLM\System\CurrentControlSet\Services\AdtServer\Parameters.

EventRetentionPeriod, if present, is expressed in hours (I forget the default).  It takes precedence over MaximumEventAge, which is in days (default=1).  Both of these values control the backlog of events that will be sent from agents to the collector on agent connect, but as mentioned, EventRetentionPeriod wins any conflict.  MaximumEventAge used to control database retention in early beta builds but does not anymore, since the database moved to a partitioning mechanism.  You might encounter MaximumEventAge if you are migrating from ACS beta to Operations Manager 2007 ACS.

Grooming is now governed entirely by the grooming algorithm.  The grooming algorithm is simple: partitions will be deleted by the next grooming job as soon as they are eligible for deletion.

Eligible for deletion means:

• dtPartition.Status == 2 AND


• dtPartition.LastCreationTime < (now() – (partitionDuration * numPartitions))

Think of (partitionDuration * numPartitions) as the retention period before data is groomed from the database. 

• partitionDuration = dtConfig[5]


• numPartitions = dtConfig[6]

Note that dtPartition[<partitionId>].LastCreationTime defaults to 12:00am 1/1/2000 (collector local time).  After successful execution of the close partition script, this field’s value is set to max(dtEvent_<partitionId>.CreationTime) for the partition in question.  There is an implication here that if you update status to 2 without updating LastCreationTime, then the partition is immediately eligible for grooming assuming your clock is accurate.

The partition switch offset (time of day to switch partitions) value in dtConfig has no effect on grooming, other than that grooming will not occur during a partition switch.

Grooming runs at startup and immediately after checkpointing.  The default checkpoint interval is 198 seconds but this interval can be configured  by the DWORD registry value CheckPointInterval on the collector, in the same location as the other registry values.  A successful checkpoint logs an event in the database, event ID 0 with a source of “_acs” (you might have seen these on an “idle” ACS and wondered how they got there…)

We got several reports recently of a bug in ACS that certain DS Access events, primarily for dnsNode and dnsZone objects, don’t properly get looked up.

Some background: the event log in Windows prefers to log invariants such as message IDs, parameter message IDs, SIDs (security IDs which represent users and groups, etc.), and GUIDs (globally unique IDs which represent objects in Active Directory), rather than the actual names of the objects.  At view time the viewing application is expected to look up the name associated with the invariant and display it to the user.

The reasons that Windows does this are (1) that it enables localization, so that English speakers can see “Administrator” and French speakers can see “Administrateur”, and (2) that it provides rename safefy- many objects are rename-able, such as domain accounts and other AD objects.

Anyway in ACS we had to solve the problem of how to store mountains of log data in a database, make it queryable in meaningful ways, preserve original format, present to users in a recognizable/understandable format, etc.

The way we chose to solve our several problems was to take strings that contained an invariant and append the translated name or string.

For example:
%{e0fa1e8c-9b45-11d0-afdd-00c04fd930c9}
would be translated to:
%{e0fa1e8c-9b45-11d0-afdd-00c04fd930c9}=”dnsNode”

and
%%7685
becomes:
%%7685=”Write Property”

As I mentioned, though, we ran into a problem recently.  Some of our customers were monitoring AD objects with ACS and noticed that ACS was not translating the GUIDs for certain objects.  When they manually looked up the GUIDs they noticed that they were for AD-integrated DNS objects.

After investigation, we found that AD was logging certain audit events for the objects, before all the attributes of the objects had been populated- DNS was populating the objects in multiple operations per object and each operation causes a separate event.  So ACS, which operates as close to real-time as we could get, was actually noticing the first event and asking AD “what’s this?” before DNS had finished updating AD with things like the object’s name.  The difference in time was literally only milliseconds.

Anyway I didn’t really feel it was an ACS bug and wanted to file a bug against Windows DNS Server.  However the Operations Manager team has prototyped a configurable behavior for the ACS agent that lets it wait a very short time (configurable number of ms) and retry, when it fails to look up an AD object because the object doesn’t exist.  This might be released as a public patch and/or in a future Service Pack.

I thought you might appreciate stories of the kinds of weirdness we run into.

We got several reports recently of a bug in ACS that certain DS Access events, primarily for dnsNode and dnsZone objects, don’t properly get looked up.

Some background: the event log in Windows prefers to log invariants such as message IDs, parameter message IDs, SIDs (security IDs which represent users and groups, etc.), and GUIDs (globally unique IDs which represent objects in Active Directory), rather than the actual names of the objects.  At view time the viewing application is expected to look up the name associated with the invariant and display it to the user.

The reasons that Windows does this are (1) that it enables localization, so that English speakers can see “Administrator” and French speakers can see “Administrateur”, and (2) that it provides rename safefy- many objects are rename-able, such as domain accounts and other AD objects.

Anyway in ACS we had to solve the problem of how to store mountains of log data in a database, make it queryable in meaningful ways, preserve original format, present to users in a recognizable/understandable format, etc.

The way we chose to solve our several problems was to take strings that contained an invariant and append the translated name or string.

For example:
%{e0fa1e8c-9b45-11d0-afdd-00c04fd930c9}
would be translated to:
%{e0fa1e8c-9b45-11d0-afdd-00c04fd930c9}=”dnsNode”

and
%%7685
becomes:
%%7685=”Write Property”

As I mentioned, though, we ran into a problem recently.  Some of our customers were monitoring AD objects with ACS and noticed that ACS was not translating the GUIDs for certain objects.  When they manually looked up the GUIDs they noticed that they were for AD-integrated DNS objects.

After investigation, we found that AD was logging certain audit events for the objects, before all the attributes of the objects had been populated- DNS was populating the objects in multiple operations per object and each operation causes a separate event.  So ACS, which operates as close to real-time as we could get, was actually noticing the first event and asking AD “what’s this?” before DNS had finished updating AD with things like the object’s name.  The difference in time was literally only milliseconds.

Anyway I didn’t really feel it was an ACS bug and wanted to file a bug against Windows DNS Server.  However the Operations Manager team has prototyped a configurable behavior for the ACS agent that lets it wait a very short time (configurable number of ms) and retry, when it fails to look up an AD object because the object doesn’t exist.  This might be released as a public patch and/or in a future Service Pack.

I thought you might appreciate stories of the kinds of weirdness we run into.

We got several reports recently of a bug in ACS that certain DS Access events, primarily for dnsNode and dnsZone objects, don’t properly get looked up.

Some background: the event log in Windows prefers to log invariants such as message IDs, parameter message IDs, SIDs (security IDs which represent users and groups, etc.), and GUIDs (globally unique IDs which represent objects in Active Directory), rather than the actual names of the objects.  At view time the viewing application is expected to look up the name associated with the invariant and display it to the user.

The reasons that Windows does this are (1) that it enables localization, so that English speakers can see “Administrator” and French speakers can see “Administrateur”, and (2) that it provides rename safefy- many objects are rename-able, such as domain accounts and other AD objects.

Anyway in ACS we had to solve the problem of how to store mountains of log data in a database, make it queryable in meaningful ways, preserve original format, present to users in a recognizable/understandable format, etc.

The way we chose to solve our several problems was to take strings that contained an invariant and append the translated name or string.

For example:
%{e0fa1e8c-9b45-11d0-afdd-00c04fd930c9}
would be translated to:
%{e0fa1e8c-9b45-11d0-afdd-00c04fd930c9}=”dnsNode”

and
%%7685
becomes:
%%7685=”Write Property”

As I mentioned, though, we ran into a problem recently.  Some of our customers were monitoring AD objects with ACS and noticed that ACS was not translating the GUIDs for certain objects.  When they manually looked up the GUIDs they noticed that they were for AD-integrated DNS objects.

After investigation, we found that AD was logging certain audit events for the objects, before all the attributes of the objects had been populated- DNS was populating the objects in multiple operations per object and each operation causes a separate event.  So ACS, which operates as close to real-time as we could get, was actually noticing the first event and asking AD “what’s this?” before DNS had finished updating AD with things like the object’s name.  The difference in time was literally only milliseconds.

Anyway I didn’t really feel it was an ACS bug and wanted to file a bug against Windows DNS Server.  However the Operations Manager team has prototyped a configurable behavior for the ACS agent that lets it wait a very short time (configurable number of ms) and retry, when it fails to look up an AD object because the object doesn’t exist.  This might be released as a public patch and/or in a future Service Pack.

I thought you might appreciate stories of the kinds of weirdness we run into.

A judge in New Zealand declined to convict the admitted (guilty plea) botherder of a million-bot botnet, citing the negative consequences a conviction would have on the young man’s future prospects.  See the story here.

Well duh.  The whole theory of crime and punishment is that if you do something bad, you get punished, and punishment is something that is unpleasant, so you try to avoid it, hopefully by not doing the crime.  See?  One would hope that a judge would understand this concept.

I could understand if the judge said “this is just a stupid kid, he doesn’t deserve to do 20 years”, and gave the kid probation, community service and a big fine.  I don’t know if New Zealand has such options, or if the judge has latitude in sentencing.  There is probably more to the story than is being told.  But you don’t take over a million computers that don’t belong to you, personally making tens of thousands of dollars, and not realize that you’re doing something wrong.  Unless you’re a sociopath.  And in either case, you either need punishment (for doing something you know is wrong) or separation from society for the protection of society while you get treatment (if you are a sociopath).  So whatever the case, the judge got it wrong, and as a result is practically encouraging future behavior of the same sort.

A judge in New Zealand declined to convict the admitted (guilty plea) botherder of a million-bot botnet, citing the negative consequences a conviction would have on the young man’s future prospects.  See the story here.

Well duh.  The whole theory of crime and punishment is that if you do something bad, you get punished, and punishment is something that is unpleasant, so you try to avoid it, hopefully by not doing the crime.  See?  One would hope that a judge would understand this concept.

I could understand if the judge said “this is just a stupid kid, he doesn’t deserve to do 20 years”, and gave the kid probation, community service and a big fine.  I don’t know if New Zealand has such options, or if the judge has latitude in sentencing.  There is probably more to the story than is being told.  But you don’t take over a million computers that don’t belong to you, personally making tens of thousands of dollars, and not realize that you’re doing something wrong.  Unless you’re a sociopath.  And in either case, you either need punishment (for doing something you know is wrong) or separation from society for the protection of society while you get treatment (if you are a sociopath).  So whatever the case, the judge got it wrong, and as a result is practically encouraging future behavior of the same sort.

A judge in New Zealand declined to convict the admitted (guilty plea) botherder of a million-bot botnet, citing the negative consequences a conviction would have on the young man’s future prospects.  See the story here.

Well duh.  The whole theory of crime and punishment is that if you do something bad, you get punished, and punishment is something that is unpleasant, so you try to avoid it, hopefully by not doing the crime.  See?  One would hope that a judge would understand this concept.

I could understand if the judge said “this is just a stupid kid, he doesn’t deserve to do 20 years”, and gave the kid probation, community service and a big fine.  I don’t know if New Zealand has such options, or if the judge has latitude in sentencing.  There is probably more to the story than is being told.  But you don’t take over a million computers that don’t belong to you, personally making tens of thousands of dollars, and not realize that you’re doing something wrong.  Unless you’re a sociopath.  And in either case, you either need punishment (for doing something you know is wrong) or separation from society for the protection of society while you get treatment (if you are a sociopath).  So whatever the case, the judge got it wrong, and as a result is practically encouraging future behavior of the same sort.

If you haven’t used wevtutil.exe to script event log tasks in Windows Vista or Windows Server 2008, you’re missing out.  The new tool makes getting events out of the log pretty easy, but the main thing is that it doesn’t suffer from any of the drawbacks around getting field delimiting correct.

The tool’s command to query events from a log is “qe”, and takes a log name as a parameter.

If you want to specify a query expression, then you can use XPath with the /q switch.  The easiest way to do this is to use Event Viewer to build a filter for just the events that you want, and then copy just the XPath expression out of the XML tab of the filter dialog in Event Viewer.  Be careful to copy only the filter expression and not the XML that surrounds it. 

Finally, the default output format of wevtutil is XML.  However it dumps each event as XML, but does not include a root element- in other words it’s not well-formed XML by default.  To include a root element you need to include the /e switch and a root element name.

I put this all together in a batch file, with an example XPath filter that just gathers interactive logon events (event ID=4624, logon type=2).  You can save this as a .cmd file and run it as an administrator on Vista or WS08 and it will pull up a list of your interactive logons in Internet Explorer (or your default XML handler application if you’ve changed the registration).  It has to run as admin because it accesses the security event log.

If you’re really good (better than me, which is not hard) you could write an XSL style sheet and put this into a report format.

Good luck!

If you haven’t used wevtutil.exe to script event log tasks in Windows Vista or Windows Server 2008, you’re missing out.  The new tool makes getting events out of the log pretty easy, but the main thing is that it doesn’t suffer from any of the drawbacks around getting field delimiting correct.

The tool’s command to query events from a log is “qe”, and takes a log name as a parameter.

If you want to specify a query expression, then you can use XPath with the /q switch.  The easiest way to do this is to use Event Viewer to build a filter for just the events that you want, and then copy just the XPath expression out of the XML tab of the filter dialog in Event Viewer.  Be careful to copy only the filter expression and not the XML that surrounds it. 

Finally, the default output format of wevtutil is XML.  However it dumps each event as XML, but does not include a root element- in other words it’s not well-formed XML by default.  To include a root element you need to include the /e switch and a root element name.

I put this all together in a batch file, with an example XPath filter that just gathers interactive logon events (event ID=4624, logon type=2).  You can save this as a .cmd file and run it as an administrator on Vista or WS08 and it will pull up a list of your interactive logons in Internet Explorer (or your default XML handler application if you’ve changed the registration).  It has to run as admin because it accesses the security event log.

If you’re really good (better than me, which is not hard) you could write an XSL style sheet and put this into a report format.

Good luck!

If you haven’t used wevtutil.exe to script event log tasks in Windows Vista or Windows Server 2008, you’re missing out.  The new tool makes getting events out of the log pretty easy, but the main thing is that it doesn’t suffer from any of the drawbacks around getting field delimiting correct.

The tool’s command to query events from a log is “qe”, and takes a log name as a parameter.

If you want to specify a query expression, then you can use XPath with the /q switch.  The easiest way to do this is to use Event Viewer to build a filter for just the events that you want, and then copy just the XPath expression out of the XML tab of the filter dialog in Event Viewer.  Be careful to copy only the filter expression and not the XML that surrounds it. 

Finally, the default output format of wevtutil is XML.  However it dumps each event as XML, but does not include a root element- in other words it’s not well-formed XML by default.  To include a root element you need to include the /e switch and a root element name.

I put this all together in a batch file, with an example XPath filter that just gathers interactive logon events (event ID=4624, logon type=2).  You can save this as a .cmd file and run it as an administrator on Vista or WS08 and it will pull up a list of your interactive logons in Internet Explorer (or your default XML handler application if you’ve changed the registration).  It has to run as admin because it accesses the security event log.

If you’re really good (better than me, which is not hard) you could write an XSL style sheet and put this into a report format.

Good luck!