Part 2: Grab ETW Session, Providers and Events.

Introduction

In the previous post, you saw how the TraceEvent nuget helps you deciphering simple ETW events such as the one emitted when a first chance exception happens. Most situations trigger more than one event and could make their processing more complicated.

Who said Finalizer?

In the early days of .NET, you might had to deal with native resources that you were responsible for cleaning up with the related unmanaged API or legacy COM component. It was a best practice to implement a ~finalizer method to ensure that everything was deleted the right way. These times are over for most of us now. If you don’t have an IntPtr field in your class, chances are that you don’t need a ~finalizer method.

The Microsoft documentation about IDisposable/Finalizer often leads people to implement both even though only IDisposable is needed (i.e. some fields of the class implement IDisposable). Having a large number of finalizers could impact memory consumption by having objects staying alive for a longer time and maybe even increase garbage collection total duration. Last but not least, some finalizers code outside of your code base could “block” on locks during their cleanup and… drastically slow down everything else.

Getting the name of these types with TraceEvent is a two steps process. First, a TypeBulkType event is received: it contains a list of GCBulkTypeValues which binds a TypeID integer to a string type name:

OnTypeBulkType.cs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

private void OnTypeBulkType(GCBulkTypeTraceData data)
{
   if (data.ProcessID != _processId)
      return;

   // keep track of the id/name type associations
   for (int currentType = 0; currentType < data.Count; currentType++)
   {
      GCBulkTypeValues value = data.Values(currentType);
      _types[value.TypeID] = value.TypeName;
   }
}

This association is needed because when a finalizer is notified via the GCFinalizeObject event, the received data only contains the type ID:

OnGCFinalizeObject.cs

1
2
3
4
5
6
7
8

private void OnGCFinalizeObject(FinalizeObjectTraceData data)
{
   if (data.ProcessID != _processId)
      return;

   // the type id should have been associated to a name via a previous TypeBulkType event
   NotifyFinalize(data.TimeStamp, data.ProcessID, data.TypeID, _types[data.TypeID]);
}

Note that in the two code snippets, there is an explicit check to keep the events only from the process we are interested in: as explained in part 1, this is needed for older versions of Windows.

Thread contention duration

With .NET CLR LocksAndThreads “Contention Rate / sec” and “Total # of Contentions” performance counters, you can monitor how many times threads have been blocked while waiting for a lock owned by another thread. However, you don’t know for how long. The two TraceEvent ContentionStart and ContentionStop events allow you to get this crucial piece of information.

As their names imply and the corresponding documentation explains, these two events let you know respectively when a thread starts to wait on a lock and when the lock has been acquired. In addition to the process and thread identifiers, the ContentionTraceData event argument gives you the type of contention with its ContentionFlags property: either managed or native

ContentionTraceData.cs

1
2
3

public sealed class ContentionTraceData : TraceEvent
{
   public ContentionFlags ContentionFlags { get; }

Since contention is a per-thread waiting operation, you need to keep track of the starting time on a per-thread basis when ContentionStart happens.

OnContentionStart.cs

1
2
3
4
5
6
7
8
9

private void OnContentionStart(ContentionTraceData data)
{
   ContentionInfo info = _contentionStore.GetContentionInfo(data.ProcessID, data.ThreadID);
   if (info == null)
      return;

   info.TimeStamp = data.TimeStamp;
   info.ContentionStartRelativeMSec = data.TimeStampRelativeMSec;
}

The ContentionStore class keeps track of the monitored processes and assign them a ContentionInfo instance where the contention details are stored.

Now you retrieve it back when the matching ContentionStop event occurs. The rest is just a matter of computing the time difference between the two events based on their TimeStampRelativeMSec property.

OnContentionStop.cs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

private void OnContentionStop(ContentionTraceData data)
{
   ContentionInfo info = _contentionStore.GetContentionInfo(data.ProcessID, data.ThreadID);
   if (info == null)
      return;

   // unlucky case when we start to listen just after the ContentionStart event
   if (info.ContentionStartRelativeMSec == 0)
      return;

   var contentionDurationMSec = data.TimeStampRelativeMSec - info.ContentionStartRelativeMSec;
   var isManaged = (data.ContentionFlags == ContentionFlags.Managed);
}

You are now able to detect when thread contention occurs but also if the contention duration increases over time.

If you want to test contention, here is the kind of code you could use:

TestContention.cs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

_workers = new Task[workerCount];
for (int i = 0; i < workerCount; i++)
{
   _workers[i] = Task.Run(async () =>
   {
      while (true)
      {
         lock (_lock)
         {
            Thread.Sleep(5);
         }
      }
   });
}

A few tasks are created to acquire the same lock over and over and sleeping 5 milliseconds before releasing it.

How to count threads or monitor the ThreadPool usage?

In a previous post, it was mentioned that CLR performance counters related to threads were not able to provide an accurate count of the running threads. In fact, you could use the Process/Thread Count Windows kernel performance counter to get the accurate value. If you build .NET Core applications to run on Linux, you have to find other ways such as described on stackoverflow. However, there is an easy programmatic way to get the number of running threads in an application that works both on Windows and Linux: call Process.GetProcessById().Threads.Count with its process ID.

Since this is a series dedicated to ETW, you would expect to simply listen to a few events to get the thread count. Well… It is almost that simple. Each time a thread gets started, the AppDomainResourceManagement/ThreadCreated event is emitted with basically the ID of the created thread as payload. In order to receive the sibling AppDomainResourceManagement/ThreadTerminated event, you need to call AppDomain.MonitoringIsEnabled in the monitored application. The other ways described by the documentation did not work for me.

If you want to figure out if your applications are not hammering too much the .NET thread pool, the CLR provides many ETW events for you to listen that map to the following TraceEvent events:

The ThreadPoolWorkerThreadAdjustementAdjustment event (there is no typo in this name) provides a Reason property. If its value is 0x06, then it means Starvation: if this event frequency is ~1 per second, it could be a good indication that the ThreadPool is receiving a burst of workitems or tasks to process. In addition, the ThreadPoolWorkerThreadAdjustmentTraceData argument received by the handler also gives the count of threads via the NewWorkerThreadCount property.

With all these events, you should be able to monitor how the .NET ThreadPool is used in your application.

The next post will be entirely dedicated to garbage collection analysis.

Co-authored with Kevin Gosse

In the previous post, you saw that the CLR is emitting traces that could (should?) replace the performance counters you are using to monitor your application and investigate when something goes wrong. The perfview tool that was demonstrated is built on top of the Microsoft.Diagnostics.Tracing.TraceEvent Nuget package and you should leverage it to build your own monitoring system. In addition, the Microsoft.Diagnostics.Tracing.TraceEvent.Samples Nuget package contains sample code to help you ramping up.

Manage an ETW session

Create a console application and add the TraceEvent Nuget package. Your project now contains a TraceEvent.ReadMe.txt and a more detailed _TraceEventProgrammersGuide.docx Word document. You should really take the time to read the latter: it describes the architecture in great details and helps understanding what is going on under the scene.

In the Main entry point, add the following code to list existing ETW sessions:

ShowETWSessions.cs

1
2
3
4
5
6

Console.WriteLine("Current ETW sessions:");
foreach(var session in TraceEventSession.GetActiveSessionNames())
{
   Console.WriteLine(session);
}
Console.WriteLine("--------------------------------------------");

Like logman -ets command, this piece of code might be handy during debugging session. Why? Just because when you debug your code that creates an ETW session and if you stop the debugger before disposing it, the session becomes orphan and after a while, Windows simply refuses to create new ones. In addition to easily find your orphans sessions, another good reason to give a meaningful name to your session is to be able to stop it. Type the following command line: logman -ets stop to close a running session and clean up the mess your debugging sessions might have created. This is definitively better than rebooting the machine.

The next step is to create a session. You get a TraceEventSession object either by attaching to an existing session or by creating a new one as shown in this code:

MainETW.cs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

string sessionName = "EtwSessionForCLR_" + Guid.NewGuid().ToString();
Console.WriteLine($"Starting {sessionName}...\r\n");
using (TraceEventSession userSession = new TraceEventSession(sessionName, TraceEventSessionOptions.Create))
{
   Task.Run(() =>
   {
      // register handlers for events on the session source
      // more on this later...

      // decide which provider to listen to with filters if needed

      // process the events in a blocking call
   });

   // wait for the user to dismiss the session
   Console.WriteLine("Presse ENTER to exit...");
   Console.ReadLine();
}

Why is it necessary to run the code that manipulates the session in another thread with Task.Run? The call to the Process method is synchronous so it would not be possible to get the user input. When the user exits, the session is disposed and the Process method returns. If you close the session with logman, the Process method will also return. This behavior applies because the TraceEventSession.StopOnDispose property is set to true by default.

Note that if you want to use TraceEvent to parse an .etl file, you simply need to pass the filename as an additional parameter to the TraceEventSession constructor; the rest of the code will be the same.

The code running in the task first registers handlers to the events as you will soon see. Next, you need to enable the providers you are interested in receiving events from. In our case, only the ClrTraceEventParser.ProviderGuid CLR provider is enabled (read https://docs.microsoft.com/en-us/dotnet/framework/performance/clr-etw-providers for more details about the two available CLR providers). In addition to the verbosity level, you should set the keywords with the ClrTraceEventParser.Keywords enumeration values corresponding to the categories of events you want to receive

ProviderAndSource.cs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

// register handlers for events on the session source
// more on this later...

// decide which provider to listen to with filters if needed
userSession.EnableProvider(
   ClrTraceEventParser.ProviderGuid,
   TraceEventLevel.Verbose,
   (ulong)(
      ClrTraceEventParser.Keywords.Contention | // thread contention timing
      ClrTraceEventParser.Keywords.Threading | // threadpool events
      ClrTraceEventParser.Keywords.Exception | // get the first chance exceptions
      ClrTraceEventParser.Keywords.GCHeapAndTypeNames | 
      ClrTraceEventParser.Keywords.Type | // for finalizer and exceptions type names
      ClrTraceEventParser.Keywords.GC // garbage collector details
);

// this is a blocking call until the session is disposed
userSession.Source.Process();
Console.WriteLine("End of session");

Additional filters such as which process to monitor can be set by passing TraceEventProviderOptions to the EnableProvider() method. But wait! On a Windows 7 machine this kind of filtering is not working and you get events from all processes… No specific documentation to look at… This is where knowing what Win32 APIs are called behind the scene could help. Instead of decompiling the TraceEvent assembly, you should instead take a look at its implementation… because it is open sourced with Perfview! The Microsoft documentation for the called EnableTraceEx2 function states that there are several types of scope filters that allow filtering based on the event ID, the process ID (PID), executable filename, the app ID, and the app package name. This feature is supported on Windows 8.1,Windows Server 2012 R2, and later. If you need to filter out events based on process id, don’t worry: each event will provide it.

Listen to CLR events

This class derives from TraceEventDispatcher that provides the Process method. The TraceEventSource ancestor class is where the event handlers can be registered on the following properties:

Behind the scene, each provider emits strongly typed traces that could be difficult to parse manually: don’t worry, the TraceEvent library does the job for you through dedicated parsers exposed by TraceEventSource.

In case of .NET events, you usually rely on the ClrTraceEventParser that exposes via .NET event the 100+ different traces emitted by the CLR ETW provider… plus one called All just in case you want to see all of them.

Here is the first naïve implementation to display all CLR traces as shown in the previous screenshot:

NaiveListener.cs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

// listen to all CLR events
userSession.Source.Clr.All += delegate (TraceEvent data)
{
   // skip verbose and unneeded events
   if (SkipEvent(data))
      return;

   // raw dump of the events
   Console.WriteLine($"{data.ProcessID,7} <{data.ProviderName}:{data.ID}>__[{data.OpcodeName}] {data.EventName} <| {data.GetType().Name}");
};

The SkipEvent method is here just for you to filter out traces… and this is very helpful to remove meaningless noise when you are beginning to work with CLR traces and you need to see which events are generated.

Each trace payload is received as a generic TraceEvent object that exposes common properties:

• ProcessID/ProcessName: information related to the process in which the trace has been emitted by the CLR
• ThreadID: numeric identifier of the thread from which the trace was sent
• ID: numeric identifier of the trace that helps you find the corresponding event in the Microsoft documentation (i.e. Event ID)
• OpcodeName: human readable name of the trace
• EventName: concatenation of task (= group such as “Contention” corresponding to the Keyword mentioned earlier) and OpcodeName separated by ‘/’

But… what are the events to listen to?

Son, the next big step is to learn which events are interesting for you to monitor. I would suggest you read the rest of this post before going to the Microsoft documentation that describes all CLR events in details.

Let’s start with the simplest case: one trace is emitted when an exception is thrown. Microsoft documents theExceptionThrown_V1 event with ExceptionKeyword and Warning verbosity level:

Unfortunately, there is no ExceptionThrown_V1 event at the ClrTraceEventParser level:

In fact, the ID property of the received traces maps the “Event ID” of the documentation so the ExceptionStart event happens to bring the same level of information as ExceptionThrown_V1 via the ExceptionTraceData parameter passed to the handler:

The corresponding handler implementation is straightforward:

ExceptionStartHandler.cs

1
2
3
4

private static void OnExceptionStart(ExceptionTraceData data)
{
   Console.WriteLine($"{data.EventName} --> {data.ExceptionType} : {data.ExceptionMessage}");
}

The interesting properties are ExceptionType for the name of the thrown exception and ExceptionMessage for its message. Note that if the exception contains inner exceptions, you know it by taking a look at the ExceptionFlags property but you can’t access them. Remember that you should have received earlier the trace corresponding to the inner exception when it was thrown so you are just missing the relationship between the two.

If for an unknown reason the number of exceptions raises for one of your applications, listening to this event is a very cheap way to know which exceptions are thrown and search for them into the source code!

The next episode will detail other important CLR traces you need to monitor your applications and even start an investigation.

Co-authored with Kevin Gosse

At Criteo, each .NET application provides custom metrics to monitor deviation and trigger alerts. This is the first line of defense against misbehaviors. The next step is to figure out what could be the cause of these deviations. After source code changes analysis, it is often needed to dig deeper into performance counters exposed by the CLR such as the following:

Again, these counters are used to detect possible deviations in usual patterns. For example, some applications are supposed to answer under a 50 ms threshold. When the corresponding “number of timeouts” or “request time” metrics start to increase, several reasons linked to the CLR might be partially responsible but it is hard to tell:

• # of Exceps Thrown / sec: if the number increases, it is possible that performance is impacted but how to get the list of these unusual first chance exceptions caught by the applications?
• Contention rate / sec: an increase might suggest that threads are spending more time waiting for locks to be released but for how many milliseconds? This information is not available among the performance counters.
• #Gen 2 Collections: even if this counter does vary a lot, how could we be sure that blocking gen2 collections are responsible for the lack of responsiveness: no counter exposes how many milliseconds the applications threads were frozen in case of a compacting collection.

As you can see, this second line of defense is not enough to start an investigation with a clear assumption in mind.

In addition, some counters are not showing what you might think:

• # Gen Collections: gen0 counter is also incremented after gen1 and gen2 collection, gen1 counter is also incremented after gen2 collection. There is no counter for the exact count of per generation collection trigger even though you could compute their value.

• # of current logical/physical threads: it is not possible to make any link with the number of threads used by the thread pool or the TPL/Tasks. As you can see in the early versions of the Core CLR (i.e. where the performance counters code was not removed yet), the increment and decrement of counts do not seem thread safe: that could explain some issues (always less threads than expected in our monitoring boards) we faced in the past.

You need to realize that performance counters are sampling-based and could show the same value that does not represent the current reality. For example, most of the GC counters will not change until the next garbage collection occurs; i.e. the % Time in GC could stay misleading (until the next GC) and this is very far from the % CPU time you are used to.

If you are moving to .NET Core, you will discover a worse situation: There are no more performance counters on Windows to monitor your applications And if you are targeting Linux… well…

However, as the rest of the articles will demonstrate, the CLR provides even more details via strongly typed tracing through Event Tracing for Windows (ETW) and LTTng on Linux.

CLR and Event Tracing for Windows

The ETW framework has existed for a long long time and allows consumers to listen to events emitted by producers as explained in a 2007 MSDN Magazine article.

A tracing session wraps the providers and consumers together either for real-time processing or .etl file generation. If you already know the Perfview tool or the Windows Performance Toolkit/xperf, you should be familiar with the generated .etl files.

For debugging purpose, listening to events during a live session with a minimal impact on production machines is even more practical. The list of documented CLR events is huge but no fear: this series of posts will focus on exceptions, finalizers, thread contention and garbage collection.

In addition to the documentation, I would recommend that you take a look at how the tracing is implemented in .NET Core source code for two reasons. First, you get access to the exact payload schema of all generated events (even those not documented). Second, by searching the Core CLR source code for the FireEtw-prefixed methods generated at build time, you will get a better understanding of when things are happening. Don’t forget the higher level ETW::-prefixed methods and enums that are also called by the runtime to emit traces.

Perfview has already been mentioned earlier to help analyzing traces but it is also useful for deciphering events produced by the CLR. On a trace, double-click the Events node:

In the new window that pops up, select an event on the left side to get the list of occurrences on the right side:

Right-click an event occurrence and select Dump Event to get its payload details:

such as the type name of the new instance that led the GC to trigger the AllocationTick event after ~100KB was allocated.

In addition to the tooling available to get the traces, Microsoft is providing theMicrosoft.Diagnostics.Tracing.TraceEventNuget package. With this library, you will be able to build your own tool or listen to the CLR events from within your running applications to replace the performance counters. The next episode of the series will ramp you up with the implementation of a basic listener.

Co-authored with Kevin Gosse