In the previous article, I presented what is needed (i.e. listen to WaitHandleWait events) to compute lock/wait durations and call stacks for Mutex, Semaphore, SemaphoreSlim, Manual/AutoResetEvent, ManualResetEventSlim, ReaderWriterLockSlim .NET synchronization constructs for a running process.

However, since the application is already running, some JIT-related events are missing, and some frames of the call stacks cannot be symbolized. Also, it would be great to monitor an application’s startup to see if it could be faster.

This post will detail how to monitor a .NET application since the very beginning of its life and the issues you might face.

Preparing a new .NET process to be monitored

From .NET 5, the dotnet-trace CLI tool allows you to pass a command line to execute and trace it from startup. In a very interesting article, Olivier Coanet presented the gory details about how to tell the .NET runtime to start an application in a pseudo-suspended mode as shown in the following diagram:

The first step is to create a ReverseDiagnosticsServer instance with a specific port (i.e. dotnet-wait_1234 in the diagram). Next, the process to monitor is spawned with the DOTNET_DiagnosticPorts environment variable set to the same port (i.e. dotnet-wait_1234). Look at the Diagnostics documentation of the Diagnostic Ports with DOTNET_DiagnosticPorts environment variable for more details. The .NET runtime is the new process will listen to this port and… wait.

When the tool is ready, it sends a resume command via a DiagnosticsClient: from that point in time, the CLR executes the normal flow of actions to run the application and… you will receive all events without missing one!

Get my command line please

Following the dotnet-trace example, my dotnet-wait tool accepts the command line of the child process in its final arguments that follow the** — **trigger. For example, dotnet-wait — dotnet foo.dll will start the program in foo.dll by using dotnet.exe. I’m reusing the code in ReversedServerHelper.cs to deal with arguments containing spaces:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

else if (current == "--")  // this is supposed to be the last one
{
    i++;
    if (i < args.Length)
    {
        parameters.pathName = args[i];

        // use the remaining arguments as the arguments for the child app to spawn
        i++;
        if (i < args.Length)
        {
            parameters.arguments = "";
            for (int j = i; j < args.Length; j++)
            {
                if (args[j].Contains(' '))
                {
                    parameters.arguments += $"\"{args[j].Replace("\"", "\\\"")}\"";
                }
                else
                {
                    parameters.arguments += args[j];
                }

                if (j != args.Length)
                {
                    parameters.arguments += " ";
                }
            }
        }

        // no need to look for more arguments
        break;
    }
    else
    {
        throw new InvalidOperationException($"Missing path name value...");
    }
}

The code to spawn the child process is simple:

1
2
3
4
5
6
7
8
9

// start the monitored app
var psi = new ProcessStartInfo(pathName);
if (!string.IsNullOrEmpty(arguments))
{
    psi.Arguments = arguments;
}
psi.EnvironmentVariables["DOTNET_DiagnosticPorts"] = port;
psi.UseShellExecute = false;
var process = System.Diagnostics.Process.Start(psi);

Here is an example with the following prompt:

-- dotnet "C:\CommandLineTest.dll" one two "t h r e e" four 'five six'

that generates the output (the test application is just listing its arguments):

dotnet-wait v1.0.0.0 - List wait duration
by Christophe Nasarre

Press ENTER to exit...
6 arguments
   1 | one
   2 | two
   3 | t h r e e
   4 | four
   5 | 'five
   6 | six'

This test reminded me to never use simple quotes in prompts :^)

It’s my console!

Once I implemented these steps, I immediately faced a very simple problem: my dotnet-wait tool and the test application are console applications. It means that they will share the same console for both input and output. For example, both are waiting for the RETURN key to (1) stop for the tool and (2) start for the test application: too bad for me because the tool will stop as soon the application starts…

Going back in time in my Windows memories, I remembered that the Win32 CreateProcess API accepts CREATE_NEW_CONSOLE as creation flag to automagically start the child process into its own new console. Unfortunately, it is not possible to pass this flag in .NET; maybe a limitation due to Linux support.

One simple solution could be to redirect the output of the tool or the application to a file: that would avoid mixing them in the console. Note that, by default, dotnet-trace discards output from the child process (by setting RedirectStandardOutput, RedirectStandardError and RedirectStandardInput to false and by ignoring the error and output streams) except if you pass — show-child-io on the command line. In this case, no output for dotnet-trace.

I decided to do the opposite for dotnet-wait: by default, you also get the child output but you can redirect the output of the tool to a file with -o

. Still, this does not solve the input problem in case of common expected keys.

If you remember the interactions between the tool and the monitored application, the latter is suspended until DiagnosticsClient::ResumeRuntime is called. So, why not starting the tool that spawns the application in one console and another instance of the tool in a new console that will resume the application? This is exactly what my friend Kevin Gosse imagined and how dotnet-wait works.

After the timeout that you give to diagnosticsServer.AcceptAsync(cancellation.Token) has elapsed, the runtime in the child process will display the following message:

The runtime has been configured to pause during startup and is awaiting a Diagnostics IPC ResumeStartup command from a Diagnostic Port.
DOTNET_DiagnosticPorts="dotnet-wait_34296"
DOTNET_DefaultDiagnosticPortSuspend=0

And this is exactly what the -r 34296 parameter will do!

You can now install dotnet wait and monitor the lock and wait contentions of your .NET9+ applications.

Introduction

In an old post, I detailed how to use ContentionStart and ContentionStop events to measure the lock contentions duration for a .NET application. In a .NET 9 pull request, a former Criteo’s colleague Grégoire Verdier has added new events to be notified when wait time similar to lock contention is happening for Mutex, Semaphore, Manual/AutoResetEvent. Read his post for more details about what he was trying to investigate.

With asynchronous and multi-threaded algorithms, it is essential to detect unexpected wait/locks in our applications. This post shows you how to leverage these events to measure the duration of these waits and get the call stack when the wait started:

New WaitHandleWait events

These new events are emitted by the Microsoft-Windows-DotNETRuntime CLR provider when you enable the WaitHandle (= 0x40000000000) keyword with Verbose verbosity. Each time WaitOne is called on a waitable object and this object is already owned, a WaitHandleWaitStart event is emitted. When the object is released, a WaitHandleWaitStop event is emitted.

For example, the following code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

static Mutex mutex = new Mutex();

static void Main()
{
    var owningThread = new Thread(OwningThread);
    owningThread.Start();
    var mutexThread = new Thread(MutexThread);
    mutexThread.Start();

    owningThread.Join();
    mutexThread.Join();
}

static void OwningThread()
{
    Console.WriteLine($"    [{GetCurrentThreadId(), 8}] Start to hold resources");
    Console.WriteLine("___________________________________________");
    mutex.WaitOne();

    Thread.Sleep(3000);  // the wait should last ~3 seconds

    Console.WriteLine("    Release resources");
    mutex.ReleaseMutex();
}

static void MutexThread()
{
    Console.WriteLine($"    [{GetCurrentThreadId(), 8}] waiting for Mutex...");
    mutex.WaitOne();  // events are emitted in the implementation when a contention happens
    mutex.ReleaseMutex();
    Console.WriteLine("    <-- Mutex");
}

generates a Start and Stop events pair:

1
2

125980 | 00000000-0000-0000-0000-000000000000 > event 301 __ [ 1| Start] WaitHandleWait/Start
125980 | 00000000-0000-0000-0000-000000000000 > event 302 __ [ 2|  Stop] WaitHandleWait/Stop

There is no associated activity ID so you rely on the fact that the same waiter thread (125980 in the previous example) is emitting for both events.

Listening to the new Wait events

As usual, you should rely on the TraceEvent nuget to start an EventPipe session with an already running .NET application. The last version already contains the definition of the keyword:

1

keywords |= ClrTraceEventParser.Keywords.WaitHandle; // .NET 9 WaitHandle kind of contention

and the C# events for Start and Stop:

1
2

source.Clr.WaitHandleWaitStart += OnWaitHandleWaitStart;
source.Clr.WaitHandleWaitStop += OnWaitHandleWaitStop;

The handler’s implementation is straightforward. The start of the wait is recorded for the current thread:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

private void OnWaitHandleWaitStart(WaitHandleWaitStartTraceData data)
{
    // get the contention info for the current thread
    ContentionInfo info = _contentionStore.GetContentionInfo(data.ProcessID, data.ThreadID);
    if (info == null)
        return;

    // keep track of the wait start
    info.ContentionStartRelativeMSec = data.TimeStampRelativeMSec;
}

When the wait ends, the duration is computed based on the recorded wait start because it is not provided in the payload like for ContentionStop:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

private void OnWaitHandleWaitStop(WaitHandleWaitStopTraceData data)
{
    ContentionInfo info = _contentionStore.GetContentionInfo(data.ProcessID, data.ThreadID);
    if (info == null)
        return;
    // unlucky case when we start to listen just after the WaitHandleStart event
    if (info.ContentionStartRelativeMSec == 0)
    {
        return;
    }

    // Too bad the duration is not provided in the payload like in ContentionStop...
    var contentionDurationMSec = data.TimeStampRelativeMSec - info.ContentionStartRelativeMSec;
    info.ContentionStartRelativeMSec = 0;
    var duration = TimeSpan.FromMilliseconds(contentionDurationMSec);
    Console.WriteLine($"{e.ThreadId,7} | {e.Duration.TotalMilliseconds} ms");
}

This is nice but it would be more useful if we could get the call stack of long waits.

Call stacks with EventPipe

In a previous post, I explained that it is possible to get the call stack when an event is emitted thanks to the ClrStackWalk event that follows the event you are interested in. Unfortunately, this is not more the case for .NET 5+ that is using EventPipe instead of ETW.

As Olivier Coanet presents in his post, you can get the call stack as an array of addresses from the hidden event record that is mapped by the TraceEvent parameter passed to each event handlers. This EVENT_RECORD structure contains a ExtendedData field that is an array of EVENT_HEADER_EXTENDED_DATA_ITEM:

1
2
3
4
5
6
7
8

public struct EVENT_HEADER_EXTENDED_DATA_ITEM
{
    public ushort Reserved1;
    public ushort ExtType;
    public ushort Reserved2;
    public ushort DataSize;
    public ulong DataPtr;
}

If the ExtType value is EVENT_HEADER_EXT_TYPE_STACK_TRACE64 (=6) then DataPtr points to a EVENT_EXTENDED_ITEM_STACK_TRACE64 structure:

1
2
3
4
5

public struct EVENT_EXTENDED_ITEM_STACK_TRACE64
{
    public ulong MatchId;
    public unsafe fixed ulong Address[1];
}

that contains an array of 64-bit addresses. The size of this array is given by DataSize — sizeof(ulong).

For 32-bit applications, you will get EVENT_HEADER_EXT_TYPE_STACK_TRACE32 (=5) as ExtType value and DataPtr will point to EVENT_EXTENDED_ITEM_STACK_TRACE32:

1
2
3
4
5

public struct EVENT_EXTENDED_ITEM_STACK_TRACE32
{
    public ulong MatchId;
    public unsafe fixed uint Address[1];
}

that stores an array of 32-bit addresses.

Knowing that makes writing the code to get the call stacks as an array of 64-bit addresses (same with 32-bit applications for simplicity sake) pretty straightforward:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

public static EventPipeUnresolvedStack ReadStackUsingUnsafeAccessor(TraceEvent traceEvent)
{
    return GetFromEventRecord(traceEvent.eventRecord);
}

private static EventPipeUnresolvedStack GetFromEventRecord(TraceEventNativeMethods.EVENT_RECORD* eventRecord)
{
    if (eventRecord == null)
        return null;

    var extendedDataCount = eventRecord->ExtendedDataCount;

    for (var dataIndex = 0; dataIndex < extendedDataCount; dataIndex++)
    {
        var extendedData = eventRecord->ExtendedData[dataIndex];
        if (extendedData.ExtType == TraceEventNativeMethods.EVENT_HEADER_EXT_TYPE_STACK_TRACE64)
        {
            var stackRecord = (TraceEventNativeMethods.EVENT_EXTENDED_ITEM_STACK_TRACE64*)extendedData.DataPtr;
            var addresses = &stackRecord->Address[0];
            var addressCount = (extendedData.DataSize - sizeof(UInt64)) / sizeof(UInt64);
            if (addressCount == 0)
                return null;

            var callStackAddresses = new ulong[addressCount];
            for (var index = 0; index < addressCount; index++)
            {
                callStackAddresses[index] = addresses[index];
            }
            return new EventPipeUnresolvedStack(callStackAddresses);
        }
        else if (extendedData.ExtType == TraceEventNativeMethods.EVENT_HEADER_EXT_TYPE_STACK_TRACE32)
        {
            var stackRecord = (TraceEventNativeMethods.EVENT_EXTENDED_ITEM_STACK_TRACE32*)extendedData.DataPtr;
            var addresses = &stackRecord->Address[0];
            var addressCount = (extendedData.DataSize - sizeof(UInt32)) / sizeof(UInt32);
            if (addressCount == 0)
                return null;

            var callStackAddresses = new ulong[addressCount];  // store the 32 addresses as 64 bit addresses
            for (var index = 0; index < addressCount; index++)
            {
                callStackAddresses[index] = addresses[index];
            }

            return new EventPipeUnresolvedStack(callStackAddresses);
        }
    }

    return null;
}

Note that the last version of TraceEvent nuget provides a public access to the eventRecord field so it is no more needed to use the UnsafeAccessor attribute used by Olivier.

Symbolize the call stack addresses

Address is good but the corresponding method name is better. I won’t repeat what I’ve already detailed in an older post that shows how to get the name of a native and managed name from an instruction pointer address. Instead, I want to pinpoint a big limitation of this solution to listen to CLR provider MethodLoadVerbose/MethodDCStartVerboseV2 events. If the methods you are interested in are jitted BEFORE your tool attaches to the application, you will never get these events.

You could get the same mapping address span/method name via the other “Microsoft-Windows-DotNETRuntimeRundown” provider and its MethodDCEndVerbose event that contains the expected MethodStartAddress, MethodSize and MethodName in its payload. But I need this information before the end of the application…

Looking at the documentation, it seems that the rundown provider accepts the StartRundownKeyword value to emit the DCStart events when the provider is enabled! Since .NET 9, it is possible to pass the keywords you want (before, the default value did not contain StartRundownKeyword) when creating the EventPipe session

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

//                V-- this is the default rundown keyword
rundownKeywords = 0x80020139 | (long)ClrTraceEventParser.Keywords.StartEnumeration;
var config = new EventPipeSessionConfiguration(GetProviders(), 256, rundownKeywords, true);
using (var session = client.StartEventPipeSession(config))
{
    var source = new EventPipeEventSource(session.EventStream);
    RegisterListeners(source);

    // this is a blocking call
    source.Process();
}

Note that you should not add the rundown provider to the list passed as parameter.

Unfortunately, there is currently an issue in the runtime since September 2020 that pinpoints this exact problem. I even tried to create and close a session to get the DCStop events before recreating a new one, but I failed.

The next episode will talk about how it is possible to start a .NET application and get the events since its startup… with the problems that are happening.

In the previous post, I detailed how I used the undocumented events from the BCL to create the dotnet-http CLI tool to monitor your outgoing HTTP requests. After testing with older versions of .NET, I realized that the code needed to be updated and I’m sharing my findings in this post.

The main point is that url redirections could have a major impact on requests latency:

Always test supported versions…

When I wrote the initial version of dotnet-http, I only tested it with .NET 8 and .NET 9 with limited formats of urls. Unfortunately, things went bad when I tried to monitor applications running on .NET 5 and .NET 6: no events are emitted by these versions of the BCL.

So, the next step was to test .NET 7 and the result was simple: crash! After investigating, I realized that some events I looked at in .NET 8 source code did not have the same payload in .NET 7; even no payload at all:

Even though there is no version field in the events payload, it is easy to check its size such as the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

private void OnConnectionEstablished(
     DateTime timestamp,
     int threadId,
     Guid activityId,
     Guid relatedActivityId,
     byte[] eventData
     )
 {
     ...
     EventSourcePayload payload = new EventSourcePayload(eventData);
     var versionMajor = payload.GetByte();
     var versionMinor = payload.GetByte();

     // in .NET 7, nothing else is available
     Int64 connectionId = 0;
     var scheme = "";
     var host = "";
     UInt32 port = 0;
     var path = "";

     if (eventData.Length > 2)
     {
         connectionId = payload.GetInt64();
         scheme = payload.GetString();
         host = payload.GetString();
         port = payload.GetUInt32();
         path = payload.GetString();
     }
     ...

Another difference is that one even is not even emitted in .NET 7:

Explain what a redirection is please!

Because I based the code on this Redirect event, I needed to find another way to support .NET 7 even though I would not have a redirected url to display. But first, let’s see what I’m talking about in terms of HTTP communication.

When you try to get the content / status code behind a url, the code is following different phases with the related events:

• **Start **RequestStart
• **DNS resolution **ResolutionStart ResolutionStop/Fail
• **Socket connection **ConnectStart ConnectStop
• **Security hand check (HTTPS only) **HandshakeStart HandshakeStop/Failed
• **Request/response **RequestHeadersStart RequestHeadersStop ResponseHeadersStart ResponseHeadersStop Redirect (.NET 8+) ResponseContentStart ResponseContentStop
• **Request stop **RequestStop/Failed

Based on the received url, a server can decide to answer that another url should be used instead. For example, if you call github with http:// instead of https://, such a redirection will happen. Without invasive tools such as Wireshark, these redirections are impossible to detect and could cause unnecessary delay.

From the client perspective, this can be detected in the ResponseHeadersStop event payload that provides a status code. If its value is 301, then it is a redirection. The other effect of a redirection is that the BCL code will change the flow of events because it needs to start over with the new url from step 2. to step 6. As you can see, instead of paying the cost of just one request, two are actually emitted and processed.

Impact on the implementation

In addition to the payload size checks addition, my initial implementation was not properly handling the redirection because the values (timestamps and durations) where overridden by the events related to the second redirected url.

The new implementation is splitting the request details into two classes. A base class that contains common fields to both parts of the request in case of redirection:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

private class HttpRequestInfoBase
   {
       public HttpRequestInfoBase(DateTime timestamp, string scheme, string host, uint port, string path)
       {
           StartTime = timestamp;
           if (scheme == string.Empty)
           {
               Url = string.Empty;
           }
           else
           {
               if (port != 0)
               {
                   Url = $"{scheme}://{host}:{port}{path}";
               }
               else
               {
                   Url = $"{scheme}://{host}:{path}";
               }
           }
       }

       public string Url { get; set; }
       public DateTime StartTime { get; set; }
       public DateTime ReqRespStartTime { get; set; }
       public double ReqRespDuration { get; set; }

       // DNS
       public double DnsWait { get; set; }
       public DateTime DnsStartTime { get; set; }
       public double DnsDuration { get; set; }

       // HTTPS
       public double HandshakeWait { get; set; }
       public DateTime HandshakeStartTime { get; set; }
       public double HandshakeDuration { get; set; }

       // socket connection
       public DateTime SocketConnectionStartTime { get; set; }
       public double SocketWait { get; set; }
       public double SocketDuration { get; set; }

       public DateTime QueueuingEndTime { get; set; }
       public double QueueingDuration { get; set; }
   }

The second one inherits from the base class and contains addition details; including the details of the redirected url if any:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

private class HttpRequestInfo : HttpRequestInfoBase
   {
       public HttpRequestInfo(DateTime timestamp, string scheme, string host, uint port, string path)
           :
           base(timestamp, scheme, host, port, path)
       {
       }

       public HttpRequestInfoBase Redirect { get; set; }

       public UInt32 StatusCode { get; set; }

       // HTTPS
       public string HandshakeErrorMessage { get; set; }

       public string Error { get; set; }
   }

A new instance of HttpRequestInfoBase is created when a 301 status code is received in HttpResponseHeaderStop handler:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

private void OnHttpResponseHeaderStop(object sender, HttpRequestStatusEventArgs e)
    {
        // used to detect redirection in .NET 8+
        if (e.StatusCode != 301)
        {
            return;
        }

        // create a new request info for the redirected request
        // because .NET 7 does not emit a Redirect event, we need to create a new request info here
        // --> it means that the redirect url will be empty in .NET 7
        var root = GetRoot(e.ActivityId);
        if (_requests.TryGetValue(root, out HttpRequestInfo info))
        {
            info.Redirect = new HttpRequestInfoBase(e.Timestamp, "", "", 0, "");

            // if you really want to have the duration of both original request + redirected request,
            // then do the following:
            //    info.ReqRespDuration = (e.Timestamp - info.ReqRespStartTime).TotalMilliseconds;
            // However, I prefer to show the duration of the redirected request only to more easily
            // compute the cost of the initial redirected request = total duration - other durations
        }
    }

For .NET 8+, the redirected url is provided to the Redirect handler and stored in the Url field of the instance created in the previous handler:

1
2
3
4
5
6
7
8
9

private void OnHttpRedirect(object sender, HttpRedirectEventArgs e)
    {
        // since this is an Info event, the activityID is the root
        var root = ActivityHelpers.ActivityPathString(e.ActivityId);
        if (_requests.TryGetValue(root, out HttpRequestInfo info))
        {
            info.Redirect.Url = e.RedirectUrl;
        }
    }

In each handler of events that could be received for both initial and redirected requests,

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

private void OnHttpResponseContentStop(object sender, EventPipeBaseArgs e)
   {
       var root = GetRoot(e.ActivityId);
       if (!_requests.TryGetValue(root, out HttpRequestInfo info))
       {
           return;
       }

       if (info.Redirect == null)
       {
           info.ReqRespDuration = (e.Timestamp - info.ReqRespStartTime).TotalMilliseconds;
       }
       else
       {
           info.Redirect.ReqRespDuration = (e.Timestamp - info.Redirect.ReqRespStartTime).TotalMilliseconds;
       }
   }

The wait time and durations are now computed as the events are received and aggregated at the end of the request by adding the value of both parts (initial and redirected if any:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

double dnsDuration = info.DnsDuration + ((info.Redirect != null) ? info.Redirect.DnsDuration : 0);
    if (dnsDuration > 0)
    {
        double dnsWait = info.DnsWait + ((info.Redirect != null) ? info.Redirect.DnsWait : 0);
        Console.Write($"{dnsWait,9:F3} | {dnsDuration,9:F3} | ");
    }
    else
    {
        Console.Write($"          |           | ");
    }

As a conclusion, you should try to monitor these redirections by using my dotnet-http CLI tool. Feel free to download it or install it with the following command line: dotnet tool install -g dotnet-http or update to the latest version with: dotnet tool update -g dotnet-http

You could also integrate some event listening code into your framework that simply handles the ResponseHeadersStop/Redirect events.

The previous episode detailed how to find the events dealing with network requests that are emitted by the BCL classes with their undocumented payload. It is now time to see how to listen to them and extract valuable insights such as what is happening when an HTTP request is sent to a server as an example. This is how my new dotnet-http CLI tool is implemented.

You are now able to see the cost of DNS, socket connection, security and redirection as shown in the following screenshot.

As you can see, the cost of an unexpected redirection could be high and hides security handshakes. In this example, using https://github.com/Maoni0 instead of http://github.com/Maoni0 divides by 2 the request duration!

Once the DNS checks are done, the socket connections are established, and the security handshakes are validated, the corresponding phases are no more needed for the next requests.

Listen to custom EventSource events

As explained in previous posts, it is easy to listen to events emitted by .NET application thanks to the Microsoft TraceEvent nuget. You also need to use EventPipeClient from the Microsoft.Diagnostics.NETCore.Client nuget to connect to the running application. In my case, I’m relying on an older version that supports both ETW and EventPipe:

1
2
3
4
5
6
7

var configuration = new SessionConfiguration(
    circularBufferSizeMB: 2000,
    format: EventPipeSerializationFormat.NetTrace,
    providers: GetProviders()
);
var binaryReader = EventPipeClient.CollectTracing(_processId, configuration, out var sessionId);
EventPipeEventSource source = new EventPipeEventSource(binaryReader);

The configuration contains a list of providers that are emitting the events you are interested in with the right keyword and verbosity. Here is what is needed for the HTTP requests monitoring :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

private static IReadOnlyCollection<Provider> GetProviders()
{
    var providers = new Provider[]
    {
        new Provider(
            name: "System.Net.Http",
            keywords: (ulong)(1),
            eventLevel: EventLevel.Verbose),

        new Provider(
            name: "System.Net.Sockets",
            keywords: (ulong)(0xFFFFFFFF),
            eventLevel: EventLevel.Verbose),

        new Provider(
            name: "System.Net.NameResolution",
            keywords: (ulong)(0xFFFFFFFF),
            eventLevel: EventLevel.Verbose),

        new Provider(
            name: "System.Net.Security",
            keywords: (ulong)(0xFFFFFFFF),
            eventLevel: EventLevel.Verbose),

        new Provider(
            name: "System.Threading.Tasks.TplEventSource",
            keywords: (ulong)(0x80),
            eventLevel: EventLevel.Verbose),

    };

    return providers;
}

You might be surprised by the presence of the TplEventSource provider but it is required to get a correct ActivityID. This major subject is detailed later.

There are so many events to listen to that it is easier to listen to the source AllEvents C# event:

1

source.AllEvents += OnEvents;

This global handler passes the events to ParseEvent that forwards the important data to handlers for each provider:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

private static Guid NetSecurityEventSourceProviderGuid = Guid.Parse("7beee6b1-e3fa-5ddb-34be-1404ad0e2520");
private static Guid DnsEventSourceProviderGuid = Guid.Parse("4b326142-bfb5-5ed3-8585-7714181d14b0");
private static Guid SocketEventSourceProviderGuid = Guid.Parse("d5b2e7d4-b6ec-50ae-7cde-af89427ad21f");
private static Guid HttpEventSourceProviderGuid = Guid.Parse("d30b5633-7ef1-5485-b4e0-94979b102068");

private void ParseEvent(
    DateTime timestamp,
    int threadId,
    Guid activityId,
    Guid relatedActivityId,
    Guid providerGuid,
    string taskName,
    Int64 keywords,
    UInt16 id,
    byte[] eventData
    )
{
    if (providerGuid == NetSecurityEventSourceProviderGuid)
    {
        HandleNetSecurityEvent(timestamp, threadId, activityId, relatedActivityId, id, taskName, eventData);
    }
    else
    if (providerGuid == DnsEventSourceProviderGuid)
    {
        HandleDnsEvent(timestamp, threadId, activityId, relatedActivityId, id, taskName, eventData);
    }
    else
    if (providerGuid == SocketEventSourceProviderGuid)
    {
        HandleSocketEvent(timestamp, threadId, activityId, relatedActivityId, id, taskName, eventData);
    }
    else
    if (providerGuid == HttpEventSourceProviderGuid)
    {
        HandleHttpEvent(timestamp, threadId, activityId, relatedActivityId, id, taskName, eventData);
    }
    else
    {
        WriteLogLine();
    }
}

The next step is to handle each event based on its id:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

private void HandleNetSecurityEvent(
    DateTime timestamp,
    int threadId,
    Guid activityId,
    Guid relatedActivityId,
    ushort id,
    string taskName,
    byte[] eventData
    )
{
    switch (id)
    {
        case 1: // HandshakeStart
            OnHandshakeStart(timestamp, threadId, activityId, relatedActivityId, eventData);
            break;
        case 2: // HandshakeStop
            OnHandshakeStop(timestamp, threadId, activityId, relatedActivityId, eventData);
            break;
        case 3: // HandshakeFailed
            OnHandshakeFailed(timestamp, threadId, activityId, relatedActivityId, eventData);
            break;
        default:
            WriteLogLine();
            break;
    }
}

Extract information from an event payload

The payload of each event has been detailed in the previous post. For example, I need to extract the following fields from the RequestStart event payload:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

private void OnRequestStart(
    DateTime timestamp,
    int threadId,
    Guid activityId,
    Guid relatedActivityId,
    byte[] eventData
    )
{
    ...
    // string scheme
    // string host
    // int port
    // string path
    // byte versionMajor
    // byte versionMinor
    // enum HttpVersionPolicy

I’ve implemented the EventSourcePayload class that provides strongly typed helpers to get the different fields one after the other:

1
2
3
4
5
6
7
8
9

public class EventSourcePayload
{
    private byte[] _payload;
    private int _pos = 0;

    public EventSourcePayload(byte[] payload)
    {
        _payload = payload;
    }

It accepts the payload as the array of bytes received by each handler.

A string information is serialized as a list of UTF-16 Unicode characters; each one stored in 2 bytes:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

public string GetString()
    {
        StringBuilder builder = new StringBuilder();
        while (_pos < _payload.Length)
        {
            var characters = UnicodeEncoding.Unicode.GetString(_payload, _pos, 2);
            _pos += 2;

            if (characters == "\0")
            {
                break;
            }
            builder.Append(characters);
        }

        return builder.ToString();
    }

The current position in the array is incremented character by character up to the final ‘\0’.

The other helpers implementation is straightforward thanks to the BitConverter class:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

public byte GetByte()
    {
        return _payload[_pos++];
    }

    public UInt16 GetUnit16()
    {
        UInt16 value = BitConverter.ToUInt16(_payload, _pos);
        _pos += sizeof(UInt16);
        return value;
    }

    public UInt32 GetUInt32()
    {
        UInt32 value = BitConverter.ToUInt32(_payload, _pos);
        _pos += sizeof(UInt32);
        return value;
    }

    public UInt64 GetUInt64()
    {
        UInt64 value = BitConverter.ToUInt64(_payload, _pos);
        _pos += sizeof(UInt64);
        return value;
    }

    public Int64 GetInt64()
    {
        Int64 value = BitConverter.ToInt64(_payload, _pos);
        _pos += sizeof(UInt64);
        return value;
    }

    public double GetDouble()
    {
        double value = BitConverter.ToDouble(_payload, _pos);
        _pos += sizeof(double);
        return value;
    }
}

Again, the position in the array is incremented to reflect the size of the field being read.

If you look at the rest of the OnRequestStart handler, you will see how each field is extracted:

1
2
3
4
5
6
7

EventSourcePayload payload = new EventSourcePayload(eventData);
    var scheme = payload.GetString();
    var host = payload.GetString();
    var port = payload.GetUInt32();
    var path = payload.GetString();
    var versionMajor = payload.GetByte();
    var versionMinor = payload.GetByte();

My activity or not my activity: that is the question

In the previous episode, I forgot (on purpose) to mention that the EventSource is keeping track of “activities” when emitting events:

1
2
3
4

protected unsafe void WriteEventCore(int eventId, int eventDataCount, EventData* data)
{
    WriteEventWithRelatedActivityIdCore(eventId, null, eventDataCount, data);
}

The WriteEventWithRelatedActivityIdCore method looks at the event metadata and if its opcode is Start then a new activity is created; if it is Stop then the current activity ends:

1
2
3
4
5
6
7
8

if (opcode == EventOpcode.Start)
{
    m_activityTracker.OnStart(m_name, metadata.Name, metadata.Descriptor.Task, ref activityId, ref relActivityId, metadata.ActivityOptions);
}
else if (opcode == EventOpcode.Stop)
{
    m_activityTracker.OnStop(m_name, metadata.Name, metadata.Descriptor.Task, ref activityId);
}

These OnStart and OnStop methods are doing nothing if the TplEventSource is not enabled with Keywords.TasksFlowActivityIds (= 0x80) set. This explains the code in GetProviders listed earlier where this non-HTTP provider is enabled.

When a request is created, the current global count managed by the ActivityTracker is incremented and it becomes the id of the current activity. Note that there is a “root” identifier before any activity gets created corresponding to the current AppDomain ID; starting at 1. If you think of an HTTP request, after a RequestStart event, each phase starts a new activity with, for example, ResolutionStart or ConnectStart. Informative events are emitted with the current request activity such as Redirect or ConnectionEstablished.

Here is a simplified view of the events emitted for an HTTP request (without DNS nor security events):

Thread +-- Path >------- ID ---- Opcode -- Event ----------------------------------
 78568 |    1/1 >  event  1 __ [ 1| Start] RequestStart
       |
 32388 |  1/1/1 >  event  1 __ [ 1| Start] ResolutionStart
       |
 32388 |  1/1/1 >  event  2 __ [ 2|  Stop] ResolutionStop
       |
 32388 |  1/1/2 >  event  1 __ [ 1| Start] ConnectStart
       |
 32388 |  1/1/2 >  event  2 __ [ 2|  Stop] ConnectStop
       |
 32388 |    1/1 >  event  4 __ [ 0|  Info] ConnectionEstablished
       |
 53324 |    1/1 >  event  6 __ [ 0|  Info] RequestLeftQueue
       |
 53324 |  1/1/3 >  event  7 __ [ 1| Start] RequestHeadersStart
       |
 53324 |  1/1/3 >  event  8 __ [ 2|  Stop] RequestHeadersStop
       |
 68024 |  1/1/4 >  event 11 __ [ 1| Start] ResponseHeadersStart
       |
 68024 |  1/1/4 >  event 12 __ [ 2|  Stop] ResponseHeadersStop
       |
 68024 |  1/1/5 >  event 13 __ [ 1| Start] ResponseContentStart
       |
 68024 |  1/1/5 >  event 14 __ [ 2|  Stop] ResponseContentStop
       |
 68024 |    1/1 >  event  2 __ [ 2|  Stop] RequestStop
  200 <|

As you can see, different threads are emitting events associated to the same request. At the ActivityTracker level, an ActivityInfo is stored in an async local so that each thread has its own storage that will be propagated by the methods of the Task Parallel Library (a.k.a. TPL) from task to task. This is why the very asynchronous code of the HTTP client implementation can go back to the current activity from different threads.

The activities are encoded into the 16 bytes of a GUID. In fact, only the first 12 bytes are used, and the final 4 bytes contain a checksum that includes the current process ID. Since the activity identifiers in the path do, most of the time, have a small value, the encoding is using 4 bits, also known as a nibble, to encode each of them. There was a bug in ActivityTracker.AddIdToGuid that will be fixed in .NET 10. Unfortunately, the decoding code in Perfview needs to take it into account so the last activity is not lost.

Computing the different durations

With this infrastructure in place, it is now possible to extract the “root” activity path corresponding to an HTTP request during each phase and update the corresponding state that is used to output the details when a request ends:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

private class HttpRequestInfo
{
    public HttpRequestInfo(DateTime timestamp, Guid activityId, string scheme, string host, uint port, string path)
    {
        Root = ActivityHelpers.ActivityPathString(activityId);
        if (port != 0)
        {
            Url = $"{scheme}://{host}:{port}{path}";
        }
        else
        {
            Url = $"{scheme}://{host}:{path}";
        }
        StartTime = timestamp;
    }

    public string Root { get; set; }
    public string Url { get; set; }
    public string RedirectUrl { get; set; }
    public DateTime StartTime { get; set; }
    public DateTime ReqRespStartTime { get; set; }
    public double ReqRespDuration { get; set; }

    public UInt32 StatusCode { get; set; }

    // DNS
    public DateTime DnsStartTime { get; set; }
    public double DnsDuration { get; set; }

    // HTTPS
    public DateTime HandshakeStartTime { get; set; }
    public double HandshakeDuration { get; set; }
    public string HandshakeErrorMessage { get; set; }

    // socket connection
    public DateTime SocketConnectionStartTime { get; set; }
    public double SocketDuration { get; set; }
}

As you have seen earlier, the different phases of a request may be processed by different threads. It means that an available thread needs to be found in order to execute them. If the thread pool is busy, you can expect some wait time. This is shown in the different wait sections between the phases. They are easily computed using the timestamp of each event. Having long wait durations might be reduced by increasing the number of threads in the thread pool.

Feel free to use the new dotnet-http CLI tool available from nuget.org or via dotnet tool install -g dotnet-http.

The corresponding sources are available from my github repository in case you would like to integrate this kind of analysis directly in your code or your monitoring pipeline.

Happy monitoring!

Testing the statistical results

In parallel of the performance impact, it is important to validate the expected statistical distribution of the sampled allocations. Basically, I need to execute the same run of allocations multiple times in a row. Each run allocates the same number of instances of different types. For example, it is interesting to know if sampling instances of types with sizes proportional to a base value gives good results. Same question for totally different sized types or with Finalizers.

I would like to pass the number of runs to execute and a given scenario to a C# runner program and listen to the emitted events in another C# listener.

I’m facing 3 issues here:

• How many instances are allocated to validate the upscaling algorithm (sampled vs real count)
• What are the types I want to focus on because I don’t want to hard code them in the listener application.
• When does each run start?

It would be great if I could send the answer to these questions via events so the listener would know at runtime. Well… This is exactly what a class inherited from EventSource allows you to do!

In the runner application, I’ve defined the AllocationsRunEventSource that is decorated with the EventSource attribute to set its name that will be used as a provider name like Microsoft-Windows-DotNETRuntime for the .NET runtime provider.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

[EventSource(Name = "Allocations-Run")]
public class AllocationsRunEventSource : EventSource
{
    public static readonly AllocationsRunEventSource Log = new AllocationsRunEventSource();

The four implemented methods are defining which event ID for which verbosity will be emitted with which payload:
    [Event(600, Level = EventLevel.Informational)]
    public void StartRun(int iterationsCount, int allocationCount, string listOfTypes)
    {
        WriteEvent(eventId: 600, iterationsCount, allocationCount, listOfTypes);
    }

    [Event(601, Level = EventLevel.Informational)]
    public void StopRun()
    {
        WriteEvent(eventId: 601);
    }

    [Event(602, Level = EventLevel.Informational)]
    public void StartIteration(int iteration)
    {
        WriteEvent(eventId: 602, iteration);
    }

    [Event(603, Level = EventLevel.Informational)]
    public void StopIteration(int iteration)
    {
        WriteEvent(eventId: 603, iteration);
    }
}

To make the payload serialization and parsing easy, the list of types that will be allocated is passed as a string with the following format allocatedTypes = “Object24;Object48;Object72;Object32;Object64;Object96”.

The code of the runner calls these methods as expected at different moment of the execution:

1
2
3
4
5
6
7
8

AllocationsRunEventSource.Log.StartRun(iterations, allocationsCount, allocatedTypes);
 for (int i = 0; i < iterations; i++)
 {
     AllocationsRunEventSource.Log.StartIteration(i);
     allocationsRun.Allocate(allocationsCount);
     AllocationsRunEventSource.Log.StopIteration(i);
 }
 AllocationsRunEventSource.Log.StopRun();

Instead of recording the events with dotnet-trace, this time I’m using TraceEvent and Microsoft.Diagnostics.NETCore.Client to code a listener application. The code is very similar to what was presented for my dotnet-fullgc CLI tool except that I’m enabling the AllocationsRun provider corresponding to the event source of the runner in addition to the .NET runtime one:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

public static void PrintEventsLive(int processId)
{
    var providers = new List<EventPipeProvider>()
    {
        new EventPipeProvider(
                "Microsoft-Windows-DotNETRuntime",
                EventLevel.Verbose, // verbose is required for AllocationTick
                (long)0x80000000001 // new AllocationSamplingKeyword + GCKeyword
                ),
        new EventPipeProvider(
                "Allocations-Run",
                EventLevel.Informational
                ),
    };

The custom events from that provider are received via the source.Dynamic.All C# event:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

var client = new DiagnosticsClient(processId);
    using (var session = client.StartEventPipeSession(providers, false))
    {
        Console.WriteLine();

        Task streamTask = Task.Run(() =>
        {
            var source = new EventPipeEventSource(session.EventStream);
            _source = source;
            ClrTraceEventParser clrParser = new ClrTraceEventParser(source);
            clrParser.GCAllocationTick += OnAllocationTick;
            source.Dynamic.All += OnEvents;
            ...

Because TraceEvent is not already aware of the new AllocationSampled event emitted by the PR code, it will also be received via the same OnEvent handler:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

private static void OnEvents(TraceEvent eventData)
{
    if (eventData.ID == (TraceEventID)303)
    {
        // AllocationSampled parsing 
        ...
        return;
    }

    if (eventData.ID == (TraceEventID)600)  // Start run
    {
        // keep track of the expected types and the number of allocated instances 
        ...
        return;
    }

    if (eventData.ID == (TraceEventID)601)  // Stop run
    {
        // show the results of the run
        ...
        return;
    }

    if (eventData.ID == (TraceEventID)602)  // Start an iteration in a run
    {
        // reset for a new iteration
        ...
        return;
    }

    if (eventData.ID == (TraceEventID)603)  // Stop an iteration in a run
    {
        // Show iteration results
        ...
        return;
    }
}

The parsing of the payload of the run related events is done the same way as for AllocationSampled by a dedicated xxxData class:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

class AllocationsRunData
{
    const int EndOfStringCharLength = 2;
    private TraceEvent _payload;

    public AllocationsRunData(TraceEvent payload)
    {
        _payload = payload;

        ComputeFields();
    }

    public int Iterations;
    public int Count;
    public string AllocatedTypes;

    private void ComputeFields()
    {
        int offsetBeforeString = 4 + 4;

        Span<byte> data = _payload.EventData().AsSpan();
        Iterations = BitConverter.ToInt32(data.Slice(0, 4));
        Count = BitConverter.ToInt32(data.Slice(4, 4));
        AllocatedTypes = Encoding.Unicode.GetString(data.Slice(offsetBeforeString, _payload.EventDataLength - offsetBeforeString - EndOfStringCharLength));
    }
}

By keeping track of this data, it is possible to show each iteration results:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

> starts 100 iterations allocating 1000000 instances
0|
Tag  SCount  TCount          SSize          TSize   UnitSize     UpscaledSize  UpscaledCount  Name
--------------------------------------------------------------------------------------------------
 ST     247     384           5928           9216         24         24702711        1029279  Object24
 ST     322     106          10304           3392         32         32205122        1006410  Object32
 ST     435     509          20880          24432         48         43510266         906463  Object48
 ST     587     776          37568          49664         64         58718825         917481  Object64
 ST     747     481          53784          34632         72         74726662        1037870  Object72
 ST     958     916          91968          87936         96         95845392         998389  Object96

that integrates AllocationTick numbers too.

At the end of the run, error distribution per type is also computed over the iterations:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

Object72
-------------------------
   1  -10.5 %
   2   -8.9 %
   3   -8.4 %
   4   -7.8 %
   5   -6.4 %
        ...
  49    0.2 %
  50    0.2 %
  51    0.2 %
        ...
  96    6.8 %
  97    6.8 %
  98    7.7 %
  99    8.6 %
 100   10.0 %

You could use the same mechanisms (a custom EventSource to emit additional information and an EventPipe listener to aggregate the data) for your own usage. This is a different way to use EventSource rather than emitting events for monitoring like what is done by the BCL.

Testing the standalone GC

In addition to the usual .NET GC, it is needed to validate that the changes are also working for the standalone GC. Long story short, it is possible to replace the existing .NET garbage collector by your own implementation. For the test, I needed to check that the standalone GC clrgcexp.dll generated by the .NET compilation generates the expected AllocationSampled events when the corresponding keyword with informational verbosity is enabled.

Debugging NativeAOT scenarios

The final step was to implement the feature for the NativeAOT scenario. When you build your C# application for NativeAOT, a lot happens behind the scenes, based on compilers known by Visual Studio corresponding to the official released version of the .NET runtime. In my case, I needed to use the brand-new code of my local branch and debug some simple C# applications. The steps to reach that goal are not that simple.

First, you follow the steps given by the documentation to build AOT CLR in debug and libs in release:

1
2

build clr.aot+lib -rc debug -lc release
build -c release

Then, open src\coreclr\tools\aot\ilc.sln

the repro project contains a program.cs file where you write the C# code you want to test and debug.

When you build a NativeAOT application, you need to select if you want the runtime that emits events or not. This is done by setting the EventSourceSupport to true in the .csproj:

Add the following in the .csproj to get events:

1

<EventSourceSupport>true</EventSourceSupport>

However, with the repro project, you need to change ILCompiler.csproj in a different way:

1

<ReproResponseLines Include="--feature:System.Diagnostics.Tracing.EventSource.IsSupported=true" />

Also, change the reproNative.vcxproj file to bind to eventpipe-enabled.lib instead of eventpipe-disabled.lib for the platform/configuration you want to debug:

1
2
3
4
5
6

<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
    ...
    <Link>
      ...
      <AdditionalDependencies>...$(ArtifactsRoot)bin\coreclr\windows.x64.Debug\aotsdk\eventpipe-enabled.lib;...</AdditionalDependencies>
    </Link>

Then, build repro in Debug x64

Next, change the target for the ILCompiler project:

Build and run it to generate the .obj file corresponding to the repro project

Finally, open src\coreclr\tools\aot\ILCompiler\reproNative\reproNative.vcxproj that will allow you to debug the program.cs you’ve just built!

Introduction

If you have read Microsoft documentation, you probably know that it is not recommended to trigger a garbage collection in your application code. However, in some troubleshooting cases, you might want to trigger a GC. For example, you don’t want to wait for a full gen2 compacting GC to figure out if your application is really leaking memory. For web applications, you can imagine having a hidden HTTP end point that simply call GC.Collect. What if you could simply call a command line tool to trigger a GC in any .NET application? This is exactly what my new dotnet-fullgc CLI tool is doing!

This Is The Way

If you have read a few of my past posts about the .NET diagnostics mechanisms, you know that you can send commands to another .NET process via EventPipes. Well… there is no explicit command to trigger a GC.

I have also explained how you could listen to events emitted by the CLR by enabling a provider with a set of keywords with a verbosity corresponding to the events you are interested in. This is how dotnet-trace and Perfview are collecting these events. If you want to trigger a GC, you simply need to enable the Microsoft-Windows-DotNETRuntime provider with the GCHeapCollect (= 0x800000L) keyword and an informal verbosity. Yes: it is as simple as that, and it is also working for .NET Framework!

So, you could trigger a GC with dotnet-trace via the following command line:

1

dotnet trace collect -p <process id> - providers Microsoft-Windows-DotNETRuntime:0x800000:4 - duration 00:00:01

However, a .nettrace file would be generated and it is not possible to pass parameters (more on this later).

The rest of this post shows the C# code to obtain the same result. With Microsoft.Diagnostics.NETCore.Client and TraceEvent, you create a DiagnosticsClient with the ID of the process you are interested in:

1

var client = new DiagnosticsClient(processId);

The next step is to start an EventPipe session with the right provider, keyword and verbosity:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

var providers = new List<EventPipeProvider>()
{
    new EventPipeProvider(
        "Microsoft-Windows-DotNETRuntime",
        EventLevel.Informational,
        (long)ClrTraceEventParser.Keywords.GCHeapCollect,
        Arguments  // more on this later
        ),
};
using (var session = client.StartEventPipeSession(providers, false))
{

The source must be processed in another thread to avoid blocking the main thread:

1
2
3
4
5
6
7

Task streamTask = Task.Run(() =>
    {
        // without source to process, session.Stop() will not return
        var source = new EventPipeEventSource(session.EventStream);

        source.Process();
    });

The question to answer is how to stop the session: just create another task that waits for a second before stopping the session to exit from the Process call:

1
2
3
4
5
6
7
8

Task inputTask = Task.Run(() =>
    {
        Thread.Sleep(1000);
        session.Stop();
    });

    Task.WaitAny(streamTask, inputTask);
}

That’s all!

Pitfalls

Unfortunately, I faced a couple of issue during the implementation of the tool.

What is your number?

If you look at the TraceEvent implementation, you could imagine that is it possible to pass a GC ID as a parameter to the .NET provider:

1
2
3
4
5

//
// Summary:
//     Triggers a GC. Can pass a 64 bit value that will be logged with the GC Start
//     event so you know which GC you actually triggered.
GCHeapCollect = 0x800000L,

Unfortunately, this is not supported and the reasons are explained below.

If you check the .NET source code and look at EtwCallbackCommon(), you can indeed see that a numeric ID can be passed to ETW::GCLog::ForceGC(l64ClientSequenceNumber) and passed as ID in GCStart event instead of the one incrementally increased collection after collection.

At the diagnostics client level, you have the opportunity to pass a dictionary of key/value string pairs. This dictionary is used when defining the EventPipeProvider to be enabled:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

Dictionary<string, string> arguments = new Dictionary<string, string>();
arguments.Add("Id", "42");
var providers = new List<EventPipeProvider>()
{
    new EventPipeProvider(
        "Microsoft-Windows-DotNETRuntime",
        EventLevel.Informational,
        (long)ClrTraceEventParser.Keywords.GCHeapCollect,
        arguments
        ),
};

The diagnostics client transforms the dictionary into a string with key=value pairs separated by ‘;’ such as “Id=42;AnotherId=AnotherValue;…” and serializes it as is in the payload when sending a CollectTraces command.

The question is what identifier is expected by the CLR? To answer this question, you need to look at the code of provider_invoke_callback. The “key=value;…” string is stored into a buffer and each = and ; characters are transformed into \0. So, “Id=42” is transformed into Id\042\0.

The next step is done by ep_event_filter_desc_init() that uses that buffer to fill up the 3 fields of an EventFilterDescriptor:

1
2
3

uint64_t ptr = address of the buffer
uint32_t size = size of the buffer (=6 for id\042\0)
uint32_t type = 0

And finally, EtwCallbackCommon receives the filterdata and tries the following to get the collection id:

1
2
3
4
5
6
7

PEVENT_FILTER_DESCRIPTOR FilterData = (PEVENT_FILTER_DESCRIPTOR)pFilterData;
if ((FilterData != NULL) &&
   (FilterData->Type == 1) &&
   (FilterData->Size == sizeof(l64ClientSequenceNumber)))
{
    l64ClientSequenceNumber = *(LONGLONG *) (FilterData->Ptr);
}

As you can see, it will fail because:

• the received type value is 0
• the received size is not the size of a 64 bit number
• the ptr field does not point to the value as 64 bit number (but as a string)

An issue has been filed with a possible fix.

Only one collection please!

When I test my dotnet-fullgc with dotnet-gcstats, I always see 2 collections!

We investigated with my colleague Kevin Gosse and he created an issue for that. The EtwCallback() function is called whenever a session is enabled or disabled. Unfortunately, the call to ForceGC is made in both cases: so, when the session is stopped, a second garbage collection is triggered.

Next steps

Feel free to install dotnet-fullgc on your machine with dotnet tool install -g dotnet-fullgc.

Next, use dotnet fullgc to trigger two gen2 full garbage collections in your running .NET processes.

Introduction

While working on the second edition of Pro .NET Memory Management, it was needed to get statistics about each garbage collection to explain the condemned generation and other decisions taken by the GC. This post explains the different internal data structures used by the GC and how to get their value for each collection. Some require debugging the CLR and others are emitted via events. For the latter, I will show how I wrote the new dotnet-gcstats CLI tool to collect them and a personal Perfview GCStats displaying live data, garbage collection after garbage collection.

High level view of GC Internals

With regions, the GC keeps track of managed memory allocated by your application in instances of the gc_heap class. In Workstation mode, only 1 instance exists and in Server mode, by default, 1 instance is created per core. Each gc_heap keeps track of its 5 generations (gen0, gen1, gen2, Large Object Heap and Pinned Object Heap) in an array of 5 generation instances. Each generation references its dedicated regions wrapped by instances of heap_segment. These regions are reserved from a giant part of the process address space and committed as needed.

During a garbage collection, the GC code relies on global fields per gc_heap:

1
2
3
4

static gc_mechanisms settings;
gc_history_global gc_data_global;  // for non background GC including foreground GC during a background
gc_history_global bgc_data_global; // for background GC only
static dynamic_data dynamic_data_table[total_generation_count = 5];

The settings field contains a few interesting fields:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

class gc_mechanisms
{
public:
    gc_index; // starts from 1 for the first GC
    int condemned_generation;  // generation to collect
    BOOL compaction;  // true when compaction instead of sweep
    BOOL loh_compaction;  // true when LOH needs compaction
    uint32_t concurrent;  // 1 = concurrent/background GC 
    gc_reason reason;  // trigger reason
    gc_pause_mode pause_mode;  // see GCSettings.LatencyMode
    ...
};

including the trigger reason that will tell if your code called GC.Collect (i.e. induced), or if it was due to a LOH or SOH allocation for example. If compaction is true, a compacting GC will happen (instead of a sweeping one).

The gc/bgc_data_global contains almost the same information:

1
2
3
4
5
6
7
8
9

struct gc_history_global
{
    uint32_t num_heaps;  // number of gc_heap instances
    int condemned_generation;
    gc_reason reason;
    int pause_mode;
    uint32_t mem_pressure; 
    uint32_t global_mechanisms_p;
};

Most of the fields are available from different events:

• GCStart: gc_index in Count, condemned_generation in Depth, reason in Reason
• GCGlobalHeapHistory: pause_mode in PauseMode and some others in GlobalMechanisms

Which generation to collect = condemned generation

The computation of the condemned_generation is complicated and relies on many factors including metrics stored for each “generation” (gen0, gen1, gen2, LOH and POH) in an array of dynamic_data called dynamic_data_table. The dynamic_data class contains a few fields used by the GC to take decisions such as when a collection should be triggered and which generation to condemn:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

class dynamic_data
{
public:
    ptrdiff_t new_allocation;     // remaining budget = budget - allocated
    size_t    desired_allocation; // budget to trigger a GC

    // # of bytes taken by survived objects after mark.
    size_t    survived_size;

    // # of bytes taken by survived pinned plugs after mark.
    size_t    pinned_survived_size;

    // total object size after a GC, ie, doesn't include fragmentation
    size_t    current_size;
    size_t    promoted_size;
    size_t    fragmentation;
};

Most of these fields are found in the payload of GCPerHeapHistory or GCHeapStat events. However, the most interesting one, new_allocation is not available. Why is it interesting? Because it would give you which generation had its budget exceeded. It is initialized with the generation budget at the end of a GC and then, each time an allocation context gets created, its size is deducted from it. When it reaches 0, it means that the budget is exceeded, and a collection should happen.

Since I needed to debug the CLR to better understand all these algorithms, I added a breakpoint at the beginning of gc_heap::garbage_collect with the following action:

#{settings.gc_index}[{gc_trigger_reason}]{"\n",s8b} new_allocation(0) = {dynamic_data_table[0].new_allocation}{"\n",s8b} desired_allocation(0) = {dynamic_data_table[0].desired_allocation}{"\n",s8b} begin_data_size(0) = {dynamic_data_table[0].begin_data_size}{"\n",s8b} promoted_size(0) = {dynamic_data_table[0].promoted_size}{"\n",s8b}-{"\n",s8b} new_allocation(1) = 
...
{dynamic_data_table[4].new_allocation}{"\n",s8b} desired_allocation(4) = {dynamic_data_table[4].desired_allocation}{"\n",s8b} begin_data_size(4) = {dynamic_data_table[4].begin_data_size}{"\n",s8b} promoted_size(4) = {dynamic_data_table[4].promoted_size}{"\n",s8b}__________{"\n",s8b}

And now, each time a GC happens, I get the corresponding log in my Output pane in Visual Studio:

#2[reason_alloc_soh (0)]
 new_allocation(0) = -22728
 desired_allocation(0) = 134217728
 begin_data_size(0) = 8391376
 promoted_size(0) = 8383432
-
 new_allocation(1) = -5910416
 desired_allocation(1) = 2473016
 begin_data_size(1) = 375528
 promoted_size(1) = 353288
-
 new_allocation(2) = -91144
 desired_allocation(2) = 262144
 begin_data_size(2) = 0
 promoted_size(2) = 0
-
 new_allocation(3) = 28000088
 desired_allocation(3) = 28000088
 begin_data_size(3) = 8000024
 promoted_size(3) = 8000024
-
 new_allocation(4) = 3145728
 desired_allocation(4) = 3145728
 begin_data_size(4) = 32712
 promoted_size(4) = 32712

As you can see, gen0, gen1 and gen2 have all their budget exceeded (i.e. their new_allocation is negative) and it explains why a simple gen0 collection (from allocation in SOH = gen0) becomes a gen2 collection. If you wonder how gen1 and gen2 budgets are exceeded as your application is only allocating in gen0, you need to understand that when a GC copy surviving objects from one younger generation to the older, they are counted as allocations in the older and subtracted from its new_allocation metric.

The GC is encoding the different steps leading to the final condemned generation in a 32 bit value stored in a gen_to_condemn_tuning field that allows you to get:

• initial condemned generation,
• final generation to condemn,
• which generation’s budget is exceeded.

The value of the last one corresponds to the highest generation for which its new_allocation was negative.

This information is available in the CondemnReasons0 field of the GCPerHeapHistory event, and you need some arithmetic to get the generation you want:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

private const int gen_initial = 0;          // indicates the initial gen to condemn.
private const int gen_final_per_heap = 1;   // indicates the final gen to condemn per heap.
private const int gen_alloc_budget = 2;     // indicates which gen's budget is exceeded.

private const int InitialGenMask = 0x0 + 0x1 + 0x2;

static int GetGen(int val, int reason)
{
    int gen = (val >> 2 * reason) & InitialGenMask;
    return gen;
}

Building your own tool

Even though I could dig into the different matrices available in the Perfview’s GCStats view or its export to Excel, I decided to write dotnet-gcstats. This CLI tool listens to the CLR events emitted by a .NET application thanks to Microsoft.Diagnostics.NETCore.Client (connect to the application EventPipe) and TraceEvent (receive and analyze the CLR events).

The code is amazingly simple:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

var providers = new List<EventPipeProvider>()
{
    new EventPipeProvider("Microsoft-Windows-DotNETRuntime",
        EventLevel.Informational, (long)ClrTraceEventParser.Keywords.GC),
};
var client = new DiagnosticsClient(processId);

using (var session = client.StartEventPipeSession(providers, false))
{
    Console.WriteLine();

    Task streamTask = Task.Run(() =>
    {
        var source = new EventPipeEventSource(session.EventStream);

        ClrTraceEventParser clrParser = new ClrTraceEventParser(source);
        clrParser.GCPerHeapHistory += OnGCPerHeapHistory;
        clrParser.GCStart += OnGCStart;
        clrParser.GCGlobalHeapHistory += OnGCGlobalHeapHistory;

        try
        {
            source.Process();
        }
        catch (Exception e)
        {
            ShowError($"Error encountered while processing events: {e.Message}");
        }
    });

Each event handler is responsible for extracting and translating the interesting fields of its event payload with a few color enhancements:

• GCStart: collection count and reason (highlight induced collections).
• GCGlobalHeapHistory: condemned generation, pause mode and memory pressure.
• GCPerHeapHistory: starting -> final condemned generation and for each heap, budget, begin size, begin obj size, final size, promoted size and fragmentation.

The final step was to transform a simple console application into a .NET CLI tool that everyone will be able to install with dotnet tool install -g dotnet-gcstats and use with dotnet gcstats . I followed the documentation by adding the following to the project file:

1
2
3
4
5
6

<PropertyGroup>
    <PackAsTool>true</PackAsTool>
    <ToolCommandName>dotnet-gcstats</ToolCommandName>
    <PackageOutputPath>./nupkg</PackageOutputPath>
    <GeneratePackageOnBuild>true</GeneratePackageOnBuild>
  </PropertyGroup>

In addition, I provided a few additional details:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

<PropertyGroup>
    <PackageId>dotnet-gcstats</PackageId>
    <PackageVersion>1.0.0</PackageVersion>
    <Title>dotnet-gcstats</Title>
    <Authors>christophe Nasarre</Authors>
    <Owners>chrisnas</Owners>
    <RepositoryUrl>https://github.com/chrisnas</RepositoryUrl>
    <RepositoryType>git</RepositoryType>
    <PackageProjectUrl>https://github.com/chrisnas/GCStats</PackageProjectUrl>
    <PackageLicenseFile>LICENSE</PackageLicenseFile>
    <Description>Global CLI tool to display live statistics during .NET garbage collections</Description>
    <PackageReleaseNotes>Initial release</PackageReleaseNotes>
    <Copyright>Copyright Christophe Nasarre 2024-$([System.DateTime]::UtcNow.ToString(yyyy))</Copyright>
    <PackageTags>.NET TraceEvent CLR GC</PackageTags>
  </PropertyGroup>

Once built, I simply uploaded the generated package to nuget.org et voila!

Now, you should be able to better understand why some collections are triggered:

And if it is not enough, wait for reading the second edition of Pro .NET Memory Management ;^)

As shown in the previous post, the processing of ProcessInfo diagnostic commands is easy because you send a request and read the different fields from the response. This is different if you want to receive events from the CLR via EventPipe. In C#, the TraceEvent nuget package wraps everything under a nice event handler based model as shown in many of my previous posts.

Behind the scene, a StartSession command is sent (more details about the parameters later) and the response contains the numeric ID of the session. Then, the events will be read from the IPC channel as a binary stream of data with the “nettrace“ file format. The collection ends when the StopTracing command is sent.

The source code is available from my github repository.

Hidding the transport layer: IIpcEndoint

Unlike the previous post, to send the command and read the response back from the CLR , I’m wrapping the transport layer with the IIpcEndpoint interface:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

class IIpcEndpoint
{
public:
    virtual bool Write(LPCVOID buffer, DWORD bufferSize, DWORD* writtenBytes) = 0;
    virtual bool Read(LPVOID buffer, DWORD bufferSize, DWORD* readBytes) = 0;
    virtual bool ReadByte(uint8_t& byte) = 0;
    virtual bool ReadWord(uint16_t& word) = 0;
    virtual bool ReadDWord(uint32_t& dword) = 0;
    virtual bool ReadLong(uint64_t& ulong) = 0;

    virtual bool Close() = 0;
    virtual ~IIpcEndpoint() = default;
};

It abstracts the write and read accesses to the underlying transport layer. In addition, the base class accepts a “recorder” that allows me to store what is received from the CLR into any kind of storage (today only a file-based recorder that helped a lot to reproduce specific situations without the need to have a running process to connect to):

The PidEndpoint class accepts the process id of the running .NET application to monitor its CLR events and an optional recorder implementing the IIpcRecorder interface. The Create static factory implementation creates the expected named pipe on Windows (or the domain socket on Linux) and stores the handle into its _handle field:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

PidEndpoint* PidEndpoint::CreateForWindows(int pid, IIpcRecorder* pRecorder)
{
    PidEndpoint* pEndpoint = new PidEndpoint(pRecorder);

    // build the pipe name as described in the protocol
    wchar_t pszPipeName[256];
    int nCharactersWritten = -1;
    nCharactersWritten = wsprintf(
        pszPipeName,
        L"\\\\.\\pipe\\dotnet-diagnostic-%d",
        pid
    );

    // check that CLR has created the diagnostics named pipe
    if (!::WaitNamedPipe(pszPipeName, 200))
    {
        auto error = ::GetLastError();
        std::cout << "Diagnostics named pipe is not available for process #" << pid << " (" << error << ")" << "\n";
        return nullptr;
    }

    // connect to the named pipe
    HANDLE hPipe;
    hPipe = ::CreateFile(
        pszPipeName,    // pipe name
        GENERIC_READ |  // read and write access
        GENERIC_WRITE,
        0,              // no sharing
        NULL,           // default security attributes
        OPEN_EXISTING,  // opens existing pipe
        0,              // default attributes
        NULL);          // no template file

    if (hPipe == INVALID_HANDLE_VALUE)
    {
        std::cout << "Impossible to connect to " << pszPipeName << "\n";
        return nullptr;
    }

    pEndpoint->_handle = hPipe;
    return pEndpoint;
}