In the previous article, I presented what is needed (i.e. listen to WaitHandleWait events) to compute lock/wait durations and call stacks for Mutex, Semaphore, SemaphoreSlim, Manual/AutoResetEvent, ManualResetEventSlim, ReaderWriterLockSlim .NET synchronization constructs for a running process.

However, since the application is already running, some JIT-related events are missing, and some frames of the call stacks cannot be symbolized. Also, it would be great to monitor an application’s startup to see if it could be faster.

This post will detail how to monitor a .NET application since the very beginning of its life and the issues you might face.

Preparing a new .NET process to be monitored

From .NET 5, the dotnet-trace CLI tool allows you to pass a command line to execute and trace it from startup. In a very interesting article, Olivier Coanet presented the gory details about how to tell the .NET runtime to start an application in a pseudo-suspended mode as shown in the following diagram:

The first step is to create a ReverseDiagnosticsServer instance with a specific port (i.e. dotnet-wait_1234 in the diagram). Next, the process to monitor is spawned with the DOTNET_DiagnosticPorts environment variable set to the same port (i.e. dotnet-wait_1234). Look at the Diagnostics documentation of the Diagnostic Ports with DOTNET_DiagnosticPorts environment variable for more details. The .NET runtime is the new process will listen to this port and… wait.

When the tool is ready, it sends a resume command via a DiagnosticsClient: from that point in time, the CLR executes the normal flow of actions to run the application and… you will receive all events without missing one!

Get my command line please

Following the dotnet-trace example, my dotnet-wait tool accepts the command line of the child process in its final arguments that follow the** — **trigger. For example, dotnet-wait — dotnet foo.dll will start the program in foo.dll by using dotnet.exe. I’m reusing the code in ReversedServerHelper.cs to deal with arguments containing spaces:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

else if (current == "--")  // this is supposed to be the last one
{
    i++;
    if (i < args.Length)
    {
        parameters.pathName = args[i];

        // use the remaining arguments as the arguments for the child app to spawn
        i++;
        if (i < args.Length)
        {
            parameters.arguments = "";
            for (int j = i; j < args.Length; j++)
            {
                if (args[j].Contains(' '))
                {
                    parameters.arguments += $"\"{args[j].Replace("\"", "\\\"")}\"";
                }
                else
                {
                    parameters.arguments += args[j];
                }

                if (j != args.Length)
                {
                    parameters.arguments += " ";
                }
            }
        }

        // no need to look for more arguments
        break;
    }
    else
    {
        throw new InvalidOperationException($"Missing path name value...");
    }
}

The code to spawn the child process is simple:

1
2
3
4
5
6
7
8
9

// start the monitored app
var psi = new ProcessStartInfo(pathName);
if (!string.IsNullOrEmpty(arguments))
{
    psi.Arguments = arguments;
}
psi.EnvironmentVariables["DOTNET_DiagnosticPorts"] = port;
psi.UseShellExecute = false;
var process = System.Diagnostics.Process.Start(psi);

Here is an example with the following prompt:

-- dotnet "C:\CommandLineTest.dll" one two "t h r e e" four 'five six'

that generates the output (the test application is just listing its arguments):

dotnet-wait v1.0.0.0 - List wait duration
by Christophe Nasarre

Press ENTER to exit...
6 arguments
   1 | one
   2 | two
   3 | t h r e e
   4 | four
   5 | 'five
   6 | six'

This test reminded me to never use simple quotes in prompts :^)

It’s my console!

Once I implemented these steps, I immediately faced a very simple problem: my dotnet-wait tool and the test application are console applications. It means that they will share the same console for both input and output. For example, both are waiting for the RETURN key to (1) stop for the tool and (2) start for the test application: too bad for me because the tool will stop as soon the application starts…

Going back in time in my Windows memories, I remembered that the Win32 CreateProcess API accepts CREATE_NEW_CONSOLE as creation flag to automagically start the child process into its own new console. Unfortunately, it is not possible to pass this flag in .NET; maybe a limitation due to Linux support.

One simple solution could be to redirect the output of the tool or the application to a file: that would avoid mixing them in the console. Note that, by default, dotnet-trace discards output from the child process (by setting RedirectStandardOutput, RedirectStandardError and RedirectStandardInput to false and by ignoring the error and output streams) except if you pass — show-child-io on the command line. In this case, no output for dotnet-trace.

I decided to do the opposite for dotnet-wait: by default, you also get the child output but you can redirect the output of the tool to a file with -o

. Still, this does not solve the input problem in case of common expected keys.

If you remember the interactions between the tool and the monitored application, the latter is suspended until DiagnosticsClient::ResumeRuntime is called. So, why not starting the tool that spawns the application in one console and another instance of the tool in a new console that will resume the application? This is exactly what my friend Kevin Gosse imagined and how dotnet-wait works.

After the timeout that you give to diagnosticsServer.AcceptAsync(cancellation.Token) has elapsed, the runtime in the child process will display the following message:

The runtime has been configured to pause during startup and is awaiting a Diagnostics IPC ResumeStartup command from a Diagnostic Port.
DOTNET_DiagnosticPorts="dotnet-wait_34296"
DOTNET_DefaultDiagnosticPortSuspend=0

And this is exactly what the -r 34296 parameter will do!

You can now install dotnet wait and monitor the lock and wait contentions of your .NET9+ applications.

Introduction

In an old post, I detailed how to use ContentionStart and ContentionStop events to measure the lock contentions duration for a .NET application. In a .NET 9 pull request, a former Criteo’s colleague Grégoire Verdier has added new events to be notified when wait time similar to lock contention is happening for Mutex, Semaphore, Manual/AutoResetEvent. Read his post for more details about what he was trying to investigate.

With asynchronous and multi-threaded algorithms, it is essential to detect unexpected wait/locks in our applications. This post shows you how to leverage these events to measure the duration of these waits and get the call stack when the wait started:

New WaitHandleWait events

These new events are emitted by the Microsoft-Windows-DotNETRuntime CLR provider when you enable the WaitHandle (= 0x40000000000) keyword with Verbose verbosity. Each time WaitOne is called on a waitable object and this object is already owned, a WaitHandleWaitStart event is emitted. When the object is released, a WaitHandleWaitStop event is emitted.

For example, the following code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

static Mutex mutex = new Mutex();

static void Main()
{
    var owningThread = new Thread(OwningThread);
    owningThread.Start();
    var mutexThread = new Thread(MutexThread);
    mutexThread.Start();

    owningThread.Join();
    mutexThread.Join();
}

static void OwningThread()
{
    Console.WriteLine($"    [{GetCurrentThreadId(), 8}] Start to hold resources");
    Console.WriteLine("___________________________________________");
    mutex.WaitOne();

    Thread.Sleep(3000);  // the wait should last ~3 seconds

    Console.WriteLine("    Release resources");
    mutex.ReleaseMutex();
}

static void MutexThread()
{
    Console.WriteLine($"    [{GetCurrentThreadId(), 8}] waiting for Mutex...");
    mutex.WaitOne();  // events are emitted in the implementation when a contention happens
    mutex.ReleaseMutex();
    Console.WriteLine("    <-- Mutex");
}

generates a Start and Stop events pair:

1
2

125980 | 00000000-0000-0000-0000-000000000000 > event 301 __ [ 1| Start] WaitHandleWait/Start
125980 | 00000000-0000-0000-0000-000000000000 > event 302 __ [ 2|  Stop] WaitHandleWait/Stop

There is no associated activity ID so you rely on the fact that the same waiter thread (125980 in the previous example) is emitting for both events.

Listening to the new Wait events

As usual, you should rely on the TraceEvent nuget to start an EventPipe session with an already running .NET application. The last version already contains the definition of the keyword:

1

keywords |= ClrTraceEventParser.Keywords.WaitHandle; // .NET 9 WaitHandle kind of contention

and the C# events for Start and Stop:

1
2

source.Clr.WaitHandleWaitStart += OnWaitHandleWaitStart;
source.Clr.WaitHandleWaitStop += OnWaitHandleWaitStop;

The handler’s implementation is straightforward. The start of the wait is recorded for the current thread:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

private void OnWaitHandleWaitStart(WaitHandleWaitStartTraceData data)
{
    // get the contention info for the current thread
    ContentionInfo info = _contentionStore.GetContentionInfo(data.ProcessID, data.ThreadID);
    if (info == null)
        return;

    // keep track of the wait start
    info.ContentionStartRelativeMSec = data.TimeStampRelativeMSec;
}

When the wait ends, the duration is computed based on the recorded wait start because it is not provided in the payload like for ContentionStop:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

private void OnWaitHandleWaitStop(WaitHandleWaitStopTraceData data)
{
    ContentionInfo info = _contentionStore.GetContentionInfo(data.ProcessID, data.ThreadID);
    if (info == null)
        return;
    // unlucky case when we start to listen just after the WaitHandleStart event
    if (info.ContentionStartRelativeMSec == 0)
    {
        return;
    }

    // Too bad the duration is not provided in the payload like in ContentionStop...
    var contentionDurationMSec = data.TimeStampRelativeMSec - info.ContentionStartRelativeMSec;
    info.ContentionStartRelativeMSec = 0;
    var duration = TimeSpan.FromMilliseconds(contentionDurationMSec);
    Console.WriteLine($"{e.ThreadId,7} | {e.Duration.TotalMilliseconds} ms");
}

This is nice but it would be more useful if we could get the call stack of long waits.

Call stacks with EventPipe

In a previous post, I explained that it is possible to get the call stack when an event is emitted thanks to the ClrStackWalk event that follows the event you are interested in. Unfortunately, this is not more the case for .NET 5+ that is using EventPipe instead of ETW.

As Olivier Coanet presents in his post, you can get the call stack as an array of addresses from the hidden event record that is mapped by the TraceEvent parameter passed to each event handlers. This EVENT_RECORD structure contains a ExtendedData field that is an array of EVENT_HEADER_EXTENDED_DATA_ITEM:

1
2
3
4
5
6
7
8

public struct EVENT_HEADER_EXTENDED_DATA_ITEM
{
    public ushort Reserved1;
    public ushort ExtType;
    public ushort Reserved2;
    public ushort DataSize;
    public ulong DataPtr;
}

If the ExtType value is EVENT_HEADER_EXT_TYPE_STACK_TRACE64 (=6) then DataPtr points to a EVENT_EXTENDED_ITEM_STACK_TRACE64 structure:

1
2
3
4
5

public struct EVENT_EXTENDED_ITEM_STACK_TRACE64
{
    public ulong MatchId;
    public unsafe fixed ulong Address[1];
}

that contains an array of 64-bit addresses. The size of this array is given by DataSize — sizeof(ulong).

For 32-bit applications, you will get EVENT_HEADER_EXT_TYPE_STACK_TRACE32 (=5) as ExtType value and DataPtr will point to EVENT_EXTENDED_ITEM_STACK_TRACE32:

1
2
3
4
5

public struct EVENT_EXTENDED_ITEM_STACK_TRACE32
{
    public ulong MatchId;
    public unsafe fixed uint Address[1];
}

that stores an array of 32-bit addresses.

Knowing that makes writing the code to get the call stacks as an array of 64-bit addresses (same with 32-bit applications for simplicity sake) pretty straightforward:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

public static EventPipeUnresolvedStack ReadStackUsingUnsafeAccessor(TraceEvent traceEvent)
{
    return GetFromEventRecord(traceEvent.eventRecord);
}

private static EventPipeUnresolvedStack GetFromEventRecord(TraceEventNativeMethods.EVENT_RECORD* eventRecord)
{
    if (eventRecord == null)
        return null;

    var extendedDataCount = eventRecord->ExtendedDataCount;

    for (var dataIndex = 0; dataIndex < extendedDataCount; dataIndex++)
    {
        var extendedData = eventRecord->ExtendedData[dataIndex];
        if (extendedData.ExtType == TraceEventNativeMethods.EVENT_HEADER_EXT_TYPE_STACK_TRACE64)
        {
            var stackRecord = (TraceEventNativeMethods.EVENT_EXTENDED_ITEM_STACK_TRACE64*)extendedData.DataPtr;
            var addresses = &stackRecord->Address[0];
            var addressCount = (extendedData.DataSize - sizeof(UInt64)) / sizeof(UInt64);
            if (addressCount == 0)
                return null;

            var callStackAddresses = new ulong[addressCount];
            for (var index = 0; index < addressCount; index++)
            {
                callStackAddresses[index] = addresses[index];
            }
            return new EventPipeUnresolvedStack(callStackAddresses);
        }
        else if (extendedData.ExtType == TraceEventNativeMethods.EVENT_HEADER_EXT_TYPE_STACK_TRACE32)
        {
            var stackRecord = (TraceEventNativeMethods.EVENT_EXTENDED_ITEM_STACK_TRACE32*)extendedData.DataPtr;
            var addresses = &stackRecord->Address[0];
            var addressCount = (extendedData.DataSize - sizeof(UInt32)) / sizeof(UInt32);
            if (addressCount == 0)
                return null;

            var callStackAddresses = new ulong[addressCount];  // store the 32 addresses as 64 bit addresses
            for (var index = 0; index < addressCount; index++)
            {
                callStackAddresses[index] = addresses[index];
            }

            return new EventPipeUnresolvedStack(callStackAddresses);
        }
    }

    return null;
}

Note that the last version of TraceEvent nuget provides a public access to the eventRecord field so it is no more needed to use the UnsafeAccessor attribute used by Olivier.

Symbolize the call stack addresses

Address is good but the corresponding method name is better. I won’t repeat what I’ve already detailed in an older post that shows how to get the name of a native and managed name from an instruction pointer address. Instead, I want to pinpoint a big limitation of this solution to listen to CLR provider MethodLoadVerbose/MethodDCStartVerboseV2 events. If the methods you are interested in are jitted BEFORE your tool attaches to the application, you will never get these events.

You could get the same mapping address span/method name via the other “Microsoft-Windows-DotNETRuntimeRundown” provider and its MethodDCEndVerbose event that contains the expected MethodStartAddress, MethodSize and MethodName in its payload. But I need this information before the end of the application…

Looking at the documentation, it seems that the rundown provider accepts the StartRundownKeyword value to emit the DCStart events when the provider is enabled! Since .NET 9, it is possible to pass the keywords you want (before, the default value did not contain StartRundownKeyword) when creating the EventPipe session

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

//                V-- this is the default rundown keyword
rundownKeywords = 0x80020139 | (long)ClrTraceEventParser.Keywords.StartEnumeration;
var config = new EventPipeSessionConfiguration(GetProviders(), 256, rundownKeywords, true);
using (var session = client.StartEventPipeSession(config))
{
    var source = new EventPipeEventSource(session.EventStream);
    RegisterListeners(source);

    // this is a blocking call
    source.Process();
}

Note that you should not add the rundown provider to the list passed as parameter.

Unfortunately, there is currently an issue in the runtime since September 2020 that pinpoints this exact problem. I even tried to create and close a session to get the DCStop events before recreating a new one, but I failed.

The next episode will talk about how it is possible to start a .NET application and get the events since its startup… with the problems that are happening.

In the previous post, I detailed how I used the undocumented events from the BCL to create the dotnet-http CLI tool to monitor your outgoing HTTP requests. After testing with older versions of .NET, I realized that the code needed to be updated and I’m sharing my findings in this post.

The main point is that url redirections could have a major impact on requests latency:

Always test supported versions…

When I wrote the initial version of dotnet-http, I only tested it with .NET 8 and .NET 9 with limited formats of urls. Unfortunately, things went bad when I tried to monitor applications running on .NET 5 and .NET 6: no events are emitted by these versions of the BCL.

So, the next step was to test .NET 7 and the result was simple: crash! After investigating, I realized that some events I looked at in .NET 8 source code did not have the same payload in .NET 7; even no payload at all:

Even though there is no version field in the events payload, it is easy to check its size such as the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

private void OnConnectionEstablished(
     DateTime timestamp,
     int threadId,
     Guid activityId,
     Guid relatedActivityId,
     byte[] eventData
     )
 {
     ...
     EventSourcePayload payload = new EventSourcePayload(eventData);
     var versionMajor = payload.GetByte();
     var versionMinor = payload.GetByte();

     // in .NET 7, nothing else is available
     Int64 connectionId = 0;
     var scheme = "";
     var host = "";
     UInt32 port = 0;
     var path = "";

     if (eventData.Length > 2)
     {
         connectionId = payload.GetInt64();
         scheme = payload.GetString();
         host = payload.GetString();
         port = payload.GetUInt32();
         path = payload.GetString();
     }
     ...

Another difference is that one even is not even emitted in .NET 7:

Explain what a redirection is please!

Because I based the code on this Redirect event, I needed to find another way to support .NET 7 even though I would not have a redirected url to display. But first, let’s see what I’m talking about in terms of HTTP communication.

When you try to get the content / status code behind a url, the code is following different phases with the related events:

• **Start **RequestStart
• **DNS resolution **ResolutionStart ResolutionStop/Fail
• **Socket connection **ConnectStart ConnectStop
• **Security hand check (HTTPS only) **HandshakeStart HandshakeStop/Failed
• **Request/response **RequestHeadersStart RequestHeadersStop ResponseHeadersStart ResponseHeadersStop Redirect (.NET 8+) ResponseContentStart ResponseContentStop
• **Request stop **RequestStop/Failed

Based on the received url, a server can decide to answer that another url should be used instead. For example, if you call github with http:// instead of https://, such a redirection will happen. Without invasive tools such as Wireshark, these redirections are impossible to detect and could cause unnecessary delay.

From the client perspective, this can be detected in the ResponseHeadersStop event payload that provides a status code. If its value is 301, then it is a redirection. The other effect of a redirection is that the BCL code will change the flow of events because it needs to start over with the new url from step 2. to step 6. As you can see, instead of paying the cost of just one request, two are actually emitted and processed.

Impact on the implementation

In addition to the payload size checks addition, my initial implementation was not properly handling the redirection because the values (timestamps and durations) where overridden by the events related to the second redirected url.

The new implementation is splitting the request details into two classes. A base class that contains common fields to both parts of the request in case of redirection:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

private class HttpRequestInfoBase
   {
       public HttpRequestInfoBase(DateTime timestamp, string scheme, string host, uint port, string path)
       {
           StartTime = timestamp;
           if (scheme == string.Empty)
           {
               Url = string.Empty;
           }
           else
           {
               if (port != 0)
               {
                   Url = $"{scheme}://{host}:{port}{path}";
               }
               else
               {
                   Url = $"{scheme}://{host}:{path}";
               }
           }
       }

       public string Url { get; set; }
       public DateTime StartTime { get; set; }
       public DateTime ReqRespStartTime { get; set; }
       public double ReqRespDuration { get; set; }

       // DNS
       public double DnsWait { get; set; }
       public DateTime DnsStartTime { get; set; }
       public double DnsDuration { get; set; }

       // HTTPS
       public double HandshakeWait { get; set; }
       public DateTime HandshakeStartTime { get; set; }
       public double HandshakeDuration { get; set; }

       // socket connection
       public DateTime SocketConnectionStartTime { get; set; }
       public double SocketWait { get; set; }
       public double SocketDuration { get; set; }

       public DateTime QueueuingEndTime { get; set; }
       public double QueueingDuration { get; set; }
   }

The second one inherits from the base class and contains addition details; including the details of the redirected url if any:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

private class HttpRequestInfo : HttpRequestInfoBase
   {
       public HttpRequestInfo(DateTime timestamp, string scheme, string host, uint port, string path)
           :
           base(timestamp, scheme, host, port, path)
       {
       }

       public HttpRequestInfoBase Redirect { get; set; }

       public UInt32 StatusCode { get; set; }

       // HTTPS
       public string HandshakeErrorMessage { get; set; }

       public string Error { get; set; }
   }

A new instance of HttpRequestInfoBase is created when a 301 status code is received in HttpResponseHeaderStop handler:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

private void OnHttpResponseHeaderStop(object sender, HttpRequestStatusEventArgs e)
    {
        // used to detect redirection in .NET 8+
        if (e.StatusCode != 301)
        {
            return;
        }

        // create a new request info for the redirected request
        // because .NET 7 does not emit a Redirect event, we need to create a new request info here
        // --> it means that the redirect url will be empty in .NET 7
        var root = GetRoot(e.ActivityId);
        if (_requests.TryGetValue(root, out HttpRequestInfo info))
        {
            info.Redirect = new HttpRequestInfoBase(e.Timestamp, "", "", 0, "");

            // if you really want to have the duration of both original request + redirected request,
            // then do the following:
            //    info.ReqRespDuration = (e.Timestamp - info.ReqRespStartTime).TotalMilliseconds;
            // However, I prefer to show the duration of the redirected request only to more easily
            // compute the cost of the initial redirected request = total duration - other durations
        }
    }

For .NET 8+, the redirected url is provided to the Redirect handler and stored in the Url field of the instance created in the previous handler:

1
2
3
4
5
6
7
8
9

private void OnHttpRedirect(object sender, HttpRedirectEventArgs e)
    {
        // since this is an Info event, the activityID is the root
        var root = ActivityHelpers.ActivityPathString(e.ActivityId);
        if (_requests.TryGetValue(root, out HttpRequestInfo info))
        {
            info.Redirect.Url = e.RedirectUrl;
        }
    }

In each handler of events that could be received for both initial and redirected requests,

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

private void OnHttpResponseContentStop(object sender, EventPipeBaseArgs e)
   {
       var root = GetRoot(e.ActivityId);
       if (!_requests.TryGetValue(root, out HttpRequestInfo info))
       {
           return;
       }

       if (info.Redirect == null)
       {
           info.ReqRespDuration = (e.Timestamp - info.ReqRespStartTime).TotalMilliseconds;
       }
       else
       {
           info.Redirect.ReqRespDuration = (e.Timestamp - info.Redirect.ReqRespStartTime).TotalMilliseconds;
       }
   }

The wait time and durations are now computed as the events are received and aggregated at the end of the request by adding the value of both parts (initial and redirected if any:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

double dnsDuration = info.DnsDuration + ((info.Redirect != null) ? info.Redirect.DnsDuration : 0);
    if (dnsDuration > 0)
    {
        double dnsWait = info.DnsWait + ((info.Redirect != null) ? info.Redirect.DnsWait : 0);
        Console.Write($"{dnsWait,9:F3} | {dnsDuration,9:F3} | ");
    }
    else
    {
        Console.Write($"          |           | ");
    }

As a conclusion, you should try to monitor these redirections by using my dotnet-http CLI tool. Feel free to download it or install it with the following command line: dotnet tool install -g dotnet-http or update to the latest version with: dotnet tool update -g dotnet-http

You could also integrate some event listening code into your framework that simply handles the ResponseHeadersStop/Redirect events.

The previous episode detailed how to find the events dealing with network requests that are emitted by the BCL classes with their undocumented payload. It is now time to see how to listen to them and extract valuable insights such as what is happening when an HTTP request is sent to a server as an example. This is how my new dotnet-http CLI tool is implemented.

You are now able to see the cost of DNS, socket connection, security and redirection as shown in the following screenshot.

As you can see, the cost of an unexpected redirection could be high and hides security handshakes. In this example, using https://github.com/Maoni0 instead of http://github.com/Maoni0 divides by 2 the request duration!

Once the DNS checks are done, the socket connections are established, and the security handshakes are validated, the corresponding phases are no more needed for the next requests.

Listen to custom EventSource events

As explained in previous posts, it is easy to listen to events emitted by .NET application thanks to the Microsoft TraceEvent nuget. You also need to use EventPipeClient from the Microsoft.Diagnostics.NETCore.Client nuget to connect to the running application. In my case, I’m relying on an older version that supports both ETW and EventPipe:

1
2
3
4
5
6
7

var configuration = new SessionConfiguration(
    circularBufferSizeMB: 2000,
    format: EventPipeSerializationFormat.NetTrace,
    providers: GetProviders()
);
var binaryReader = EventPipeClient.CollectTracing(_processId, configuration, out var sessionId);
EventPipeEventSource source = new EventPipeEventSource(binaryReader);

The configuration contains a list of providers that are emitting the events you are interested in with the right keyword and verbosity. Here is what is needed for the HTTP requests monitoring :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

private static IReadOnlyCollection<Provider> GetProviders()
{
    var providers = new Provider[]
    {
        new Provider(
            name: "System.Net.Http",
            keywords: (ulong)(1),
            eventLevel: EventLevel.Verbose),

        new Provider(
            name: "System.Net.Sockets",
            keywords: (ulong)(0xFFFFFFFF),
            eventLevel: EventLevel.Verbose),

        new Provider(
            name: "System.Net.NameResolution",
            keywords: (ulong)(0xFFFFFFFF),
            eventLevel: EventLevel.Verbose),

        new Provider(
            name: "System.Net.Security",
            keywords: (ulong)(0xFFFFFFFF),
            eventLevel: EventLevel.Verbose),

        new Provider(
            name: "System.Threading.Tasks.TplEventSource",
            keywords: (ulong)(0x80),
            eventLevel: EventLevel.Verbose),

    };

    return providers;
}

You might be surprised by the presence of the TplEventSource provider but it is required to get a correct ActivityID. This major subject is detailed later.

There are so many events to listen to that it is easier to listen to the source AllEvents C# event:

1

source.AllEvents += OnEvents;

This global handler passes the events to ParseEvent that forwards the important data to handlers for each provider:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

private static Guid NetSecurityEventSourceProviderGuid = Guid.Parse("7beee6b1-e3fa-5ddb-34be-1404ad0e2520");
private static Guid DnsEventSourceProviderGuid = Guid.Parse("4b326142-bfb5-5ed3-8585-7714181d14b0");
private static Guid SocketEventSourceProviderGuid = Guid.Parse("d5b2e7d4-b6ec-50ae-7cde-af89427ad21f");
private static Guid HttpEventSourceProviderGuid = Guid.Parse("d30b5633-7ef1-5485-b4e0-94979b102068");

private void ParseEvent(
    DateTime timestamp,
    int threadId,
    Guid activityId,
    Guid relatedActivityId,
    Guid providerGuid,
    string taskName,
    Int64 keywords,
    UInt16 id,
    byte[] eventData
    )
{
    if (providerGuid == NetSecurityEventSourceProviderGuid)
    {
        HandleNetSecurityEvent(timestamp, threadId, activityId, relatedActivityId, id, taskName, eventData);
    }
    else
    if (providerGuid == DnsEventSourceProviderGuid)
    {
        HandleDnsEvent(timestamp, threadId, activityId, relatedActivityId, id, taskName, eventData);
    }
    else
    if (providerGuid == SocketEventSourceProviderGuid)
    {
        HandleSocketEvent(timestamp, threadId, activityId, relatedActivityId, id, taskName, eventData);
    }
    else
    if (providerGuid == HttpEventSourceProviderGuid)
    {
        HandleHttpEvent(timestamp, threadId, activityId, relatedActivityId, id, taskName, eventData);
    }
    else
    {
        WriteLogLine();
    }
}

The next step is to handle each event based on its id:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

private void HandleNetSecurityEvent(
    DateTime timestamp,
    int threadId,
    Guid activityId,
    Guid relatedActivityId,
    ushort id,
    string taskName,
    byte[] eventData
    )
{
    switch (id)
    {
        case 1: // HandshakeStart
            OnHandshakeStart(timestamp, threadId, activityId, relatedActivityId, eventData);
            break;
        case 2: // HandshakeStop
            OnHandshakeStop(timestamp, threadId, activityId, relatedActivityId, eventData);
            break;
        case 3: // HandshakeFailed
            OnHandshakeFailed(timestamp, threadId, activityId, relatedActivityId, eventData);
            break;
        default:
            WriteLogLine();
            break;
    }
}

Extract information from an event payload

The payload of each event has been detailed in the previous post. For example, I need to extract the following fields from the RequestStart event payload:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

private void OnRequestStart(
    DateTime timestamp,
    int threadId,
    Guid activityId,
    Guid relatedActivityId,
    byte[] eventData
    )
{
    ...
    // string scheme
    // string host
    // int port
    // string path
    // byte versionMajor
    // byte versionMinor
    // enum HttpVersionPolicy

I’ve implemented the EventSourcePayload class that provides strongly typed helpers to get the different fields one after the other:

1
2
3
4
5
6
7
8
9

public class EventSourcePayload
{
    private byte[] _payload;
    private int _pos = 0;

    public EventSourcePayload(byte[] payload)
    {
        _payload = payload;
    }

It accepts the payload as the array of bytes received by each handler.

A string information is serialized as a list of UTF-16 Unicode characters; each one stored in 2 bytes:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

public string GetString()
    {
        StringBuilder builder = new StringBuilder();
        while (_pos < _payload.Length)
        {
            var characters = UnicodeEncoding.Unicode.GetString(_payload, _pos, 2);
            _pos += 2;

            if (characters == "\0")
            {
                break;
            }
            builder.Append(characters);
        }

        return builder.ToString();
    }

The current position in the array is incremented character by character up to the final ‘\0’.

The other helpers implementation is straightforward thanks to the BitConverter class:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

public byte GetByte()
    {
        return _payload[_pos++];
    }

    public UInt16 GetUnit16()
    {
        UInt16 value = BitConverter.ToUInt16(_payload, _pos);
        _pos += sizeof(UInt16);
        return value;
    }

    public UInt32 GetUInt32()
    {
        UInt32 value = BitConverter.ToUInt32(_payload, _pos);
        _pos += sizeof(UInt32);
        return value;
    }

    public UInt64 GetUInt64()
    {
        UInt64 value = BitConverter.ToUInt64(_payload, _pos);
        _pos += sizeof(UInt64);
        return value;
    }

    public Int64 GetInt64()
    {
        Int64 value = BitConverter.ToInt64(_payload, _pos);
        _pos += sizeof(UInt64);
        return value;
    }

    public double GetDouble()
    {
        double value = BitConverter.ToDouble(_payload, _pos);
        _pos += sizeof(double);
        return value;
    }
}

Again, the position in the array is incremented to reflect the size of the field being read.

If you look at the rest of the OnRequestStart handler, you will see how each field is extracted:

1
2
3
4
5
6
7

EventSourcePayload payload = new EventSourcePayload(eventData);
    var scheme = payload.GetString();
    var host = payload.GetString();
    var port = payload.GetUInt32();
    var path = payload.GetString();
    var versionMajor = payload.GetByte();
    var versionMinor = payload.GetByte();

My activity or not my activity: that is the question

In the previous episode, I forgot (on purpose) to mention that the EventSource is keeping track of “activities” when emitting events:

1
2
3
4

protected unsafe void WriteEventCore(int eventId, int eventDataCount, EventData* data)
{
    WriteEventWithRelatedActivityIdCore(eventId, null, eventDataCount, data);
}

The WriteEventWithRelatedActivityIdCore method looks at the event metadata and if its opcode is Start then a new activity is created; if it is Stop then the current activity ends:

1
2
3
4
5
6
7
8

if (opcode == EventOpcode.Start)
{
    m_activityTracker.OnStart(m_name, metadata.Name, metadata.Descriptor.Task, ref activityId, ref relActivityId, metadata.ActivityOptions);
}
else if (opcode == EventOpcode.Stop)
{
    m_activityTracker.OnStop(m_name, metadata.Name, metadata.Descriptor.Task, ref activityId);
}

These OnStart and OnStop methods are doing nothing if the TplEventSource is not enabled with Keywords.TasksFlowActivityIds (= 0x80) set. This explains the code in GetProviders listed earlier where this non-HTTP provider is enabled.

When a request is created, the current global count managed by the ActivityTracker is incremented and it becomes the id of the current activity. Note that there is a “root” identifier before any activity gets created corresponding to the current AppDomain ID; starting at 1. If you think of an HTTP request, after a RequestStart event, each phase starts a new activity with, for example, ResolutionStart or ConnectStart. Informative events are emitted with the current request activity such as Redirect or ConnectionEstablished.

Here is a simplified view of the events emitted for an HTTP request (without DNS nor security events):

Thread +-- Path >------- ID ---- Opcode -- Event ----------------------------------
 78568 |    1/1 >  event  1 __ [ 1| Start] RequestStart
       |
 32388 |  1/1/1 >  event  1 __ [ 1| Start] ResolutionStart
       |
 32388 |  1/1/1 >  event  2 __ [ 2|  Stop] ResolutionStop
       |
 32388 |  1/1/2 >  event  1 __ [ 1| Start] ConnectStart
       |
 32388 |  1/1/2 >  event  2 __ [ 2|  Stop] ConnectStop
       |
 32388 |    1/1 >  event  4 __ [ 0|  Info] ConnectionEstablished
       |
 53324 |    1/1 >  event  6 __ [ 0|  Info] RequestLeftQueue
       |
 53324 |  1/1/3 >  event  7 __ [ 1| Start] RequestHeadersStart
       |
 53324 |  1/1/3 >  event  8 __ [ 2|  Stop] RequestHeadersStop
       |
 68024 |  1/1/4 >  event 11 __ [ 1| Start] ResponseHeadersStart
       |
 68024 |  1/1/4 >  event 12 __ [ 2|  Stop] ResponseHeadersStop
       |
 68024 |  1/1/5 >  event 13 __ [ 1| Start] ResponseContentStart
       |
 68024 |  1/1/5 >  event 14 __ [ 2|  Stop] ResponseContentStop
       |
 68024 |    1/1 >  event  2 __ [ 2|  Stop] RequestStop
  200 <|

As you can see, different threads are emitting events associated to the same request. At the ActivityTracker level, an ActivityInfo is stored in an async local so that each thread has its own storage that will be propagated by the methods of the Task Parallel Library (a.k.a. TPL) from task to task. This is why the very asynchronous code of the HTTP client implementation can go back to the current activity from different threads.

The activities are encoded into the 16 bytes of a GUID. In fact, only the first 12 bytes are used, and the final 4 bytes contain a checksum that includes the current process ID. Since the activity identifiers in the path do, most of the time, have a small value, the encoding is using 4 bits, also known as a nibble, to encode each of them. There was a bug in ActivityTracker.AddIdToGuid that will be fixed in .NET 10. Unfortunately, the decoding code in Perfview needs to take it into account so the last activity is not lost.

Computing the different durations

With this infrastructure in place, it is now possible to extract the “root” activity path corresponding to an HTTP request during each phase and update the corresponding state that is used to output the details when a request ends:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

private class HttpRequestInfo
{
    public HttpRequestInfo(DateTime timestamp, Guid activityId, string scheme, string host, uint port, string path)
    {
        Root = ActivityHelpers.ActivityPathString(activityId);
        if (port != 0)
        {
            Url = $"{scheme}://{host}:{port}{path}";
        }
        else
        {
            Url = $"{scheme}://{host}:{path}";
        }
        StartTime = timestamp;
    }

    public string Root { get; set; }
    public string Url { get; set; }
    public string RedirectUrl { get; set; }
    public DateTime StartTime { get; set; }
    public DateTime ReqRespStartTime { get; set; }
    public double ReqRespDuration { get; set; }

    public UInt32 StatusCode { get; set; }

    // DNS
    public DateTime DnsStartTime { get; set; }
    public double DnsDuration { get; set; }

    // HTTPS
    public DateTime HandshakeStartTime { get; set; }
    public double HandshakeDuration { get; set; }
    public string HandshakeErrorMessage { get; set; }

    // socket connection
    public DateTime SocketConnectionStartTime { get; set; }
    public double SocketDuration { get; set; }
}

As you have seen earlier, the different phases of a request may be processed by different threads. It means that an available thread needs to be found in order to execute them. If the thread pool is busy, you can expect some wait time. This is shown in the different wait sections between the phases. They are easily computed using the timestamp of each event. Having long wait durations might be reduced by increasing the number of threads in the thread pool.

Feel free to use the new dotnet-http CLI tool available from nuget.org or via dotnet tool install -g dotnet-http.

The corresponding sources are available from my github repository in case you would like to integrate this kind of analysis directly in your code or your monitoring pipeline.

Happy monitoring!

I’ve presented in depth the events emitted by the CLR in many posts to get insightful details about how the .NET runtime is working (lock contention, GC, allocations, …). Some .NET features are not implemented at the runtime level but at the Base Class Library (a.k.a. BCL) level. For example, if you are using HttpClient, you might want to measure how long it takes to get the response to your HTTP requests.

In this new series, I will describe how the BCL is using EventSource to emit the events, how you can listen to them with TraceEvent; focusing on HTTP requests.

Where do these events come from?

When you look for these BCL events in the documentation, you end up to this well-known event provides in .NET page and in the Framework libraries section. It lists the providers with their name and the emitted events with their keyword and verbosity. However, there is no detail about their payload! For example, for the “System.Net.Http” provider, you know that the RequestStart informal event is emitted each time “an HTTP request has started”. But you don’t know the corresponding url… Let me tell you how to get these tiny details.

Within the BCL, the classes responsible for emitting events derive from EventSource with the “Telemetry” suffix as a naming convention. They are decorated with an EventSource attribute to define their name (used as provider name by a listener) and their Guid that will be provided with each event. For example, here is the declaration of the class responsible for events related to sending HTTP requests:

1
2
3

[EventSource(Name = "System.Net.Http")]
internal sealed partial class HttpTelemetry : EventSource
{

If a Guid is not provided, one is automatically computed (see the corresponding code in Perfview).

Next, public helper methods decorated with a NonEvent attribute are provided to be used in the BCL code when it is needed to emit events. These methods are calling private methods decorated with an Event attribute to define their unique ID and their verbosity level:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

[NonEvent]
public void RequestStart(HttpRequestMessage request)
{
    ...
    RequestStart(
        request.RequestUri.Scheme,
        request.RequestUri.IdnHost,
        request.RequestUri.Port,
        request.RequestUri.PathAndQuery,
        (byte)request.Version.Major,
        (byte)request.Version.Minor,
        request.VersionPolicy);
} 

[Event(1, Level = EventLevel.Informational)]
private void RequestStart(string scheme, string host, int port, string pathAndQuery, byte versionMajor, byte versionMinor, HttpVersionPolicy versionPolicy)
{
    ...
    WriteEvent(eventId: 1, scheme, host, port, pathAndQuery, versionMajor, versionMinor, versionPolicy);
}

The different WriteEvent overloads are responsible for filling an array of EventData elements (each one contains a pointer to the data and its size) that is passed to EventSource.WriteEventCore. This helper methods dispatches the event to EventPipe/ETW pipelines.

Document the undocumented

Unlike CLR events, their payload is not explicitly described in a text file but, instead, you have to look at the implementation of the different WriteEvent overloads. The rest of this section provides the details of the sources I’ve looked at and the corresponding events payload.

HTTP

Sources: https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.Http/src/System/Net/Http/HttpTelemetry.cs Name: System.Net.Http Guid: d30b5633–7ef1–5485-b4e0–94979b102068

Sockets

Sources: https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketsTelemetry.cs

Name: System.Net.Sockets Guid: d5b2e7d4-b6ec-50ae-7cde-af89427ad21f

DNS

Sources: https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.NameResolution/src/System/Net/NameResolutionTelemetry.cs

Name: System.Net.NameResolution Guid: 4b326142-bfb5–5ed3–8585–7714181d14b0

Network Security

Sources: https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.Security/src/System/Net/Security/NetSecurityTelemetry.cs

Name: System.Net.Security  Guid: 7beee6b1-e3fa-5ddb-34be-1404ad0e2520

The next episode will describe how to listen to these events and extract their payload in C#.

Introduction

If you have read Microsoft documentation, you probably know that it is not recommended to trigger a garbage collection in your application code. However, in some troubleshooting cases, you might want to trigger a GC. For example, you don’t want to wait for a full gen2 compacting GC to figure out if your application is really leaking memory. For web applications, you can imagine having a hidden HTTP end point that simply call GC.Collect. What if you could simply call a command line tool to trigger a GC in any .NET application? This is exactly what my new dotnet-fullgc CLI tool is doing!

This Is The Way

If you have read a few of my past posts about the .NET diagnostics mechanisms, you know that you can send commands to another .NET process via EventPipes. Well… there is no explicit command to trigger a GC.

I have also explained how you could listen to events emitted by the CLR by enabling a provider with a set of keywords with a verbosity corresponding to the events you are interested in. This is how dotnet-trace and Perfview are collecting these events. If you want to trigger a GC, you simply need to enable the Microsoft-Windows-DotNETRuntime provider with the GCHeapCollect (= 0x800000L) keyword and an informal verbosity. Yes: it is as simple as that, and it is also working for .NET Framework!

So, you could trigger a GC with dotnet-trace via the following command line:

1

dotnet trace collect -p <process id> - providers Microsoft-Windows-DotNETRuntime:0x800000:4 - duration 00:00:01

However, a .nettrace file would be generated and it is not possible to pass parameters (more on this later).

The rest of this post shows the C# code to obtain the same result. With Microsoft.Diagnostics.NETCore.Client and TraceEvent, you create a DiagnosticsClient with the ID of the process you are interested in:

1

var client = new DiagnosticsClient(processId);

The next step is to start an EventPipe session with the right provider, keyword and verbosity:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

var providers = new List<EventPipeProvider>()
{
    new EventPipeProvider(
        "Microsoft-Windows-DotNETRuntime",
        EventLevel.Informational,
        (long)ClrTraceEventParser.Keywords.GCHeapCollect,
        Arguments  // more on this later
        ),
};
using (var session = client.StartEventPipeSession(providers, false))
{

The source must be processed in another thread to avoid blocking the main thread:

1
2
3
4
5
6
7

Task streamTask = Task.Run(() =>
    {
        // without source to process, session.Stop() will not return
        var source = new EventPipeEventSource(session.EventStream);

        source.Process();
    });

The question to answer is how to stop the session: just create another task that waits for a second before stopping the session to exit from the Process call:

1
2
3
4
5
6
7
8

Task inputTask = Task.Run(() =>
    {
        Thread.Sleep(1000);
        session.Stop();
    });

    Task.WaitAny(streamTask, inputTask);
}

That’s all!

Pitfalls

Unfortunately, I faced a couple of issue during the implementation of the tool.

What is your number?

If you look at the TraceEvent implementation, you could imagine that is it possible to pass a GC ID as a parameter to the .NET provider:

1
2
3
4
5

//
// Summary:
//     Triggers a GC. Can pass a 64 bit value that will be logged with the GC Start
//     event so you know which GC you actually triggered.
GCHeapCollect = 0x800000L,

Unfortunately, this is not supported and the reasons are explained below.

If you check the .NET source code and look at EtwCallbackCommon(), you can indeed see that a numeric ID can be passed to ETW::GCLog::ForceGC(l64ClientSequenceNumber) and passed as ID in GCStart event instead of the one incrementally increased collection after collection.

At the diagnostics client level, you have the opportunity to pass a dictionary of key/value string pairs. This dictionary is used when defining the EventPipeProvider to be enabled:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

Dictionary<string, string> arguments = new Dictionary<string, string>();
arguments.Add("Id", "42");
var providers = new List<EventPipeProvider>()
{
    new EventPipeProvider(
        "Microsoft-Windows-DotNETRuntime",
        EventLevel.Informational,
        (long)ClrTraceEventParser.Keywords.GCHeapCollect,
        arguments
        ),
};

The diagnostics client transforms the dictionary into a string with key=value pairs separated by ‘;’ such as “Id=42;AnotherId=AnotherValue;…” and serializes it as is in the payload when sending a CollectTraces command.

The question is what identifier is expected by the CLR? To answer this question, you need to look at the code of provider_invoke_callback. The “key=value;…” string is stored into a buffer and each = and ; characters are transformed into \0. So, “Id=42” is transformed into Id\042\0.

The next step is done by ep_event_filter_desc_init() that uses that buffer to fill up the 3 fields of an EventFilterDescriptor:

1
2
3

uint64_t ptr = address of the buffer
uint32_t size = size of the buffer (=6 for id\042\0)
uint32_t type = 0

And finally, EtwCallbackCommon receives the filterdata and tries the following to get the collection id:

1
2
3
4
5
6
7

PEVENT_FILTER_DESCRIPTOR FilterData = (PEVENT_FILTER_DESCRIPTOR)pFilterData;
if ((FilterData != NULL) &&
   (FilterData->Type == 1) &&
   (FilterData->Size == sizeof(l64ClientSequenceNumber)))
{
    l64ClientSequenceNumber = *(LONGLONG *) (FilterData->Ptr);
}

As you can see, it will fail because:

• the received type value is 0
• the received size is not the size of a 64 bit number
• the ptr field does not point to the value as 64 bit number (but as a string)

An issue has been filed with a possible fix.

Only one collection please!

When I test my dotnet-fullgc with dotnet-gcstats, I always see 2 collections!

We investigated with my colleague Kevin Gosse and he created an issue for that. The EtwCallback() function is called whenever a session is enabled or disabled. Unfortunately, the call to ForceGC is made in both cases: so, when the session is stopped, a second garbage collection is triggered.

Next steps

Feel free to install dotnet-fullgc on your machine with dotnet tool install -g dotnet-fullgc.

Next, use dotnet fullgc to trigger two gen2 full garbage collections in your running .NET processes.

As shown in the previous post, the processing of ProcessInfo diagnostic commands is easy because you send a request and read the different fields from the response. This is different if you want to receive events from the CLR via EventPipe. In C#, the TraceEvent nuget package wraps everything under a nice event handler based model as shown in many of my previous posts.

Behind the scene, a StartSession command is sent (more details about the parameters later) and the response contains the numeric ID of the session. Then, the events will be read from the IPC channel as a binary stream of data with the “nettrace“ file format. The collection ends when the StopTracing command is sent.

The source code is available from my github repository.

Hidding the transport layer: IIpcEndoint

Unlike the previous post, to send the command and read the response back from the CLR , I’m wrapping the transport layer with the IIpcEndpoint interface:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

class IIpcEndpoint
{
public:
    virtual bool Write(LPCVOID buffer, DWORD bufferSize, DWORD* writtenBytes) = 0;
    virtual bool Read(LPVOID buffer, DWORD bufferSize, DWORD* readBytes) = 0;
    virtual bool ReadByte(uint8_t& byte) = 0;
    virtual bool ReadWord(uint16_t& word) = 0;
    virtual bool ReadDWord(uint32_t& dword) = 0;
    virtual bool ReadLong(uint64_t& ulong) = 0;

    virtual bool Close() = 0;
    virtual ~IIpcEndpoint() = default;
};

It abstracts the write and read accesses to the underlying transport layer. In addition, the base class accepts a “recorder” that allows me to store what is received from the CLR into any kind of storage (today only a file-based recorder that helped a lot to reproduce specific situations without the need to have a running process to connect to):

The PidEndpoint class accepts the process id of the running .NET application to monitor its CLR events and an optional recorder implementing the IIpcRecorder interface. The Create static factory implementation creates the expected named pipe on Windows (or the domain socket on Linux) and stores the handle into its _handle field:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

PidEndpoint* PidEndpoint::CreateForWindows(int pid, IIpcRecorder* pRecorder)
{
    PidEndpoint* pEndpoint = new PidEndpoint(pRecorder);

    // build the pipe name as described in the protocol
    wchar_t pszPipeName[256];
    int nCharactersWritten = -1;
    nCharactersWritten = wsprintf(
        pszPipeName,
        L"\\\\.\\pipe\\dotnet-diagnostic-%d",
        pid
    );

    // check that CLR has created the diagnostics named pipe
    if (!::WaitNamedPipe(pszPipeName, 200))
    {
        auto error = ::GetLastError();
        std::cout << "Diagnostics named pipe is not available for process #" << pid << " (" << error << ")" << "\n";
        return nullptr;
    }

    // connect to the named pipe
    HANDLE hPipe;
    hPipe = ::CreateFile(
        pszPipeName,    // pipe name
        GENERIC_READ |  // read and write access
        GENERIC_WRITE,
        0,              // no sharing
        NULL,           // default security attributes
        OPEN_EXISTING,  // opens existing pipe
        0,              // default attributes
        NULL);          // no template file

    if (hPipe == INVALID_HANDLE_VALUE)
    {
        std::cout << "Impossible to connect to " << pszPipeName << "\n";
        return nullptr;
    }

    pEndpoint->_handle = hPipe;
    return pEndpoint;
}

The next step is to open a tracing session by sending the StartSession command.

The Trace diagnostic commands

Following the same object model provided by the Microsoft.Diagnostics.NETCore.Client nuget, my DiagnosticsClient class hides the transport layer. It also exposes high level functions such as OpenEventPipeSession to initiate a trace event session with the CLR:

1

EventPipeSession* OpenEventPipeSession(uint64_t keywords, EventVerbosityLevel verbosity);

If you remember from TraceEvent, you need a few parameters to create a session:

• size of circular buffers used by the CLR to cache events (same as Perfview, use 16 MB as default)
• netttrace format (i.e. value of 1)
• if rundown events are needed
• a list of providers (“Microsoft-Windows-DotNETRuntime” for the CLR in my case)
• keywords
• verbosity level
• possible arguments (none here)

Here is the corresponding C++ description of the command type:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

const uint8_t DotnetProviderMagicLength = 32;
struct MagicProvider
{
    wchar_t Magic[DotnetProviderMagicLength];
};

// 32 wchar_t (including \0)
const MagicProvider DotnetProviderMagic = { L"Microsoft-Windows-DotNETRuntime" };

const uint32_t CircularBufferMBSize = 16;
const uint32_t NetTraceFormat = 1;

#pragma pack(1)
struct StartSessionMessage : public IpcHeader
{
    uint32_t CircularBufferMB;  // 16 MB
    uint32_t Format;            // 1 for NetTrace format
    uint8_t RequestRundown;     // 0 because don't want rundown

    // array of provider configuration
    uint32_t ProviderCount;     // 1 only: Microsoft-Windows-DotNETRuntime
    uint64_t Keywords;          // from EventKeyword
    uint32_t Verbosity;         // from EventPipeEventLevel
    uint32_t ProviderStringLen; // number of UTF16 characters = 32 (including last \0)
    union                       // dotnet provider name
    {
        MagicProvider _magic;
        uint8_t Provider[2 * DotnetProviderMagicLength];
    };
    uint32_t Arguments;         // 0 for empty string (no argument)
};

The code to fill up the command is straightforward:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

StartSessionMessage* CreateStartSessionMessage(uint64_t keywords, EventVerbosityLevel verbosity)
{
    auto message = new StartSessionMessage();
    ::ZeroMemory(message, sizeof(message));
    memcpy(message->Magic, &DotnetIpcMagic_V1, sizeof(message->Magic));
    message->Size = sizeof(StartSessionMessage);  
    message->CommandSet = (uint8_t)DiagnosticServerCommandSet::EventPipe;
    message->CommandId = (uint8_t)EventPipeCommandId::CollectTracing2;
    message->Reserved = 0;
    message->CircularBufferMB = CircularBufferMBSize;
    message->Format = NetTraceFormat;
    message->RequestRundown = 0;
    message->ProviderCount = 1;
    message->Keywords = (uint64_t)keywords;
    message->Verbosity = (uint32_t)verbosity;
    message->ProviderStringLen = DotnetProviderMagicLength;
    memcpy(message->Provider, &DotnetProviderMagic, sizeof(message->Provider));
    message->Arguments = 0;

    return message;
}

The provider list is defined with the ProviderCount field and string containing the list (only one here) follows the Verbosity field. To start the session, it is needed to send the StartSession message and read the session id from the response:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

bool EventPipeStartRequest::Process(IIpcEndpoint* pEndpoint, uint64_t keywords, EventVerbosityLevel verbosity)
{
    // send an StartSessionMessage and parse the response
    StartSessionMessage* pMessage = CreateStartSessionMessage(keywords, verbosity);

    DWORD writtenBytes = 0;
    if (!pEndpoint->Write(pMessage, pMessage->Size, &writtenBytes))
    {
        return false;
    }

    // analyze the response
    IpcHeader response = {};
    DWORD bytesReadCount = 0;
    if (!pEndpoint->Read(&response, sizeof(response), &bytesReadCount))
    {
        return false;
    }

    if (response.CommandId != (uint8_t)DiagnosticServerResponseId::OK)
    {
        return false;
    }

    // get the session ID from the payload
    uint16_t payloadSize = response.Size - sizeof(response);
    if (payloadSize < sizeof(uint64_t))
    {
        return false;
    }

    if (!pEndpoint->ReadLong(SessionId))
    {
        return false;
    }

    return true;
}

Once the StartSession command has been sent, the events corresponding to the given provider/keywords/verbosity (here the CLR runtime/gc+exception+contention/verbose)

1
2
3
4
5
6

auto pSession = pClient->OpenEventPipeSession(
        EventKeyword::gc |
          EventKeyword::exception |
          EventKeyword::contention,
        EventVerbosityLevel::Verbose  // required for AllocationTick
        );

will be read from the event pipe. Since this action will be synchronous, it is recommended to dedicate a thread to read and process the events:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

    if (pSession != nullptr)
    {
        DWORD tid = 0;
        auto hThread = ::CreateThread(nullptr, 0, ListenToEvents, pSession, 0, &tid);

        std::cout << "Press ENTER to stop listening to events...\n\n";
        std::string line;
        std::getline(std::cin, line);

        std::cout << "Stopping session\n\n";
        pSession->Stop();
        std::cout << "Session stopped\n\n";

        // test if it works
        ::Sleep(1000);

        ::CloseHandle(hThread);
    }

The ListenToEvents callback executed by the new thread is “simply” listening to the event pipe of the session:

1
2
3
4
5
6
7
8

DWORD WINAPI ListenToEvents(void* pParam)
{
    EventPipeSession* pSession = static_cast<EventPipeSession*>(pParam);

    pSession->Listen();

    return 0;
}

Before describing how to read the events, it is important to understand how to stop the flow. First, inside the EventPipeSession, the internal loop that reads events needs to exit thanks to the _stopRequested boolean:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

bool EventPipeSession::Stop()
{
    _stopRequested = true;

    if (_pid == -1)
        return true;

    // it is neeeded to use a different ipc connection to stop the Session
    DiagnosticsClient* pStopClient = DiagnosticsClient::Create(_pid, nullptr);
    pStopClient->StopEventPipeSession(SessionId);
    delete pStopClient;

    return true; 
}

In addition, a message with StopTracing command id from the EventPipe command set needs to be sent to tell the CLR to stop sending the events. This message must be sent through a different IPC channel (hence the pStopClient variable used in the previous code. The StopEventPipeSession helper function uses the EventPipeStopRequest wrapper:

1
2
3
4
5

bool DiagnosticsClient::StopEventPipeSession(uint64_t sessionId)
{
    EventPipeStopRequest request;
    return request.Process(_pEndpoint, sessionId);
}

The StopSession command accepts the session ID as single parameter:

1
2
3
4
5

#pragma pack(1)
struct StopSessionMessage : public IpcHeader
{
    uint64_t SessionId;
};

The processing of the stop request is to create such a message and send it through the IPC channel:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

StopSessionMessage* CreateStopMessage(uint64_t sessionId)
{
    StopSessionMessage* message = new StopSessionMessage();
    ::ZeroMemory(message, sizeof(message));
    memcpy(message->Magic, &DotnetIpcMagic_V1, sizeof(message->Magic));
    message->Size = sizeof(StopSessionMessage);
    message->CommandSet = (uint8_t)DiagnosticServerCommandSet::EventPipe;
    message->CommandId = (uint8_t)EventPipeCommandId::StopTracing;
    message->Reserved = 0;
    message->SessionId = sessionId;

    return message;
}

bool EventPipeStopRequest::Process(IIpcEndpoint* pEndpoint, uint64_t sessionId)
{
    StopSessionMessage* pMessage = CreateStopMessage(sessionId);
    DWORD writtenBytes;
    if (!pEndpoint->Write(pMessage, pMessage->Size, &writtenBytes))
    {
        Error = ::GetLastError();
        std::cout << "Error while sending EventPipe Stop message to the CLR: 0x" << std::hex << Error << std::dec << "\n";
        delete pMessage;
        return false;
    }
    
    delete pMessage;

    ... // handle the response 

    return true;
}

When the stop command is received by the CLR, the remaining “data” (more on this in the next episode) is sent through the first IPC channel before being closed. This is how the code knows that the session can stop listening to the EventPipe.

The next episode will start to parse the nettrace stream of events.

Resources

• Episode 1 — Digging into the CLR Diagnostics IPC Protocol in C#
• Episode 2 — .NET Diagnostic IPC protocol: the C++ way
• Source code for the C++ implementation of CLR events listener
• Diagnostics IPC protocol documentation
• Microsoft.Diagnostics.NETCore.Client source code

Introduction

As I explained during a DotNext conference session, the .NET CLI tools such as dotnet-trace, dotnet-counter or dotnet-dump are communicating with the CLR thanks to Named Pipe on Windows and Domain Socket on Linux. Within the CLR, a diagnostic server thread is responsible for answering requests. A communication protocol allows a tool to send commands and expect responses. This Diagnostic IPC Protocol is pretty well documented in the dotnet Diagnostics repository.

Before going into the protocol details, here is a list of the available commands and their effect:

This series will detail how to communicate with a CLR using this protocol both in C# and in C++. Also note that processing CLR events thanks to EventPipe will also be covered.

Make it simple: use Microsoft.Diagnostics.NETCore.Client nuget

With TraceEvent nugget package, Microsoft provided a great library to easily listen to CLR events in C#. If you want to easily send CLR diagnostic IPC protocol commands to a CLR in a .NET process, Microsoft.Diagnostics.NETCore.Client nuget package is for you. Remember that EventPipe is implemented by .NET Core and .NET 5+ (so no .NET Framework support)

The Swiss knife class DiagnosticsClient gives you access to most of the commands plus a way to list .NET processes as a bonus:

If you want to get the pid of all supported running .NET applications, call the static GetPublishedProcesses() method. Beware that the pid of your own application will also be included.

1
2
3
4
5
6
7
8
9

private static void ListProcesses()
{
    var selfPid = Process.GetCurrentProcess().Id;
    foreach (var pid in DiagnosticsClient.GetPublishedProcesses())
    {
        var process = Process.GetProcessById(pid);
        Console.WriteLine($"{pid,6}{GetSeparator(pid == selfPid)}{process.ProcessName}");
    }
}