I’ve presented in depth the events emitted by the CLR in many posts to get insightful details about how the .NET runtime is working (lock contention, GC, allocations, …). Some .NET features are not implemented at the runtime level but at the Base Class Library (a.k.a. BCL) level. For example, if you are using HttpClient, you might want to measure how long it takes to get the response to your HTTP requests.

In this new series, I will describe how the BCL is using EventSource to emit the events, how you can listen to them with TraceEvent; focusing on HTTP requests.

Where do these events come from?

When you look for these BCL events in the documentation, you end up to this well-known event provides in .NET page and in the Framework libraries section. It lists the providers with their name and the emitted events with their keyword and verbosity. However, there is no detail about their payload! For example, for the “System.Net.Http” provider, you know that the RequestStart informal event is emitted each time “an HTTP request has started”. But you don’t know the corresponding url… Let me tell you how to get these tiny details.

Within the BCL, the classes responsible for emitting events derive from EventSource with the “Telemetry” suffix as a naming convention. They are decorated with an EventSource attribute to define their name (used as provider name by a listener) and their Guid that will be provided with each event. For example, here is the declaration of the class responsible for events related to sending HTTP requests:

1
2
3

[EventSource(Name = "System.Net.Http")]
internal sealed partial class HttpTelemetry : EventSource
{

If a Guid is not provided, one is automatically computed (see the corresponding code in Perfview).

Next, public helper methods decorated with a NonEvent attribute are provided to be used in the BCL code when it is needed to emit events. These methods are calling private methods decorated with an Event attribute to define their unique ID and their verbosity level:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

[NonEvent]
public void RequestStart(HttpRequestMessage request)
{
    ...
    RequestStart(
        request.RequestUri.Scheme,
        request.RequestUri.IdnHost,
        request.RequestUri.Port,
        request.RequestUri.PathAndQuery,
        (byte)request.Version.Major,
        (byte)request.Version.Minor,
        request.VersionPolicy);
} 

[Event(1, Level = EventLevel.Informational)]
private void RequestStart(string scheme, string host, int port, string pathAndQuery, byte versionMajor, byte versionMinor, HttpVersionPolicy versionPolicy)
{
    ...
    WriteEvent(eventId: 1, scheme, host, port, pathAndQuery, versionMajor, versionMinor, versionPolicy);
}

The different WriteEvent overloads are responsible for filling an array of EventData elements (each one contains a pointer to the data and its size) that is passed to EventSource.WriteEventCore. This helper methods dispatches the event to EventPipe/ETW pipelines.

Document the undocumented

Unlike CLR events, their payload is not explicitly described in a text file but, instead, you have to look at the implementation of the different WriteEvent overloads. The rest of this section provides the details of the sources I’ve looked at and the corresponding events payload.

HTTP

Sources: https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.Http/src/System/Net/Http/HttpTelemetry.cs Name: System.Net.Http Guid: d30b5633–7ef1–5485-b4e0–94979b102068

Sockets

Sources: https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketsTelemetry.cs

Name: System.Net.Sockets Guid: d5b2e7d4-b6ec-50ae-7cde-af89427ad21f

DNS

Sources: https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.NameResolution/src/System/Net/NameResolutionTelemetry.cs

Name: System.Net.NameResolution Guid: 4b326142-bfb5–5ed3–8585–7714181d14b0

Network Security

Sources: https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.Security/src/System/Net/Security/NetSecurityTelemetry.cs

Name: System.Net.Security  Guid: 7beee6b1-e3fa-5ddb-34be-1404ad0e2520

The next episode will describe how to listen to these events and extract their payload in C#.

Testing the statistical results

In parallel of the performance impact, it is important to validate the expected statistical distribution of the sampled allocations. Basically, I need to execute the same run of allocations multiple times in a row. Each run allocates the same number of instances of different types. For example, it is interesting to know if sampling instances of types with sizes proportional to a base value gives good results. Same question for totally different sized types or with Finalizers.

I would like to pass the number of runs to execute and a given scenario to a C# runner program and listen to the emitted events in another C# listener.

I’m facing 3 issues here:

• How many instances are allocated to validate the upscaling algorithm (sampled vs real count)
• What are the types I want to focus on because I don’t want to hard code them in the listener application.
• When does each run start?

It would be great if I could send the answer to these questions via events so the listener would know at runtime. Well… This is exactly what a class inherited from EventSource allows you to do!

In the runner application, I’ve defined the AllocationsRunEventSource that is decorated with the EventSource attribute to set its name that will be used as a provider name like Microsoft-Windows-DotNETRuntime for the .NET runtime provider.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

[EventSource(Name = "Allocations-Run")]
public class AllocationsRunEventSource : EventSource
{
    public static readonly AllocationsRunEventSource Log = new AllocationsRunEventSource();

The four implemented methods are defining which event ID for which verbosity will be emitted with which payload:
    [Event(600, Level = EventLevel.Informational)]
    public void StartRun(int iterationsCount, int allocationCount, string listOfTypes)
    {
        WriteEvent(eventId: 600, iterationsCount, allocationCount, listOfTypes);
    }

    [Event(601, Level = EventLevel.Informational)]
    public void StopRun()
    {
        WriteEvent(eventId: 601);
    }

    [Event(602, Level = EventLevel.Informational)]
    public void StartIteration(int iteration)
    {
        WriteEvent(eventId: 602, iteration);
    }

    [Event(603, Level = EventLevel.Informational)]
    public void StopIteration(int iteration)
    {
        WriteEvent(eventId: 603, iteration);
    }
}

To make the payload serialization and parsing easy, the list of types that will be allocated is passed as a string with the following format allocatedTypes = “Object24;Object48;Object72;Object32;Object64;Object96”.

The code of the runner calls these methods as expected at different moment of the execution:

1
2
3
4
5
6
7
8

AllocationsRunEventSource.Log.StartRun(iterations, allocationsCount, allocatedTypes);
 for (int i = 0; i < iterations; i++)
 {
     AllocationsRunEventSource.Log.StartIteration(i);
     allocationsRun.Allocate(allocationsCount);
     AllocationsRunEventSource.Log.StopIteration(i);
 }
 AllocationsRunEventSource.Log.StopRun();

Instead of recording the events with dotnet-trace, this time I’m using TraceEvent and Microsoft.Diagnostics.NETCore.Client to code a listener application. The code is very similar to what was presented for my dotnet-fullgc CLI tool except that I’m enabling the AllocationsRun provider corresponding to the event source of the runner in addition to the .NET runtime one:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

public static void PrintEventsLive(int processId)
{
    var providers = new List<EventPipeProvider>()
    {
        new EventPipeProvider(
                "Microsoft-Windows-DotNETRuntime",
                EventLevel.Verbose, // verbose is required for AllocationTick
                (long)0x80000000001 // new AllocationSamplingKeyword + GCKeyword
                ),
        new EventPipeProvider(
                "Allocations-Run",
                EventLevel.Informational
                ),
    };

The custom events from that provider are received via the source.Dynamic.All C# event:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

var client = new DiagnosticsClient(processId);
    using (var session = client.StartEventPipeSession(providers, false))
    {
        Console.WriteLine();

        Task streamTask = Task.Run(() =>
        {
            var source = new EventPipeEventSource(session.EventStream);
            _source = source;
            ClrTraceEventParser clrParser = new ClrTraceEventParser(source);
            clrParser.GCAllocationTick += OnAllocationTick;
            source.Dynamic.All += OnEvents;
            ...

Because TraceEvent is not already aware of the new AllocationSampled event emitted by the PR code, it will also be received via the same OnEvent handler:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

private static void OnEvents(TraceEvent eventData)
{
    if (eventData.ID == (TraceEventID)303)
    {
        // AllocationSampled parsing 
        ...
        return;
    }

    if (eventData.ID == (TraceEventID)600)  // Start run
    {
        // keep track of the expected types and the number of allocated instances 
        ...
        return;
    }

    if (eventData.ID == (TraceEventID)601)  // Stop run
    {
        // show the results of the run
        ...
        return;
    }

    if (eventData.ID == (TraceEventID)602)  // Start an iteration in a run
    {
        // reset for a new iteration
        ...
        return;
    }

    if (eventData.ID == (TraceEventID)603)  // Stop an iteration in a run
    {
        // Show iteration results
        ...
        return;
    }
}

The parsing of the payload of the run related events is done the same way as for AllocationSampled by a dedicated xxxData class:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

class AllocationsRunData
{
    const int EndOfStringCharLength = 2;
    private TraceEvent _payload;

    public AllocationsRunData(TraceEvent payload)
    {
        _payload = payload;

        ComputeFields();
    }

    public int Iterations;
    public int Count;
    public string AllocatedTypes;

    private void ComputeFields()
    {
        int offsetBeforeString = 4 + 4;

        Span<byte> data = _payload.EventData().AsSpan();
        Iterations = BitConverter.ToInt32(data.Slice(0, 4));
        Count = BitConverter.ToInt32(data.Slice(4, 4));
        AllocatedTypes = Encoding.Unicode.GetString(data.Slice(offsetBeforeString, _payload.EventDataLength - offsetBeforeString - EndOfStringCharLength));
    }
}

By keeping track of this data, it is possible to show each iteration results:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

> starts 100 iterations allocating 1000000 instances
0|
Tag  SCount  TCount          SSize          TSize   UnitSize     UpscaledSize  UpscaledCount  Name
--------------------------------------------------------------------------------------------------
 ST     247     384           5928           9216         24         24702711        1029279  Object24
 ST     322     106          10304           3392         32         32205122        1006410  Object32
 ST     435     509          20880          24432         48         43510266         906463  Object48
 ST     587     776          37568          49664         64         58718825         917481  Object64
 ST     747     481          53784          34632         72         74726662        1037870  Object72
 ST     958     916          91968          87936         96         95845392         998389  Object96

that integrates AllocationTick numbers too.

At the end of the run, error distribution per type is also computed over the iterations:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

Object72
-------------------------
   1  -10.5 %
   2   -8.9 %
   3   -8.4 %
   4   -7.8 %
   5   -6.4 %
        ...
  49    0.2 %
  50    0.2 %
  51    0.2 %
        ...
  96    6.8 %
  97    6.8 %
  98    7.7 %
  99    8.6 %
 100   10.0 %

You could use the same mechanisms (a custom EventSource to emit additional information and an EventPipe listener to aggregate the data) for your own usage. This is a different way to use EventSource rather than emitting events for monitoring like what is done by the BCL.

Testing the standalone GC

In addition to the usual .NET GC, it is needed to validate that the changes are also working for the standalone GC. Long story short, it is possible to replace the existing .NET garbage collector by your own implementation. For the test, I needed to check that the standalone GC clrgcexp.dll generated by the .NET compilation generates the expected AllocationSampled events when the corresponding keyword with informational verbosity is enabled.

Debugging NativeAOT scenarios

The final step was to implement the feature for the NativeAOT scenario. When you build your C# application for NativeAOT, a lot happens behind the scenes, based on compilers known by Visual Studio corresponding to the official released version of the .NET runtime. In my case, I needed to use the brand-new code of my local branch and debug some simple C# applications. The steps to reach that goal are not that simple.

First, you follow the steps given by the documentation to build AOT CLR in debug and libs in release:

1
2

build clr.aot+lib -rc debug -lc release
build -c release

Then, open src\coreclr\tools\aot\ilc.sln

the repro project contains a program.cs file where you write the C# code you want to test and debug.

When you build a NativeAOT application, you need to select if you want the runtime that emits events or not. This is done by setting the EventSourceSupport to true in the .csproj:

Add the following in the .csproj to get events:

1

<EventSourceSupport>true</EventSourceSupport>

However, with the repro project, you need to change ILCompiler.csproj in a different way:

1

<ReproResponseLines Include="--feature:System.Diagnostics.Tracing.EventSource.IsSupported=true" />

Also, change the reproNative.vcxproj file to bind to eventpipe-enabled.lib instead of eventpipe-disabled.lib for the platform/configuration you want to debug:

1
2
3
4
5
6

<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
    ...
    <Link>
      ...
      <AdditionalDependencies>...$(ArtifactsRoot)bin\coreclr\windows.x64.Debug\aotsdk\eventpipe-enabled.lib;...</AdditionalDependencies>
    </Link>

Then, build repro in Debug x64

Next, change the target for the ILCompiler project:

Build and run it to generate the .obj file corresponding to the repro project

Finally, open src\coreclr\tools\aot\ILCompiler\reproNative\reproNative.vcxproj that will allow you to debug the program.cs you’ve just built!

Introduction

If you have read Microsoft documentation, you probably know that it is not recommended to trigger a garbage collection in your application code. However, in some troubleshooting cases, you might want to trigger a GC. For example, you don’t want to wait for a full gen2 compacting GC to figure out if your application is really leaking memory. For web applications, you can imagine having a hidden HTTP end point that simply call GC.Collect. What if you could simply call a command line tool to trigger a GC in any .NET application? This is exactly what my new dotnet-fullgc CLI tool is doing!

This Is The Way

If you have read a few of my past posts about the .NET diagnostics mechanisms, you know that you can send commands to another .NET process via EventPipes. Well… there is no explicit command to trigger a GC.

I have also explained how you could listen to events emitted by the CLR by enabling a provider with a set of keywords with a verbosity corresponding to the events you are interested in. This is how dotnet-trace and Perfview are collecting these events. If you want to trigger a GC, you simply need to enable the Microsoft-Windows-DotNETRuntime provider with the GCHeapCollect (= 0x800000L) keyword and an informal verbosity. Yes: it is as simple as that, and it is also working for .NET Framework!

So, you could trigger a GC with dotnet-trace via the following command line:

1

dotnet trace collect -p <process id> - providers Microsoft-Windows-DotNETRuntime:0x800000:4 - duration 00:00:01

However, a .nettrace file would be generated and it is not possible to pass parameters (more on this later).

The rest of this post shows the C# code to obtain the same result. With Microsoft.Diagnostics.NETCore.Client and TraceEvent, you create a DiagnosticsClient with the ID of the process you are interested in:

1

var client = new DiagnosticsClient(processId);

The next step is to start an EventPipe session with the right provider, keyword and verbosity:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

var providers = new List<EventPipeProvider>()
{
    new EventPipeProvider(
        "Microsoft-Windows-DotNETRuntime",
        EventLevel.Informational,
        (long)ClrTraceEventParser.Keywords.GCHeapCollect,
        Arguments  // more on this later
        ),
};
using (var session = client.StartEventPipeSession(providers, false))
{

The source must be processed in another thread to avoid blocking the main thread:

1
2
3
4
5
6
7

Task streamTask = Task.Run(() =>
    {
        // without source to process, session.Stop() will not return
        var source = new EventPipeEventSource(session.EventStream);

        source.Process();
    });

The question to answer is how to stop the session: just create another task that waits for a second before stopping the session to exit from the Process call:

1
2
3
4
5
6
7
8

Task inputTask = Task.Run(() =>
    {
        Thread.Sleep(1000);
        session.Stop();
    });

    Task.WaitAny(streamTask, inputTask);
}

That’s all!

Pitfalls

Unfortunately, I faced a couple of issue during the implementation of the tool.

What is your number?

If you look at the TraceEvent implementation, you could imagine that is it possible to pass a GC ID as a parameter to the .NET provider:

1
2
3
4
5

//
// Summary:
//     Triggers a GC. Can pass a 64 bit value that will be logged with the GC Start
//     event so you know which GC you actually triggered.
GCHeapCollect = 0x800000L,

Unfortunately, this is not supported and the reasons are explained below.

If you check the .NET source code and look at EtwCallbackCommon(), you can indeed see that a numeric ID can be passed to ETW::GCLog::ForceGC(l64ClientSequenceNumber) and passed as ID in GCStart event instead of the one incrementally increased collection after collection.

At the diagnostics client level, you have the opportunity to pass a dictionary of key/value string pairs. This dictionary is used when defining the EventPipeProvider to be enabled:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

Dictionary<string, string> arguments = new Dictionary<string, string>();
arguments.Add("Id", "42");
var providers = new List<EventPipeProvider>()
{
    new EventPipeProvider(
        "Microsoft-Windows-DotNETRuntime",
        EventLevel.Informational,
        (long)ClrTraceEventParser.Keywords.GCHeapCollect,
        arguments
        ),
};

The diagnostics client transforms the dictionary into a string with key=value pairs separated by ‘;’ such as “Id=42;AnotherId=AnotherValue;…” and serializes it as is in the payload when sending a CollectTraces command.

The question is what identifier is expected by the CLR? To answer this question, you need to look at the code of provider_invoke_callback. The “key=value;…” string is stored into a buffer and each = and ; characters are transformed into \0. So, “Id=42” is transformed into Id\042\0.

The next step is done by ep_event_filter_desc_init() that uses that buffer to fill up the 3 fields of an EventFilterDescriptor:

1
2
3

uint64_t ptr = address of the buffer
uint32_t size = size of the buffer (=6 for id\042\0)
uint32_t type = 0

And finally, EtwCallbackCommon receives the filterdata and tries the following to get the collection id:

1
2
3
4
5
6
7

PEVENT_FILTER_DESCRIPTOR FilterData = (PEVENT_FILTER_DESCRIPTOR)pFilterData;
if ((FilterData != NULL) &&
   (FilterData->Type == 1) &&
   (FilterData->Size == sizeof(l64ClientSequenceNumber)))
{
    l64ClientSequenceNumber = *(LONGLONG *) (FilterData->Ptr);
}

As you can see, it will fail because:

• the received type value is 0
• the received size is not the size of a 64 bit number
• the ptr field does not point to the value as 64 bit number (but as a string)

An issue has been filed with a possible fix.

Only one collection please!

When I test my dotnet-fullgc with dotnet-gcstats, I always see 2 collections!

We investigated with my colleague Kevin Gosse and he created an issue for that. The EtwCallback() function is called whenever a session is enabled or disabled. Unfortunately, the call to ForceGC is made in both cases: so, when the session is stopped, a second garbage collection is triggered.

Next steps

Feel free to install dotnet-fullgc on your machine with dotnet tool install -g dotnet-fullgc.

Next, use dotnet fullgc to trigger two gen2 full garbage collections in your running .NET processes.

Introduction

While working on the second edition of Pro .NET Memory Management, it was needed to get statistics about each garbage collection to explain the condemned generation and other decisions taken by the GC. This post explains the different internal data structures used by the GC and how to get their value for each collection. Some require debugging the CLR and others are emitted via events. For the latter, I will show how I wrote the new dotnet-gcstats CLI tool to collect them and a personal Perfview GCStats displaying live data, garbage collection after garbage collection.

High level view of GC Internals

With regions, the GC keeps track of managed memory allocated by your application in instances of the gc_heap class. In Workstation mode, only 1 instance exists and in Server mode, by default, 1 instance is created per core. Each gc_heap keeps track of its 5 generations (gen0, gen1, gen2, Large Object Heap and Pinned Object Heap) in an array of 5 generation instances. Each generation references its dedicated regions wrapped by instances of heap_segment. These regions are reserved from a giant part of the process address space and committed as needed.

During a garbage collection, the GC code relies on global fields per gc_heap:

1
2
3
4

static gc_mechanisms settings;
gc_history_global gc_data_global;  // for non background GC including foreground GC during a background
gc_history_global bgc_data_global; // for background GC only
static dynamic_data dynamic_data_table[total_generation_count = 5];

The settings field contains a few interesting fields:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

class gc_mechanisms
{
public:
    gc_index; // starts from 1 for the first GC
    int condemned_generation;  // generation to collect
    BOOL compaction;  // true when compaction instead of sweep
    BOOL loh_compaction;  // true when LOH needs compaction
    uint32_t concurrent;  // 1 = concurrent/background GC 
    gc_reason reason;  // trigger reason
    gc_pause_mode pause_mode;  // see GCSettings.LatencyMode
    ...
};

including the trigger reason that will tell if your code called GC.Collect (i.e. induced), or if it was due to a LOH or SOH allocation for example. If compaction is true, a compacting GC will happen (instead of a sweeping one).

The gc/bgc_data_global contains almost the same information:

1
2
3
4
5
6
7
8
9

struct gc_history_global
{
    uint32_t num_heaps;  // number of gc_heap instances
    int condemned_generation;
    gc_reason reason;
    int pause_mode;
    uint32_t mem_pressure; 
    uint32_t global_mechanisms_p;
};

Most of the fields are available from different events:

• GCStart: gc_index in Count, condemned_generation in Depth, reason in Reason
• GCGlobalHeapHistory: pause_mode in PauseMode and some others in GlobalMechanisms

Which generation to collect = condemned generation

The computation of the condemned_generation is complicated and relies on many factors including metrics stored for each “generation” (gen0, gen1, gen2, LOH and POH) in an array of dynamic_data called dynamic_data_table. The dynamic_data class contains a few fields used by the GC to take decisions such as when a collection should be triggered and which generation to condemn:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

class dynamic_data
{
public:
    ptrdiff_t new_allocation;     // remaining budget = budget - allocated
    size_t    desired_allocation; // budget to trigger a GC

    // # of bytes taken by survived objects after mark.
    size_t    survived_size;

    // # of bytes taken by survived pinned plugs after mark.
    size_t    pinned_survived_size;

    // total object size after a GC, ie, doesn't include fragmentation
    size_t    current_size;
    size_t    promoted_size;
    size_t    fragmentation;
};

Most of these fields are found in the payload of GCPerHeapHistory or GCHeapStat events. However, the most interesting one, new_allocation is not available. Why is it interesting? Because it would give you which generation had its budget exceeded. It is initialized with the generation budget at the end of a GC and then, each time an allocation context gets created, its size is deducted from it. When it reaches 0, it means that the budget is exceeded, and a collection should happen.

Since I needed to debug the CLR to better understand all these algorithms, I added a breakpoint at the beginning of gc_heap::garbage_collect with the following action:

#{settings.gc_index}[{gc_trigger_reason}]{"\n",s8b} new_allocation(0) = {dynamic_data_table[0].new_allocation}{"\n",s8b} desired_allocation(0) = {dynamic_data_table[0].desired_allocation}{"\n",s8b} begin_data_size(0) = {dynamic_data_table[0].begin_data_size}{"\n",s8b} promoted_size(0) = {dynamic_data_table[0].promoted_size}{"\n",s8b}-{"\n",s8b} new_allocation(1) = 
...
{dynamic_data_table[4].new_allocation}{"\n",s8b} desired_allocation(4) = {dynamic_data_table[4].desired_allocation}{"\n",s8b} begin_data_size(4) = {dynamic_data_table[4].begin_data_size}{"\n",s8b} promoted_size(4) = {dynamic_data_table[4].promoted_size}{"\n",s8b}__________{"\n",s8b}

And now, each time a GC happens, I get the corresponding log in my Output pane in Visual Studio:

#2[reason_alloc_soh (0)]
 new_allocation(0) = -22728
 desired_allocation(0) = 134217728
 begin_data_size(0) = 8391376
 promoted_size(0) = 8383432
-
 new_allocation(1) = -5910416
 desired_allocation(1) = 2473016
 begin_data_size(1) = 375528
 promoted_size(1) = 353288
-
 new_allocation(2) = -91144
 desired_allocation(2) = 262144
 begin_data_size(2) = 0
 promoted_size(2) = 0
-
 new_allocation(3) = 28000088
 desired_allocation(3) = 28000088
 begin_data_size(3) = 8000024
 promoted_size(3) = 8000024
-
 new_allocation(4) = 3145728
 desired_allocation(4) = 3145728
 begin_data_size(4) = 32712
 promoted_size(4) = 32712

As you can see, gen0, gen1 and gen2 have all their budget exceeded (i.e. their new_allocation is negative) and it explains why a simple gen0 collection (from allocation in SOH = gen0) becomes a gen2 collection. If you wonder how gen1 and gen2 budgets are exceeded as your application is only allocating in gen0, you need to understand that when a GC copy surviving objects from one younger generation to the older, they are counted as allocations in the older and subtracted from its new_allocation metric.

The GC is encoding the different steps leading to the final condemned generation in a 32 bit value stored in a gen_to_condemn_tuning field that allows you to get:

• initial condemned generation,
• final generation to condemn,
• which generation’s budget is exceeded.

The value of the last one corresponds to the highest generation for which its new_allocation was negative.

This information is available in the CondemnReasons0 field of the GCPerHeapHistory event, and you need some arithmetic to get the generation you want:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

private const int gen_initial = 0;          // indicates the initial gen to condemn.
private const int gen_final_per_heap = 1;   // indicates the final gen to condemn per heap.
private const int gen_alloc_budget = 2;     // indicates which gen's budget is exceeded.

private const int InitialGenMask = 0x0 + 0x1 + 0x2;

static int GetGen(int val, int reason)
{
    int gen = (val >> 2 * reason) & InitialGenMask;
    return gen;
}

Building your own tool

Even though I could dig into the different matrices available in the Perfview’s GCStats view or its export to Excel, I decided to write dotnet-gcstats. This CLI tool listens to the CLR events emitted by a .NET application thanks to Microsoft.Diagnostics.NETCore.Client (connect to the application EventPipe) and TraceEvent (receive and analyze the CLR events).

The code is amazingly simple:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

var providers = new List<EventPipeProvider>()
{
    new EventPipeProvider("Microsoft-Windows-DotNETRuntime",
        EventLevel.Informational, (long)ClrTraceEventParser.Keywords.GC),
};
var client = new DiagnosticsClient(processId);

using (var session = client.StartEventPipeSession(providers, false))
{
    Console.WriteLine();

    Task streamTask = Task.Run(() =>
    {
        var source = new EventPipeEventSource(session.EventStream);

        ClrTraceEventParser clrParser = new ClrTraceEventParser(source);
        clrParser.GCPerHeapHistory += OnGCPerHeapHistory;
        clrParser.GCStart += OnGCStart;
        clrParser.GCGlobalHeapHistory += OnGCGlobalHeapHistory;

        try
        {
            source.Process();
        }
        catch (Exception e)
        {
            ShowError($"Error encountered while processing events: {e.Message}");
        }
    });

Each event handler is responsible for extracting and translating the interesting fields of its event payload with a few color enhancements:

• GCStart: collection count and reason (highlight induced collections).
• GCGlobalHeapHistory: condemned generation, pause mode and memory pressure.
• GCPerHeapHistory: starting -> final condemned generation and for each heap, budget, begin size, begin obj size, final size, promoted size and fragmentation.

The final step was to transform a simple console application into a .NET CLI tool that everyone will be able to install with dotnet tool install -g dotnet-gcstats and use with dotnet gcstats . I followed the documentation by adding the following to the project file:

1
2
3
4
5
6

<PropertyGroup>
    <PackAsTool>true</PackAsTool>
    <ToolCommandName>dotnet-gcstats</ToolCommandName>
    <PackageOutputPath>./nupkg</PackageOutputPath>
    <GeneratePackageOnBuild>true</GeneratePackageOnBuild>
  </PropertyGroup>

In addition, I provided a few additional details:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

<PropertyGroup>
    <PackageId>dotnet-gcstats</PackageId>
    <PackageVersion>1.0.0</PackageVersion>
    <Title>dotnet-gcstats</Title>
    <Authors>christophe Nasarre</Authors>
    <Owners>chrisnas</Owners>
    <RepositoryUrl>https://github.com/chrisnas</RepositoryUrl>
    <RepositoryType>git</RepositoryType>
    <PackageProjectUrl>https://github.com/chrisnas/GCStats</PackageProjectUrl>
    <PackageLicenseFile>LICENSE</PackageLicenseFile>
    <Description>Global CLI tool to display live statistics during .NET garbage collections</Description>
    <PackageReleaseNotes>Initial release</PackageReleaseNotes>
    <Copyright>Copyright Christophe Nasarre 2024-$([System.DateTime]::UtcNow.ToString(yyyy))</Copyright>
    <PackageTags>.NET TraceEvent CLR GC</PackageTags>
  </PropertyGroup>

Once built, I simply uploaded the generated package to nuget.org et voila!

Now, you should be able to better understand why some collections are triggered:

And if it is not enough, wait for reading the second edition of Pro .NET Memory Management ;^)

Introduction

During the Datadog R&D week, my goal is to mimic the generation of a .gcdump from our .NET profiler. I’ve already written most of the code for a previous post and after changing the required plumbing, it is time to test the workflow.

Unfortunately, I’m facing the dreaded stack corruption dialog:

The rest of the post explains the different steps I’m following to investigate this issue.

Trying to understand the problem

This stack check is done by the debug version of the C Runtime library by basically adding some special bytes on the stack before calling a function and checking these bytes are not tampered when returning from the call.

So, the next step is to debug the application to get more details and at least where in the code the problem happened:

The failed check occurs at the end of a function that looks like the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

bool GcDumpProvider::Get(IGcDumpProvider::gcdump_t& gcDump)
{
    // trigger the GC and get the dump
    GcDump gcd(::GetCurrentProcessId());
    gcd.TriggerDump();

    auto const& dump = gcd.GetGcDumpState();
    auto& types = dump._types;
    for (auto& type : types)
    {
        auto& typeInfo = type.second;

        uint64_t instancesCount = typeInfo._instances.size();
        uint64_t instancesSize = 0;
        for (size_t i = 0; i < instancesCount; i++)
        {
            instancesSize += typeInfo._instances[i]._size;
        }

        gcDump.push_back({typeInfo._name, instancesCount, instancesSize});
    }
    return true;
}

There is much more code behind this; especially in the TriggerDump() function. I have already tested this code many times when I dug into the .gcdump generation process without facing this stack corruption. I’m spending a few hours digging back into the code because:

• I’m not running inside the profiled process and not outside like in the blog post
• I’m introducing a “slight” change because I need to exit the communication with the CLR when the GC ends.
• I need to mention that Visual Studio is refusing to debug (Step Over or Step Into) and only Run to Cursor was possible due to mixed mode (managed and native) debugging. So, I created a simple native console application with my updated code for easier and faster debugging.

After a couple of hours, it is time to go back to the Get() implementation because I do not find anything obviously wrong.

Make it simpler and simpler and simpler again

In that type of situation, I recommend the “remove code and debug” strategy (from “divide and conquer” attributed to Julius Cesar). From the simplified console application, the code now looks like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

bool GetGcDump(int pid, IGcDumpProvider::gcdump_t& gcDump)
{
    GcDump gcd(pid);

    // no more gcd.TriggerDump()

    // no more for (auto& type : types)

    return true;
}

No more complicated call nor iteration on the vector of results. Guess what? Same stack corruption.

It is time to go one level deeper: what does this GcDump class look like?

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

class GcDump
{
public:
    GcDump(int pid);
    ~GcDump();

...

private:
    int _pid;
    DiagnosticsClient* _pClient;
    EventPipeSession* _pSession;
    HANDLE _hListenerThread;
    GcDumpState _gcDumpState;
};

The constructor is setting the fields value to zero/nullptr and the destructor is cleaning up these fields if necessary. Since TriggerDump() is no more called, these fields never change.

I’m commenting out all fields until only _gcDumpState remains and it continues to crash. When is it commented out, it is not more crashing.

Use your debugger Luke!

Let’s turn to the GcDumpState class that is even simpler:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

class GcDumpState
{
public:
    GcDumpState();
    ~GcDumpState();

public:
    // fields removed for brevity

private:
    bool _isStarted;
    bool _hasEnded;
    uint32_t _collectionIndex;
};

The code of the destructor only sends a trace to the console (removing it completely does not fix the issue) and here is the constructor code:

1
2
3
4
5
6

GcDumpState::GcDumpState()
{
    _isStarted = false;
    _hasEnded = false;
    _collectionIndex = 0;
}

Again, same strategy: remove one field after the other. This time, the code stops crashing if the two Boolean fields are removed or if the 32 bits index is removed. If the index field is not set, no more corruption!

How could this assignation corrupt the stack? It is time to use the Visual Studio debugger to better understand what is going on.

First, set a breakpoint on the assignment line and click Debug | Disassembly to see the corresponding assembly code:

The two lines of assembly code are easy to understand:

1
2

mov rax,qword ptr [this]  
mov dword ptr [rax+4],0

• The this pointer is stored in the rax register
• The 32 bits (mov dword) memory starting 4 bytes after the beginning of the object pointed to by this, is set to 0

I enter “this” in a Memory panel (Debug | Windows | Memory xx)

And Visual Studio gives me the corresponding address and the content of the memory there:

Pressing F10 twice to Step Over the two assembly instructions and this is confirmed:

Instead of storing the 32 bits 0 value just after the two bytes corresponding to the bool fields, it is stored 2 bytes away. It looks like a padding is added on my behalf.

I change the build settings for the GcDumpState.cpp file to enable all warnings:

The compilation confirms what has been seen in the memory:

1

GcDumpState.h(41,14): warning C4820: 'GcDumpState': '2' bytes padding added after data member 'GcDumpState::_hasEnded'

What’s next?

My understanding is that the compiler is:

• adding a 2 bytes padding to align the 32 bits index field
• generating the constructor code based on that padding
• but the stack corruption checking code does not take it into account

The solution is to either add a uint16_t field after the 2 bool fields as an explicit padding or use #pragma pack(1) to decorate the class definition.

However, this looks really weird to me. We should have faced this issue a long time ago because we were never cautious about alignment in all the classes and structures that we allocate in our code. To validate the assumption, I’m writing a small reproduction code outside of all the .gcdump complexity. And guess what? I’m not able to reproduce the stack corruption. Another mystery of the C++ compilation optimizations probably…

This is the end of my debugging Friday at Datadog :^)

Introduction

When I started to work on the second edition of Pro .NET Memory Management : For Better Code, Performance, and Scalability by Konrad Kokosa, I already spent some time in the CLR code for a couple of pull requests related to the garbage collector. However, updating the book to cover 5 new versions of .NET requires looking at new APIs but also digging deep inside the CLR (and especially the GC) hundreds of thousand lines of code!

The first step is to install Visual Studio 2022 Preview that allows you to compile and run projects targeting .NET 8. Then, goto https://github.com/dotnet/runtime and git clone the tag of the .NET 8 preview version you have installed.

That way, you will be able to directly run the same version that you will debug.

And now, what are the next steps?

The goal of this post is to share with you the tips and tricks I used to navigate into the CLR implementation so you could better understand how things are working.

From C# to C++

As a .NET developer, I’m used to the APIs provided by the Base Class Library built on top of the CLR. Let’s take as an example the following code that is using the GC.AllocateArray method that allows you to allocate a pinned in memory array and available since .NET 5.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

using System;

internal class Program
{
    static void Main(string[] args)
    {
        byte[] pinned = GC.AllocateArray<byte>(90000, true);
        Console.WriteLine($"generation = {GC.GetGeneration(pinned)}");
    }
}

When you Ctrl+click the method name (or use F12), thanks to Source Link integration, you go to its implementation where you can even set breakpoint:

If you don’t use Visual Studio, you could open the generated assembly into a decompiler such as ILSpy or DnSpy. The latter even allows you to set breakpoints and debug the disassembly IL without any source.

In both cases, only the managed implementation will be available: you soon end up to an “internal call” corresponding to a native function implemented by the CLR. The managed methods are decorated with the MethodImplOptions.InternalCall attribute.

For the garbage collector code, you can look into the GC.CoreCLR.cs file where these methods are defined. You can note some methods decorated with the DllImport attribute to bind to native functions exported by a “QCall” library. There is an optimized path in P/Invoke done by the JIT to transform these calls not like a usual LoadLibrary/GetProcAddress as you could expect. Instead, they will be routed to the exported methods by coredll.dll and defined in the s_QCall array in qcallentrypoints.cpp. But where to look further for the native implementation?

Instead of searching among the thousands of files, focus on comutilnative.h that defines the signature of most exported functions. The implementation of the exported native functions is found in comutilnative.cpp. This is where you should start your journey in the native implementation of the CLR. For the list of all functions called by the libraries in the runtime, look at the ecalllist.h file (around gGCInterfaceFuncs and gGCSettingsFuncs specifically for the GC).

Note that you might also find some implementations under the classlibnative folder like in the system.cpp file for GCSettings.IsServerGC.

CLR Source code debugging

It is nice to know that the implementation of most CLR exported native functions used by the BCL is in comutilnative.cpp. For the GC, the functions are either statics from the GCInterface class or static functions prefixed by GCInterface_; I don’t know why all are not part of GCInterface…

When you look at the GC-related methods implementation, a lot are calling methods from the instance returned by GCHeapUtilities::GetGCHeap() that corresponds to the static g_pGCHeap global variable. It is interesting to follow the threads of calls like that, but I have to admit that, after a few hops, I’m starting to get lost. So, I’m drawing boxes for types on a piece of paper and arrays from their fields to other types as boxes.

However, with a code base that big, I definitively prefer to set breakpoints and write a small C# application to call the methods I’m interested in and see what data structures are used in the different layers of implementation. Don’t be scared: WinDBG is not required to achieve this goal. As this page explains, you need to type the following commands in a shell at the root of the repo:

.\build.cmd -s clr -c Debug .\build.cmd clr.nativeprereqs -a x64 -c debug .\build.cmd -msbuild

The last command generates a CoreCLR.sln solution file in artifacts\obj\coreclr\windows.x64.Debug\ide) that you can open in Visual Studio 2022 Preview.

In VS, right-click the INSTALL project, select Properties and setup the Debugging properties

Here are the details of each property:

It could be interesting to set some environment variables such as DOTNET_gcServer to 1 for a GC Server configuration instead of workstation. In that case, click the <Edit..> choice in the combo-box:

And update the textbox at the top:

The final step is to set this project as the startup project:

You are now able to set the breakpoint you want in the native code of the CLR and type F5/Debug in Visual Studio to step into the code!

And what about the assembly code?

Some specific data structures, such as the NonGC Heap, are used by the JIT compiler when generating the assembly code from the IL compiled from your C# code. It means that you need to look at that JITted code to fully understand what is going on.

A first way to get it is to use https://sharplab.io/, type your C# code and select x64 for Core of x86/x64 for Framework:

But as you can see from this screenshot, it is using the .NET 7 compiler. What if you would like to see the .NET 8 compilation result just in case something changed?

The solution I’m using is to generate a memory dump with procdump -ma of a test application. Before opening the dump in WinDBG, there is something you should be aware of: with the tiered compilation, you will need to call a method several times before the final optimized assembly code gets JITed. Or… decorate the method you are interested in with the [MethodImpl(MethodImplOptions.AggressiveOptimization)] attribute to instruct the JIT to directly generate the most optimized tier.

Once the dump loaded in WinDBG, the first step is to get the MethodTable pointer corresponding to the method you are interested in. For that, use the name2ee SOS command:

Click the link corresponding to MethodDesc to run the dumpmd SOS command:

The last step is to click the link corresponding to CodeAddr to run the U command and see the JITted assembly code:

If you compare this code to get the “Hello, World!” string, with the one shown by sharplab,

1
2
3
4

Program.Hello()
    L0000: mov rcx, 0x257f7cbc368
    L000a: mov rcx, [rcx]
    L000d: jmp qword ptr [0x7ff9c9bd7f48]

you might notice a tiny difference: there is one less indirection in .NET 8!  But this is another story that will be told in the second edition of the “Pro .NET Memory Management: For Better Code, Performance, and Scalability” book ;^)

The .NET runtime (both .NET Framework and .NET Core) allows you to generate a lightweight dump containing the allocated type instances count and references including roots. They are usually generated into .gcdump files by tools such as Perfview or dotnet-gcdump and can also be viewed in Visual Studio. In addition to a view of the allocated types in the managed heap, these files are often used during memory leak investigations because they are much smaller than full memory dump and they contain explicit dependency information between types up to their roots.

The goal of this document is to dig into their generation and see how to leverage the same mechanisms from the .NET CLR for live heap profiling and memory leak detection.

Simply listening to CLR events

Both .NET Framework and .NET Core work the same way to allow a tool to generate a .gcdump file:

• It seems that the type table needs to be flushed for .NET Core by first creating and immediately closing an event session with the Microsoft-DotNETCore-SampleProfiler provider.
• You create an event session (either through ETW for Framework or EventPipe for Core) where the Microsoft-Windows-DotNETRuntime provider is enabled with a verbose level for a long list of keywords corresponding to the 0x1980001 value:

This will trigger an induced non concurrent gen2 garbage collection during which the GC will walk the remaining live objects with their size and emit tons of events, most of them not documented:

• GCStart: wait for the first induced gen2 foreground GC (Depth = 2, Type = GCType.NonConcurrentGC and Reason = GCReason.Induced)
• GCStop: detect when the heap walk is over
• BulkType: type blocks are enqueued
• GCBulkNode: node blocks are enqueued
• GCBulkEdge: edge blocks are enqueued
• GCBulkRootEdge: enqueue non-weak reference roots (GCRootFlag & GCRootFlags.WeakRef != 0) based on their GCRootKind
• GCBulkRootStaticVar: static variable blocks
• GCBulkRCW and GCBulkRootCCW for Runtime Callable Wrappers and COM Callable Wrappers COM-based roots
• GCBulkRootConditionalWeakTableElementEdge: ??
• GCGenerationRange: one for each managed heap segment with generations boundary (not really needed to build the dependency graph but interesting to figure out objects in generation 2)

The payload of most of these events contains arrays of instances with their dependencies. For Perfview and dotnet-gcdump, the whole graph is built in the ConvertHeapDataToGraph method after the garbage collection ends.

Deciphering CLR events payload

When you look at the dotnet-gcdump implementation, you realize that most of the complex code to compute the .gcdump files is physically copied from the Perfview repository.

The GCBulkXXX events payload contains an array of elements; each element being different. The common Count field contains the number of elements. If the element contains a string such as for BulkType, it means that each one has a different size and the string must be read entirely from the payload before accessing the next element.

Type definition

To avoid sending expensive type names all the time, each type will have an identifier that will be used in the GCBulkXXX-nodes related events.

The BulkType events contain an array of types definition elements with the following layout:

• TypeID: id of the type (i.e. pointer to the Method Table)
• ModuleID: id of the module where the type is defined
• TypeNameID: if Name is empty, use this address as a name
• Flags: if this bitset contains 0x8, it is an array so append “[]” to the name in that case
• CorElementType:?
• Name: Unicode string corresponding to the name of the type where `xxx need to be removed in case of generics
• TypeParameterCount: for generics but not used
• Array of type parameter: for generics but not used

Once the type mappings are known, it becomes possible to build the graph of live type instances instead of just nodes with IDs.

Listing Live Objects and References

The live objects are sent in the GCBulkNode events payload:

• Index: incrementing index of the bulk starting from 0
• Count: number of objects in the array

followed by an array of Values

• Address: address in memory where the object is stored
• Size: size of the object (including for arrays)
• TypeID: identifier of the object class usable
• EdgeCount: number of objects pointed to by this object (i.e. non null reference type fields)

Each event contains an array of live objects identified by their address. The Size and TypeID fields are easy to understand but what does the EdgeCount field represent? This is the number of objects that are referenced by this object. At the code level, this is the count of non-null reference type fields. For example, if a class A defines one integer field and a second one as a reference to an instance of type B, the EdgeCount would be 1 (because an integer is not a reference type).

So the next question is from where do you get which instances are referenced by the objects received in GCBulkNode payload? Since these objects are in memory, they are part of the GCBulkNode events payload but where is the relationship between the EdgeCount value and the corresponding objects? You will have to rebuild this relationship because the referenced objects are received in the GCBulkEdge events payload:

• Index: incrementing index of the bulk starting from 0
• Count: number of objects in the array

followed by an array of Values

• Value: address in memory where the object is stored
• ReferencingFieldID: this is not used and is always 0

Again, an array of elements is received as payload and the only interesting information is the address of the object.

But the magic is that both nodes and edges events payload are “in sync”: when an object is read from a GCBulkNode array with let’s say 2 as EdgeCount value, the current 2 elements in the array of the GCBulkEdge payload will contain the addresses of these 2 objects. If the next object in the GCBulkNode array has 1 as EdgeCount value, the next element in the array of the GCBulkEdge payload will be address of this object as shown in the following figure:

It means that both payloads must be iterated in sync.

Just with these two events, it is possible to get a detailed view of the objects still used in memory with their size and their type like what you get with dotnet-gcdump report or !sos.dumpheap -stat.

With the nodes (live objects) and the edges (objects referenced by each object), it is now possible to build the reference graph of live objects.

Listing Roots

In addition to the objects related events, the GC is also emitting events to list the roots that are referencing objects in the managed heap from the stack, statics, handles or other weird places.

The most interesting roots are available thanks to the following events:

GCBulkRootEdge

• Index: incrementing index of the bulk starting from 0
• Count: number of roots in the array

followed by an array of Values

• RootedNodeAddress: address in memory of the root object
• GCRootKind: is Stack for local variables
• GCRootFlag: if not a local variable, could be RefCounted, Finalizer, strong/pinning handles, or other handles
• GCRootID: address of the handle that points to the root object

The static ones are given by the following events:

GCBulkRootStaticVar

• Count: number of static roots in the array
• AppDomainID: app domain in which the static variable is stored

followed by an array of Values

• GCRootID:address of the handle that points to the root object
• ObjectID: address of the root object
• TypeID: type identifier of the root object
• Flags: could be ThreadLocal or not
• FieldName: Unicode string corresponding to the name of the field in the type corresponding to the root

Other rarely used roots are available from the GCBulkRootConditionalWeakTableElementEdge and COM-related ones from GCBulkRootCCW/GCBulkRCW with the ref count for example.

So, each of these events provides arrays of root objects addresses. These can be used in conjunction with the reference graph built from the previous node/edge events to identify the reason why objects stay in memory. Like for the ICorProfilerCallback::ObjectReferences usage previously described, it is needed to rebuild the inverse reference chain from the reference graph:

and deal with cycles:

These roots could be an expected cache or a memory leak. For the memory leak scenario, filtering on objects in the gen2 could definitely help. This is where the GCGenerationRange events could help because their payload contains the ranges of memory addresses in each segment with the corresponding generation:

GCGenerationRange

• Generation: generation of the segment
• RangeStart: address of the start of the segment
• RangeUsedLength: size of the committed part of the segment
• RangeReservedLength: size of the reserved part of the segment

When an address fits inside RangeStart and RangeStart + RangeUsedLength, it is part of this segment. The generation of the segment could be 0, 1 or 2 for the ephemeral segments, 3 for the Large Object Heap, and 4 for the Pinned Object Heap.

Integration with a .NET Profiler

As a .NET Profiler, it is possible to listen to CLR events via ICorProfilerCallback::EventPipeEventDelivered. If the same keywords as 0x1980001 have been enabled thanks to ICorProfilerInfo12::EventPipeStartSession, the corresponding messages will be received and you have to keep track of the fact that a gcdump is in progress. This should not be a big deal because only the GC keyword (0x1) might be already used and events describing the collections will be processed anyway. There won’t be duplication of events in that case.

However, since it is needed to start a session with the right keyword to trigger the special garbage collection, the ICorProfilerCallback mechanism cannot be used to continuously process the corresponding specific messages. This one time EventPipe session should be started independently by manually connecting to the EventPipe of the currently running CLR as described in details in this blog series.

You are now ready to integrate this feature of the CLR without the need to install a tool!

Introduction

It’s been almost 12 years since I wrote LeakShell to help me automate the search of memory leaks in .NET. The idea was simple: compare 2 memory dumps of a leaking .NET application to show the types with increasing instances count.

Today, you could use Visual Studio Memory Usage tool to do the same but with a much better user interface! The additional killer feature is the ability to see the references chain that explains why a “leaky” object stays in memory.

My previous series about building your own .NET memory profiler in C# is based on CLR events and does not allow to get the references chain. This post explains how you could write your own memory profiler based on.NET profiler APIs in C++. Refer to this post for an introduction of how to implement ICorProfilerCallback to be loaded by the CLR in a .NET process.

How to detect memory leaks

From a high level view, detecting a memory leak means being able to know which objects stay alive garbage collection after garbage collection:

• implement ICorProfilerCallback::ObjectAllocated to keep track of ALL objects in the heap,
• use ICorProfilerCallback::MovedReferences2 to fixup the addresses when the live objects are moved during compaction garbage collections,
• since** **ICorProfilerCallback::ObjectReferences is called for each surviving object, clean your list of live objects.

The first drawback of this solution is that the CLR has to disable concurrent GC to call these functions with probable impact on performances. However, if you can’t find a leak in production that leads to out of memory crashes, running one instance in this mode is perfectly acceptable. The second drawback is the complexity of keeping track of objects through compacting GCs.

This is why I implemented in .NET 7 a new set of functions in ICorProfilerInfo13 to mimic what you can do in C# with a weak reference:

• CreateHandle : create a weak handle to wrap an object,
• GetObjectIDFromHandle: get the address of the wrapped object or null if the object is no more in the heap,
• DestroyHandle: clean up the weak handle.

Creating such a weak handle for allocated objects get rid of the address fixup complexity. However, you should not create a handle for ALL allocated objects because it will slow down the garbage collections. So the next step is to listen to the AllocationTick CLR event and create a weak handle for each sampled allocation. Even though the statistical distribution of such 100 KB threshold-based sampling is not perfect, leaking objects should appear.

After each garbage collection detected in ICorProfilerCallback2::GarbageCollectionFinished or via specific GC events, you could clean up this list of allocated objects by removing those for which GetObjectIDFromHandle returns null.

Feel free to look at the corresponding implementation in Datadog .NET profiler code.

Rebuild references chain up to a root

Even though it is possible to get the call stack that led to allocating a leaking object thanks to the AllocationTick event, it would be better to know why it stays in memory. So the next step is to rebuild the references chain up to the root.

As explained in a previous post, it is possible, for a given object, to get the list of its fields and build a graph of dependencies from a parent to its children. However, you are interested in the opposite and it would require to get these parent/children references for ALL objects in the heap. And this is not possible with the sampled AllocationTick event…

This is where ICorProfilerCallback::ObjectReferences shines:

1
2
3
4
5
6

HRESULT ObjectReferences(
   ObjectID objectId,
   ClassID  classId,
   ULONG    cObjectRefs,
   ObjectID objectRefIds[] 
);

This method is called during a garbage collection for all objects (i.e. objectId first parameter) still alive and lists its fields referencing objects in the heap (i.e. objectRefIds last parameter).

You could store each object as an ObjectNode:

1
2
3
4
5
6
7
8
9

struct ObjectNode
{
public:
    ObjectNode(ObjectID objectId);

public:
    ObjectID instance;
    std::vector<ObjectNode*> rootRefs;
};

in a vector that represents the heap.

For each fields, look for its corresponding node in the vector and add the node of its parent (given by objectId) to its rootRefs vector of parents. That way, you are building a back reference graph:

The small blue arrows show the parent/children reference given by ObjectReferences and the large purple ones are kept to build a reverse references graph you are interested in.

You know when all live objects in the heap have been listed when ICorProfilerCallback2::GarbageCollectionFinished is called. It is now time to get the build the references chain for all sampled objects still alive (thanks to GetObjectIDFromHandle returning non null address).

It is important to understand that it is a graph and not a tree because cycles exist in .NET.

This is common in situations where objects need to keep a reference to their “parents”. It means that these cycles should be detected when looking for the list of references of a given live object to avoid infinite recursion:

bool DumpNode(ObjectNode* node, std::vector& referenceStack)

The traversing DumpNode method takes a node (i.e. an object of the heap) and a stack where the parents will be added as we dig into the graph.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

bool DumpNode(ObjectNode* node, std::vector<ObjectID>& referenceStack)
{
    // end of recursion: the node is a root
    if (node->rootRefs.size() == 0)
    {
        //  dump the root
        std::cout << std::endl;
        std::cout << std::hex << node->instance << std::dec;
        COR_PRF_GC_ROOT_KIND kind;
        COR_PRF_GC_ROOT_FLAGS flags;
        if (FindRoot(_roots, node->instance, kind, flags))
        {
            std::cout << " | ";
            DumpKind(kind);
            std::cout << " - ";
            DumpFlags(flags);
        }
        else
        {
            std::cout << " | ?";
        }
        std::cout << " = ";

        DumpObjectType(node->instance, _pCorProfilerInfo, _pFrameStore);
        std::cout << std::endl;

        // dump the references from the root
        for (int16_t i = referenceStack.size()-1; i >= 0; i--)
        {
            ObjectID reference = referenceStack[i];
            std::cout << " --> ";
            std::cout << std::hex << reference << std::dec;
            std::cout << " = ";
            DumpObjectType(reference, _pCorProfilerInfo, _pFrameStore);
            std::cout << std::endl;
        }

        return true;
    }

    // detect cycles
    if (Find(referenceStack, node->instance))
    {
        return false;
    }

    // go up into the reference chain
    referenceStack.push_back(node->instance);
    for (auto& parentNode : node->rootRefs)
    {
        if (DumpNode(parentNode, referenceStack))
        {
            return true;
        }
    }
    referenceStack.pop_back();

    return false;
}

If a parent node is already in the stack, a cycle is detected and that path is not used. Once a root is reached, the stack is dumped as shown in the following output:

OnGarbageCollectionFinished: 3859 objects in the heap.

OnRootReferences2: 90/109 roots.
            stack:40
        finalizer:1
           handle:49
            other:0
------------------

21266c00020 | H - 0 = Object[]
 --> 21268c092c8 = NativeRuntimeEventSource
 --> 21268c3fbe8 = EventSource.EventMetadata[]
 --> 21268c18010 = ParameterInfo[]
 --> 21268c17ef0 = RuntimeParameterInfo
 --> 21268c0ed58 = RuntimeMethodInfo
 --> 21268c0e708 = RuntimeType.RuntimeTypeCache
 --> 21268c0e840 = RuntimeType.RuntimeTypeCache.MemberInfoCache<System.Reflection.RuntimeMethodInfo>
 --> 21268c14978 = RuntimeMethodInfo[]
 --> 21268c10d70 = RuntimeMethodInfo
 --> 21268c29720 = Signature
 --> 21268c29770 = RuntimeType[]
=====================================

As shown in the output, it is possible to provide details about the kind of root is keeping the references chain alive thanks to ICorProfilerCallback::RootReferences2:

1
2
3
4
5
6
7

HRESULT RootReferences2(  
   ULONG  cRootRefs,  
   ObjectID rootRefIds[],  
   COR_PRF_GC_ROOT_KIND rootKinds[],  
   COR_PRF_GC_ROOT_FLAGS rootFlags[],  
   UINT_PTR rootIds[]
);

This function is called with three synchronized arrays cRootRefs long that contain for each root:

• the address (**rootRefsIds **objectID),
• the kind (rootKind for stack, finalizer, handle and other)
• and flags (rootFlags for pinned, weak reference interior or ref counted).

These are stored in a vector of ObjectRoot:

Goodies: how to get arrays type name

I did not mention how to get the type name of either an ObjectID or a ClassID because it is explained in a previous post. However, I forgot to explain how to deal with the different kinds of arrays: single dimension (ex: byte[]), multidimensional (ex: byte[,]) or jagged (ex: byte[][]).

When you call ICorProfilerInfo::GetClassInfo on a ClassID corresponding to an array,

1
2
3

ModuleID moduleId;
mdTypeDef typeDefToken;
hr = _pCorProfilerInfo->GetClassIDInfo(classId, &moduleId, &typeDefToken);

it won’t fail but the module id and the metadata token will both be set to 0.

Instead, you have to call ICorProfilerInfo::IsArrayClass to get the rank and the item class ID of the array. This is then done recursively on the item class ID until it fails:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

std::string arrayBuilder;

CorElementType baseElementType;
ClassID itemClassId;
ULONG rank = 0;
if (_pCorProfilerInfo->IsArrayClass(classId, &baseElementType, &itemClassId, &rank) == S_OK)
{
    classId = itemClassId;
    isArray = true;
    AppendArrayRank(arrayBuilder, rank);

    // in case of matrices, it is needed to look for the last "good" item class ID
    // because all others might be array of array of ...
    for (size_t i = 0; i < rank; i++)
    {
        HRESULT hr = _pCorProfilerInfo->IsArrayClass(classId, &baseElementType, &itemClassId, &rank);
        if ((hr == S_FALSE) || FAILED(hr))
        {
            itemClassId = classId;

            break;
        }

        AppendArrayRank(arrayBuilder, rank);
        classId = itemClassId;
    }
}

Notice that the way to concatenate the possible [] / [,] / [][] could is the opposite of how the array type is defined in C#:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

void AppendArrayRank(std::string& arrayBuilder, ULONG rank)
{
    if (rank == 1)
    {
        arrayBuilder = "[]" + arrayBuilder;
    }
    else
    {
        std::stringstream builder;
        builder << "[";
        for (size_t i = 0; i < rank - 1; i++)
        {
            builder << ",";
        }
        builder << "]";

        arrayBuilder = builder.str() + arrayBuilder;
    }
}

For example, a byte[][,] is defined as an rank 2 array of array of byte.

References

• Automate the search of memory leaks with LeakShell
• Building your own .NET memory profiler in C#
• Introduction to .NET Profiling with ICorProfilerCallback
• Pull request in .NET 7 for ICorProfilerInfo13 to create weak handles
• Datadog .NET Live Heap Profiler implementation

As shown in the previous post, the processing of ProcessInfo diagnostic commands is easy because you send a request and read the different fields from the response. This is different if you want to receive events from the CLR via EventPipe. In C#, the TraceEvent nuget package wraps everything under a nice event handler based model as shown in many of my previous posts.

Behind the scene, a StartSession command is sent (more details about the parameters later) and the response contains the numeric ID of the session. Then, the events will be read from the IPC channel as a binary stream of data with the “nettrace“ file format. The collection ends when the StopTracing command is sent.

The source code is available from my github repository.

Hidding the transport layer: IIpcEndoint

Unlike the previous post, to send the command and read the response back from the CLR , I’m wrapping the transport layer with the IIpcEndpoint interface:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

class IIpcEndpoint
{
public:
    virtual bool Write(LPCVOID buffer, DWORD bufferSize, DWORD* writtenBytes) = 0;
    virtual bool Read(LPVOID buffer, DWORD bufferSize, DWORD* readBytes) = 0;
    virtual bool ReadByte(uint8_t& byte) = 0;
    virtual bool ReadWord(uint16_t& word) = 0;
    virtual bool ReadDWord(uint32_t& dword) = 0;
    virtual bool ReadLong(uint64_t& ulong) = 0;

    virtual bool Close() = 0;
    virtual ~IIpcEndpoint() = default;
};

It abstracts the write and read accesses to the underlying transport layer. In addition, the base class accepts a “recorder” that allows me to store what is received from the CLR into any kind of storage (today only a file-based recorder that helped a lot to reproduce specific situations without the need to have a running process to connect to):

The PidEndpoint class accepts the process id of the running .NET application to monitor its CLR events and an optional recorder implementing the IIpcRecorder interface. The Create static factory implementation creates the expected named pipe on Windows (or the domain socket on Linux) and stores the handle into its _handle field:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

PidEndpoint* PidEndpoint::CreateForWindows(int pid, IIpcRecorder* pRecorder)
{
    PidEndpoint* pEndpoint = new PidEndpoint(pRecorder);

    // build the pipe name as described in the protocol
    wchar_t pszPipeName[256];
    int nCharactersWritten = -1;
    nCharactersWritten = wsprintf(
        pszPipeName,
        L"\\\\.\\pipe\\dotnet-diagnostic-%d",
        pid
    );

    // check that CLR has created the diagnostics named pipe
    if (!::WaitNamedPipe(pszPipeName, 200))
    {
        auto error = ::GetLastError();
        std::cout << "Diagnostics named pipe is not available for process #" << pid << " (" << error << ")" << "\n";
        return nullptr;
    }

    // connect to the named pipe
    HANDLE hPipe;
    hPipe = ::CreateFile(
        pszPipeName,    // pipe name
        GENERIC_READ |  // read and write access
        GENERIC_WRITE,
        0,              // no sharing
        NULL,           // default security attributes
        OPEN_EXISTING,  // opens existing pipe
        0,              // default attributes
        NULL);          // no template file

    if (hPipe == INVALID_HANDLE_VALUE)
    {
        std::cout << "Impossible to connect to " << pszPipeName << "\n";
        return nullptr;
    }

    pEndpoint->_handle = hPipe;
    return pEndpoint;
}

The next step is to open a tracing session by sending the StartSession command.

The Trace diagnostic commands

Following the same object model provided by the Microsoft.Diagnostics.NETCore.Client nuget, my DiagnosticsClient class hides the transport layer. It also exposes high level functions such as OpenEventPipeSession to initiate a trace event session with the CLR:

1

EventPipeSession* OpenEventPipeSession(uint64_t keywords, EventVerbosityLevel verbosity);

If you remember from TraceEvent, you need a few parameters to create a session:

• size of circular buffers used by the CLR to cache events (same as Perfview, use 16 MB as default)
• netttrace format (i.e. value of 1)
• if rundown events are needed
• a list of providers (“Microsoft-Windows-DotNETRuntime” for the CLR in my case)
• keywords
• verbosity level
• possible arguments (none here)

Here is the corresponding C++ description of the command type:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

const uint8_t DotnetProviderMagicLength = 32;
struct MagicProvider
{
    wchar_t Magic[DotnetProviderMagicLength];
};

// 32 wchar_t (including \0)
const MagicProvider DotnetProviderMagic = { L"Microsoft-Windows-DotNETRuntime" };

const uint32_t CircularBufferMBSize = 16;
const uint32_t NetTraceFormat = 1;

#pragma pack(1)
struct StartSessionMessage : public IpcHeader
{
    uint32_t CircularBufferMB;  // 16 MB
    uint32_t Format;            // 1 for NetTrace format
    uint8_t RequestRundown;     // 0 because don't want rundown

    // array of provider configuration
    uint32_t ProviderCount;     // 1 only: Microsoft-Windows-DotNETRuntime
    uint64_t Keywords;          // from EventKeyword
    uint32_t Verbosity;         // from EventPipeEventLevel
    uint32_t ProviderStringLen; // number of UTF16 characters = 32 (including last \0)
    union                       // dotnet provider name
    {
        MagicProvider _magic;
        uint8_t Provider[2 * DotnetProviderMagicLength];
    };
    uint32_t Arguments;         // 0 for empty string (no argument)
};

The code to fill up the command is straightforward:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

StartSessionMessage* CreateStartSessionMessage(uint64_t keywords, EventVerbosityLevel verbosity)
{
    auto message = new StartSessionMessage();
    ::ZeroMemory(message, sizeof(message));
    memcpy(message->Magic, &DotnetIpcMagic_V1, sizeof(message->Magic));
    message->Size = sizeof(StartSessionMessage);  
    message->CommandSet = (uint8_t)DiagnosticServerCommandSet::EventPipe;
    message->CommandId = (uint8_t)EventPipeCommandId::CollectTracing2;
    message->Reserved = 0;
    message->CircularBufferMB = CircularBufferMBSize;
    message->Format = NetTraceFormat;
    message->RequestRundown = 0;
    message->ProviderCount = 1;
    message->Keywords = (uint64_t)keywords;
    message->Verbosity = (uint32_t)verbosity;
    message->ProviderStringLen = DotnetProviderMagicLength;
    memcpy(message->Provider, &DotnetProviderMagic, sizeof(message->Provider));
    message->Arguments = 0;

    return message;
}

The provider list is defined with the ProviderCount field and string containing the list (only one here) follows the Verbosity field. To start the session, it is needed to send the StartSession message and read the session id from the response:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

bool EventPipeStartRequest::Process(IIpcEndpoint* pEndpoint, uint64_t keywords, EventVerbosityLevel verbosity)
{
    // send an StartSessionMessage and parse the response
    StartSessionMessage* pMessage = CreateStartSessionMessage(keywords, verbosity);

    DWORD writtenBytes = 0;
    if (!pEndpoint->Write(pMessage, pMessage->Size, &writtenBytes))
    {
        return false;
    }

    // analyze the response
    IpcHeader response = {};
    DWORD bytesReadCount = 0;
    if (!pEndpoint->Read(&response, sizeof(response), &bytesReadCount))
    {
        return false;
    }

    if (response.CommandId != (uint8_t)DiagnosticServerResponseId::OK)
    {
        return false;
    }

    // get the session ID from the payload
    uint16_t payloadSize = response.Size - sizeof(response);
    if (payloadSize < sizeof(uint64_t))
    {
        return false;
    }

    if (!pEndpoint->ReadLong(SessionId))
    {
        return false;
    }

    return true;
}

Once the StartSession command has been sent, the events corresponding to the given provider/keywords/verbosity (here the CLR runtime/gc+exception+contention/verbose)

1
2
3
4
5
6

auto pSession = pClient->OpenEventPipeSession(
        EventKeyword::gc |
          EventKeyword::exception |
          EventKeyword::contention,
        EventVerbosityLevel::Verbose  // required for AllocationTick
        );

will be read from the event pipe. Since this action will be synchronous, it is recommended to dedicate a thread to read and process the events:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

    if (pSession != nullptr)
    {
        DWORD tid = 0;
        auto hThread = ::CreateThread(nullptr, 0, ListenToEvents, pSession, 0, &tid);

        std::cout << "Press ENTER to stop listening to events...\n\n";
        std::string line;
        std::getline(std::cin, line);

        std::cout << "Stopping session\n\n";
        pSession->Stop();
        std::cout << "Session stopped\n\n";

        // test if it works
        ::Sleep(1000);

        ::CloseHandle(hThread);
    }

The ListenToEvents callback executed by the new thread is “simply” listening to the event pipe of the session:

1
2
3
4
5
6
7
8

DWORD WINAPI ListenToEvents(void* pParam)
{
    EventPipeSession* pSession = static_cast<EventPipeSession*>(pParam);

    pSession->Listen();

    return 0;
}

Before describing how to read the events, it is important to understand how to stop the flow. First, inside the EventPipeSession, the internal loop that reads events needs to exit thanks to the _stopRequested boolean:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

bool EventPipeSession::Stop()
{
    _stopRequested = true;

    if (_pid == -1)
        return true;

    // it is neeeded to use a different ipc connection to stop the Session
    DiagnosticsClient* pStopClient = DiagnosticsClient::Create(_pid, nullptr);
    pStopClient->StopEventPipeSession(SessionId);
    delete pStopClient;

    return true; 
}

In addition, a message with StopTracing command id from the EventPipe command set needs to be sent to tell the CLR to stop sending the events. This message must be sent through a different IPC channel (hence the pStopClient variable used in the previous code. The StopEventPipeSession helper function uses the EventPipeStopRequest wrapper:

1
2
3
4
5

bool DiagnosticsClient::StopEventPipeSession(uint64_t sessionId)
{
    EventPipeStopRequest request;
    return request.Process(_pEndpoint, sessionId);
}

The StopSession command accepts the session ID as single parameter:

1
2
3
4
5

#pragma pack(1)
struct StopSessionMessage : public IpcHeader
{
    uint64_t SessionId;
};

The processing of the stop request is to create such a message and send it through the IPC channel:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

StopSessionMessage* CreateStopMessage(uint64_t sessionId)
{
    StopSessionMessage* message = new StopSessionMessage();
    ::ZeroMemory(message, sizeof(message));
    memcpy(message->Magic, &DotnetIpcMagic_V1, sizeof(message->Magic));
    message->Size = sizeof(StopSessionMessage);
    message->CommandSet = (uint8_t)DiagnosticServerCommandSet::EventPipe;
    message->CommandId = (uint8_t)EventPipeCommandId::StopTracing;
    message->Reserved = 0;
    message->SessionId = sessionId;

    return message;
}

bool EventPipeStopRequest::Process(IIpcEndpoint* pEndpoint, uint64_t sessionId)
{
    StopSessionMessage* pMessage = CreateStopMessage(sessionId);
    DWORD writtenBytes;
    if (!pEndpoint->Write(pMessage, pMessage->Size, &writtenBytes))
    {
        Error = ::GetLastError();
        std::cout << "Error while sending EventPipe Stop message to the CLR: 0x" << std::hex << Error << std::dec << "\n";
        delete pMessage;
        return false;
    }
    
    delete pMessage;

    ... // handle the response 

    return true;
}

When the stop command is received by the CLR, the remaining “data” (more on this in the next episode) is sent through the first IPC channel before being closed. This is how the code knows that the session can stop listening to the EventPipe.

The next episode will start to parse the nettrace stream of events.

Resources

• Episode 1 — Digging into the CLR Diagnostics IPC Protocol in C#
• Episode 2 — .NET Diagnostic IPC protocol: the C++ way
• Source code for the C++ implementation of CLR events listener
• Diagnostics IPC protocol documentation
• Microsoft.Diagnostics.NETCore.Client source code

Introduction

With the new 2.10 release of the Datadog .NET Tracer and Continuous Profiler available, it is time to update some investigation workflows I already introduced. New features have been added to help you diagnose performance issues in your .NET applications:

• Linux support!
• Code Hotspots: allow you to automatically navigate from lengthy spans and requests to profiles
• CPU profiling: pinpoint high CPU consuming methods
• Exceptions profiling: identify exceptions distributions
• Profile sequence: easily profile an application startup

The goal of this post is to show you how all these features make your investigations easier. I would recommend reading the previous post; especially for the environment setup that I won’t repeat here.

It’s Linux showtime!

The .NET Continuous Profiler is now available for Linux. The only limitation is the presence of glibc 2.18+ in the distribution; for example, CentOS 7 is not supported. Beyond that, we provide features parity between Linux and Windows.

In terms of installation, download the .NET Tracer package that supports your operating system and architecture. Go to the documentation for the additional configuration steps.

From spans to profiles

When analysing lengthy requests, you usually start from looking at the corresponding spans in the APM Traces part of the UI. It is now possible to view the corresponding profiles by clicking the “View Profile” button in the “Code Hotspots” tab:

Before digging into the profiling information, you are already able to see that more than half of the time is spent in Buffer._Memmove that is called by Buggybits ProductsController.Index method:

From the profile view, it is also possible to come back to the traces:

Let’s see now what new features are available at the profiling side.

CPU profiling

The most demanded feature was the ability to analyse CPU consumption (a.k.a. CPU profiling). The idea is to be able to identify code that really consumes CPU usage and optimize it. This is particularly important in the context of cloud-based computing where what you pay is related to the consumed CPU.

In term of implementation, unlike Wall Time profiling, we look at the time spent by a thread on a CPU core and not the elapsed time since the last time we checked (every ~10ms). We also collect the call stack of a thread only if it is currently running on a core. Why? Because we want to only record call stacks corresponding to code paths that are consuming CPU. For example, ThreadPool threads are usually waiting (not interesting call stack) for a work item to process (interesting call stack).

In the last blog post, the code responsible for lengthy requests is doing too many string concatenations (look for += in the following code):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

public IActionResult Index()
{
    var sw = new Stopwatch();
    sw.Start();
    var products = dataLayer.GetAllProducts();
    var productsTable = "<table><tr><th>Product Name</th><th>Description</th><th>Price</th></tr>";
    foreach (var product in products)
    {
        productsTable += $"<tr><td>{product.ProductName}</td><td>{product.Description}</td><td>{product.Price}</td></tr>";
    }

    productsTable += "</table>";
    sw.Stop();

    ViewData["ElapsedTimeInMs"] = sw.ElapsedMilliseconds;
    ViewData["ProductsTable"] = productsTable;
    return View();
}

The Wall Time view was very explicit about ProductsController.Index() calling String.Concat() culprit:

A simple solution is to use a StringBuilder to optimize the concatenations:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

public IActionResult Builder()
{
    var sw = new Stopwatch();
    sw.Start();
    var products = dataLayer.GetAllProducts();
    var productsTable = new StringBuilder(1000 * 80);
    productsTable.Append("<table><tr><th>Product Name</th><th>Description</th><th>Price</th></tr>");
    foreach (var product in products)
    {
        productsTable.Append($"<tr><td>{product.ProductName}</td><td>{product.Description}</td><td>{product.Price}</td></tr>");
    }

    productsTable.Append("</table>");
    sw.Stop();

    ViewData["ElapsedTimeInMs"] = sw.ElapsedMilliseconds;
    ViewData["ProductsTable"] = productsTable;
    return View("Index");
}

The Wall Time view of the profile with the fix does not provide anything useful…

If you want to go deeper, you need to look at the CPU consumption:

The ProductController.Builder method is handling the request (like Index() in the String.Concat case) and calls DataLayer.GetAllProducts() where most of the CPU-related work is done.

Would it be interesting to continue optimizing the code? Notice that GetAllProducts() is “only” consuming 94ms and the other Number.* and String.Concat method around 350ms. So a gain might be neglectable compared to the total ~3 seconds CPU usage

Remember that you should not optimize for the sake of “optimizing”: you should have metrics that tell you when to start (too lengthy request processing) and when to stop.

Exceptions profiling

In the .NET world, exceptions are at the center of errors handling. It is now possible to get a sampled view of the exceptions that happened during an application lifetime; by type:

and by message:

At the implementation level, the new exceptions profiler is notified by the CLR when an exception is thrown. Since in special cases (such as network issue or invalid parsed data for example), an application could trigger thousands of exceptions in a very short period, it is needed to sample them. Otherwise, the impact on performances would be severe; especially if the call stack needs to be rebuilt for each exception.

First, at least one exception per type is kept, ensuring that weird specific exceptions are not lost in the flow. Second, exceptions are sampled over time based on a fixed number of exceptions per profile and the rate of appearance. For knowing the exact number of exceptions, feel free to leverage the Runtime Metrics package as explained in the previous blog post.

Profiling the application bootstrap

In some situations, you are interested in analysing an application bootstrap. In Datadog APM Profile Search UI, it means finding the first profile of the given service execution. However, even with the date and time column, it is not obvious to find the right one:

To help you find the initial profile of a service execution, a new “profile_seq” tag has been added to the HTTP request used to upload the profiles. It contains the count of generated profiles for a given execution of a service, starting from 0.

So now, in the Options of the profile list, add a “profile_seq” column:

The first profile is then easily spottable with a 0 value:

In the future, a more visual hint might be added to identify it without the need to add the column.

Major implementation refactoring

Finally, our implementation has benefited from a large code refactoring. As the previous post explained, the generation of .pprof files and their upload was done in C#. This has been replaced by using a rust library shared amongst different profiler libraries (native, Ruby, .NET).

It means that you should not anymore see these frames in the application call stacks:

It does not mean that one third of the processing has been removed! Just that no more C# code is running with performance gain. First, the managed implementation was allocating objects managed by the garbage collector; adding pressure that might trigger more collections. Second, with the native rust implementation, there is no need to duplicate data between the collecting native part of the continuous profiler and the managed code used to serialize it.

In addition, several optimizations have been done in the symbol’s resolution (i.e., type and method names) part of the code that also reduce memory consumption and CPU usage.

Happy profiling!

References

• Datadog Tracer & Continuous Profiler .msi Installer and Linux tar.gz
• Datadog Continuous Profiler documentation
• Datadog Tracer documentation
• Datadog Runtime metrics documentation
• Tess Ferrandez repository for BuggyBits labs

Introduction

The beta of Datadog .NET Continuous Profiler is available!

This is a great opportunity to show how to use the different tools provided by Datadog to troubleshoot .NET applications facing performance issues. Tess Ferrandez updated her famous BuggyBits application to .NET Core. Among the different available scenarios, let’s see how to investigate the Lab 4 — High CPU Hang with Datadog. It will be completely different from Tess way: no need to analyze memory dump anymore.

Setup the environment

First, you need to download and run the .msi from our Tracer repository: it will install both the Tracer and the Profiler. The former allows you, among other things, to see how long it takes to process ASP.NET Core requests. The latter is in Beta today and provides wall time duration of your threads (more on this later). Look at the corresponding documentations for the details of enabling tracing and profiling once installed.

Next, ensure that .NET Runtime Metrics are installed for your organization:

Add DD_RUNTIME_METRICS_ENABLED=true environment variable for the application/service you want to monitor. Once enabled for your application, this package allows you to see the evolution of important metrics including some that you won’t find anywhere else such as GC pause time, thread contention time or count of exceptions per type.

Ensure that DogstatsD is setup for the Agent

Looking at the symptoms

In my example, the buggybits application is running under the datadog.demos.buggybits service name. This is how I can filter the related traces in the APM/Traces part of the Datadog portal:

In this screenshot, the Products/Index requests duration is around 6 seconds; which is way too long!

When clicking such a request, the details panel provides the exact URL in the Tags tab:

The Metrics tab shows CPU usage and other few metrics around the trace time:

So, in addition to having slow requests, the CPU usage seems to increase.

It is now time to go to the .NET runtime metrics dashboard and look at what is going on in more details. The first graph that shows up is the number of gen2 collections:

It means that during the 10 minutes test where very few requests are processed, almost 800 gen2 GC are happening every 10s (all runtime metrics are computed every 10 seconds).

The load test corresponding to these requests lasted 10 minutes between 4pm and 6+pm. Each time the requests were processed:

• the CPU usage increased
• the number of gen2 collections increased
• the duration of pauses due to garbage collections increased
• the threads contention time increased

In addition to being slow, it seems that the code that processes products/index HTTP requests has also an impact on the CPU (i.e. on the overall application and machine performances).

It would be great if we could see the callstacks corresponding to this processing. This is exactly what the .NET Wall time Continuous Profiler is all about: looking at the duration of methods through a flamegraph representation.

Here comes the profiler

Today, there is no direct way to jump from a trace to the profile containing the callstacks while the corresponding request was processed. We are currently working on this new feature called Code Hotspots.

However, it is easy to use the service name to filter the profiles and select the period of time from APM/Profile Search:

When you click a one-minute profile (the callstacks are gathered and sent every minute), a panel appears with the Performance tab selected. It shows a framegraph on the left and a list on the right.

Getting used to flamegraph

When you look at the wall time flamegraph, you see everything that happened during a single minute:

The previous screenshot highlights the groups of callstacks corresponding to the different threads of execution (from left to right):

• the Main() entry point of the application
• the code in the CLR responsible for sending counters
• the code in the Profiler in charge of generating and sending (the very thin spike) the profile every minute
• the Tracer code
• the code that listens to the CLR events to generate the runtime metrics
• …and the application code that processes the requests!

In the flamegraph, the width of each frame on a row represents the relative time during which the frame was found on a callstack. For example, in our tests, we have 4 threads simply calling Thread.Sleep; one for 10 seconds, one for 20 seconds, one for 30 seconds and a last one for 40 seconds. This is the expected result in a flamegraph (i.e. the widths are consistent with the 1/2/3/4 ratio):

This also applies to CPU-bound threads. For example, if 3 threads are computing the sum of numbers in a tight loop, this is the expected result (i.e. all OnCPUxxx have the same width)

These explanations should stop the fear that started to crawl inside your head about the “visible cost” of the Datadog Tracer and Profiler based on the previous screenshot. The large width of the Datadog threads frames is all about wall time, not CPU time: we are mostly sleeping or waiting but we don’t stop :^)

Investigate the performance issue

The next step is to focus on the stack frames corresponding to the request processing to better understand what is going on.

Basically, you would like to either remove a branch or keep only a branch. You simply have to move the mouse over a frame (i.e. ThreadPoolWorkQueue in the previous screenshot) and click the three dots that just appeared. Next, select Show From to keep only that branch in the flamegraph:

Now, scroll-down into the flamegraph and the flow of execution corresponding to processing the Products/Index request becomes more visible:

It seems that the Index() method of the ProductsController is spending most of its time calling String.Concat().

Let’s have a look at the source code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

// GET: Products
public IActionResult Index()
{
    var products = dataLayer.GetAllProducts();
    var productsTable = "<table><tr><th>Product Name</th><th>Description</th><th>Price</th></tr>";
    foreach (var product in products)
    {
        productsTable += $"<tr><td>{product.ProductName}</td><td>{product.Description}</td><td>{product.Price}</td></tr>";
    }
    productsTable += "</table>";

    ViewData["ProductsTable"] = productsTable;
    return View();
}

But still no sign of String.Concat()… Well, this is because the C# compiler is hiding it from you with the += syntaxic sugar. Let’s have a look at the decompiled code as shown by IlSpy (without the string.Concat transformation):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

public Microsoft.AspNetCore.Mvc.IActionResult Index()
{
    var products = dataLayer.GetAllProducts();
    string productsTable = "<table><tr><th>Product Name</th><th>Description</th><th>Price</th></tr>";
    var enumerator = products.GetEnumerator();
    try
    {
        while (enumerator.MoveNext())
        {
            BuggyBits.Models.Product product = enumerator.Current;
            string[] array = new string[8];
            array[0] = productsTable;
            array[1] = "<tr><td>";
            array[2] = product.ProductName;
            array[3] = "</td><td>";
            array[4] = product.Description;
            array[5] = "</td><td>";
            array[6] = product.Price;
            array[7] = "</td></tr>";
            productsTable = string.Concat(array);
        }
    }
    finally
    {
        ((System.IDisposable)enumerator).Dispose();
    }
    productsTable = string.Concat(productsTable, "</table>");
    base.ViewData["ProductsTable"] = productsTable;
    return View();
}