In my previous posts, I explained how to use DIA and DbgHelp to map a method to its line in source code. I forgot to mention that it was correct for .NET Core but not for the “old” .NET Framework Windows PDB format. Instead of encoding the method token in the name, the symbol file contains the name of the methods. So, how to do the mapping for .NET Framework assemblies? You will find the answer (plus some tricks) in this article.

When I started to work on the support of the old Windows PDB format, I looked at what existed to parse the raw format and… I decided to try DbgHelp instead. With this first implementation, I realized that some token where missing and source code information was not retrieved for most of the methods.

So, I looked for another API to use and I found ISymUnmanagedReader. The usage philosophy is totally different from DIA or DbgHelp.

A little bit of magic

This interface is implemented in diasymreader.dll that comes with every .NET Framework installation. But you need to do COM magic to get it. After having called CoInitialize to setup COM, you ask for an instance of ISymUnmanagedBinder from CLSID_CorSymBinder_SxS:

1
2
3
4
5
6
7
8

CComPtr<ISymUnmanagedBinder> pBinder;
hr = CoCreateInstance(
    CLSID_CorSymBinder_SxS,
    NULL,
    CLSCTX_INPROC_SERVER,
    IID_ISymUnmanagedBinder,
    (void**)&pBinder
);

From the binder, you can get the ISymUnmanagedReader interface corresponding to the assembly you are interested in with GetReaderForFile. However, there are two tiny details to consider.

First, one parameter expects the path to the assembly, not to the .pdb file. That symbol file has to be stored in the same folder but note that the documentation states that you could have more flexible search with ISymUnmanagedBinder2::GetReaderForFile2 but I did not test it.

The second detail is the first parameter: an instance of IMetaDataImport for the same assembly. The steps to get it are… complicated.

Hosting the CLR

The idea is to host the .NET Framework and get the corresponding ICLRMetaHost interface:

1
2

CComPtr<ICLRMetaHost> pMetaHost;
HRESULT hr = CLRCreateInstance(CLSID_CLRMetaHost, IID_ICLRMetaHost, (void**)&pMetaHost);

Calling the CLRCreateInstance API allows you to get an instance of ICLRMetaHost from which you could enumerate installed version of .NET Framework. In my case, I know which version I want:

1
2
3

// Get the installed .NET Framework runtime (v4.0+)
CComPtr<ICLRRuntimeInfo> pRuntimeInfo;
hr = pMetaHost->GetRuntime(L"v4.0.30319", IID_ICLRRuntimeInfo, (void**)&pRuntimeInfo);

The ICLRRuntimeInfo interface allows you to get access to runtime services via GetInterface:

1
2

CComPtr<IMetaDataDispenser> pDispenser;
hr = pRuntimeInfo->GetInterface(CLSID_CorMetaDataDispenser, IID_IMetaDataDispenser, (void**)&pDispenser);

The service I’m interested in is the IMetadataDispenser interface that allows you to “open a scope” on the assembly you are interested in:

1
2
3
4
5
6

hr = pDispenser->OpenScope(
    wModulePath.c_str(),
    ofRead,
    IID_IMetaDataImport,
    (IUnknown**)&_pMetaDataImport
);

Note that the first parameter is the path to the assembly not to the .pdb file. The scope is abstracted by an IMetadataImport interface I have already described and that is needed to call GetReaderForFile: and get the ISymUnmanagedReader:

hr = pBinder->GetReaderForFile(_pMetaDataImport, wModulePath.c_str(), nullptr, &_pReader);

The road to get symbol details for a method

The ISymUnmanagedReader interface implements GetMethod to get details about a given method token via an ISymUnmanagedMethod interface. So, the next question is how to get these tokens. If you remember the previous article, these tokens are from the 06 MethodDef table in the assembly metadata; starting from 06000001 to the last one.

This means that you could write a simple loop starting from 1 up to a hardcoded maximum value, call TokenFromRid(index, mdtMethodDef) to get the corresponding token. However, since you are a professional developer, you would search for the exact number of tokens from IMetadataTables retrieved from IMetadataImport:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

ULONG cRows = 0;

// Get IMetaDataTables interface to query the MethodDef table
CComPtr<IMetaDataTables> pTables;
hr = _pMetaDataImport->QueryInterface(IID_IMetaDataTables, (void**)&pTables);
if (FAILED(hr) || pTables == nullptr)
{
    cRows = LAST_METHODDEF_TOKEN;
}
else
{
    // Get the number of rows in the MethodDef table (table index 0x06 = Method)
    hr = pTables->GetTableInfo(
        0x06,           // MethodDef table
        NULL,           // cbRow (not needed)
        &cRows,         // pcRows (number of methods)
        NULL,           // pcCols (not needed)
        NULL,           // piKey (not needed)
        NULL            // ppName (not needed)
    );

    if (FAILED(hr))
    {
        cRows = LAST_METHODDEF_TOKEN;
    }
}

Now that you have the number of rows (i.e. number of methods defined in the metadata), it is easy and safe to get method information from symbols:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

for (uint32_t i = 1; i <= cRows; i++)
{
    mdMethodDef token = TokenFromRid(i, mdtMethodDef);

    CComPtr<ISymUnmanagedMethod> pMethod;
    hr = _pReader->GetMethod(token, &pMethod);
    if (SUCCEEDED(hr) && pMethod != nullptr)
    {
        MethodInfo info;
        if (GetMethodInfoFromSymbol(pMethod, info))
        {
            _methods.push_back(info);
        }
    }
}

Note that GetMethod might fail (returning E_FAIL) for P/Invoked functions, abstract methods, or methods decorated with DebuggerHidden attribute.

Give me line and source code!

For the other methods with symbol information, you can get its token via the GetToken method. The ISymUnmanagedMethod interface allows low level access to line/column mapping that is beyond the scope of this article. At a high level, positions in source file are named sequence points. Call GetSequencePointCount to get… the number of sequence points for a given method.

The next step is to call GetSequencePoints with the number of points you want and the corresponding arrays of offsets, lines, columns, end lines, end columns and ISymUnmanagedDocument. In my case, I’m only interested in where the method starts so the first sequence point is good enough:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

// Get sequence points (source line information)
ULONG32 cPoints = 0;
hr = pMethod->GetSequencePointCount(&cPoints);
if (SUCCEEDED(hr) && (cPoints > 0))
{
    cPoints = 1; // We only need the first sequence point for start line
    std::vector<ULONG32> offsets(cPoints);
    std::vector<ULONG32> lines(cPoints);
    std::vector<ULONG32> columns(cPoints);
    std::vector<ULONG32> endLines(cPoints);
    std::vector<ULONG32> endColumns(cPoints);
    std::vector<ISymUnmanagedDocument*> documents(cPoints);

    ULONG32 actualCount = 0;
    hr = pMethod->GetSequencePoints(
        cPoints,
        &actualCount,
        &offsets[0],
        &documents[0],
        &lines[0],
        &columns[0],
        &endLines[0],
        &endColumns[0]
    );

    if (SUCCEEDED(hr) && (actualCount > 0))
    {

The source file is described by ISymUnmanagedDocument that provides its name when GetURL is called:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

// Get the first sequence point's document and line
        ISymUnmanagedDocument* pDoc = documents[0];
        if (pDoc != nullptr)
        {
            // Get document URL (file path)
            ULONG32 urlLen = 0;
            hr = pDoc->GetURL(0, &urlLen, NULL);
            if (SUCCEEDED(hr) && (urlLen > 0))
            {
                std::vector<WCHAR> url(urlLen);
                hr = pDoc->GetURL(urlLen, &urlLen, &url[0]);
                if (SUCCEEDED(hr))
                {
                    // Convert wide string to narrow string
                    int len = WideCharToMultiByte(CP_UTF8, 0, &url[0], urlLen, NULL, 0, NULL, NULL);
                    std::string narrowUrl(len, '\0');
                    WideCharToMultiByte(CP_UTF8, 0, &url[0], urlLen, &narrowUrl[0], len, NULL, NULL);
                    info.sourceFile = narrowUrl;
                }
            }

            // NOTE: 0xFEEFEE is a special value indicating hidden lines
            info.lineNumber = lines[0];
            pDoc->Release();
        }
    }
}

The final interesting trick is that the line number might have the special 0xFEEFEE value. It means that the line is hidden. I have seen it for methods generated by the C# compiler such as MoveNext for async state machines or anonymous methods:

The source code is available from my Github repository.

Happy coding!

References

• Archived Microsoft-pdb repository
• DIA implementation article
• DbgHelp implementation article

In the previous post, I presented the new commands that were added to dotnet-dump and how to use them. It is now time to show how to implement such a command.

But before jumping into the code, you should first ensure that you have a valid use case that the Diagnostics team is not currently working on. I recommend to create an issue in the Diagnostics repository to explain what is missing for which scenario and propose to implement the corresponding command.

What is a dotnet-dump command?

Here is the directory structure related to the dotnet-dump tool in the Diagnostics repository:

The built binaries are generated under artifacts\bin\dotnet-dump<Release or Debug>\netcoreapp2.1 folder if you need to test them outside of Visual Studio.

The eng folder contains the versions.props file that lists the versions for nuget dependencies. In my case, I had to reference the ParallelStacks.Runtime nuget so I added the following line:2.0.1

And in the dotnet-dump.csproj, this nuget is referenced with the same variable: `

Missed the first part of the story? Read it here:

How to extend dotnet-dump (1/2) — What are the new commands? *This first post describes the new commands, when to use them, and the git setup I used to implement them.*medium.com

Want to work with Christophe or other teams? Check out our open positions:

Careers at Criteo | Criteo jobs *Find opportunities everywhere. ​Choose your next challenge. Find the job opportunities at Criteo in Product, research &…*careers.criteo.com

Introduction

To ease our troubleshooting sessions at Criteo, new high level commands for WinDBG have been written and grouped in the gsose extension. As we moved to Linux, Kevin implemented the plumbing to be able to load gsose into LLDB. In both cases, our extension commands are based on ClrMD to dig into a memory dump.

As Microsoft pushed for dotnet-dump as the new cross-platform way to deal with memory dump, it was obvious that we would have to be able to use our extension commands in dotnet-dump. Unfortunately dotnet-dump does not support any extension mechanism. In May this year, Kevin updated a minimum of code to load a list of extension assemblies at the startup of dotnet-dump. I followed another direction by adding a “load” command to dynamically add extensions commands.

However, the Diagnostics team was focusing on supporting .NET Core 5 release and adding extensibility was not planned before next year. Due to our own time constraints at Criteo, gsose extension commands were really needed so we agreed with the Diagnostics team to implement these commands directly into dotnet-dump as pull requests.

This first post describes the new commands, when to use them, and the git setup I used to implement them.

Commands details and challenges

Here is the list of extension commands as shown by the help command:

As of today with the last 3.1.141901 official version, only timerinfo is available but feel free to clone the repository and rebuild it to get all commands.

pstacks: almost Parallel Stacks a la Visual Studio

When you start an investigation, you are usually interested in getting a high level view of what the running threads are doing and this is what pstacks provides. When hundreds of threads are running, commands such as clrthreads are useless. Instead, pstacks merges the common parts of each call stack and present them in a tree:

Don’t be scared by the layout; here is a description of each section:

If you look at the threads 9c6c and 4020 (after the last ~~~~), it means that 2 threads (only the first 4 threads id are shown except if you pass — all as a parameter) share the same callstack:

~~~~ 9c6c,4020
2 System.Threading.Monitor.Wait(Object, Int32, Boolean)
4 System.Threading.Tasks.Task+c.cctor>b__278_1(Object)
  ...
4 System.Threading.Tasks.Task.ExecuteEntryUnsafe()
4 System.Threading.Tasks.Task.ExecuteWorkItem()
7 System.Threading.ThreadPoolWorkQueue.Dispatch()
7 System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

Compared to the “list” representation, the “tree” representation is useful when you have a lot of threads to deal with and see the different branches in the groups of stack frames.

ti: see all your timers

If you don’t find anything interesting in the running threads (i.e. not hundreds of threads with the same code stack blocked on a Wait call for example), you should look at what will run as timer callbacks with the ti command.

Here is example of what you will get:

> ti
(S) 0x00007F7D84529CD0 @    1000 ms every     1000 ms |  0000000000000000 () -> Kafka.Batching.AccumulatorByTopic<Kafka.Routing.FetchMessage>.Tick
    ...
(L) 0x00007F7D844FA7A0 @    1000 ms every     1000 ms |  0000000000000000 () -> Kafka.Batching.AccumulatorByTopic<Kafka.Routing.OffsetMessage>.Tick
    ...
(L) 0x00007F7D842042F8 @     999 ms every     1000 ms |  0000000000000000 () -> Criteo.DevKit.TimeStamp+<>c.cctor>b__35_0
    ...
(L) 0x00007F7D846314C8 @     999 ms every     1000 ms |  00007F7D8462ED70 (Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.Heartbeat) ->
    ...
 
   190 timers
-----------------------------------------------
   1 | (S) @     999 ms every     1000 ms | 0000000000000000 () -> Kafka.Batching.AccumulatorByTopic<Kafka.Routing.FetchMessage>.Tick
   1 | (L) @     996 ms every     1000 ms | 0000000000000000 () -> Kafka.Batching.AccumulatorByTopic<Kafka.Routing.OffsetMessage>.Tick
   1 | (L) @     993 ms every     1000 ms | 0000000000000000 () -> Kafka.Batching.AccumulatorByTopic<Kafka.Routing.FetchMessage>.Tick
   1 | (L) @     999 ms every     1000 ms | 0000000000000000 () -> Criteo.DevKit.TimeStamp+<>c.cctor>b__35_0
   ...
   9 | (L) @    1000 ms every     1000 ms | 0000000000000000 () -> Kafka.Batching.AccumulatorByTopic<Kafka.Routing.FetchMessage>.Tick
  33 | (L) @     999 ms every     1000 ms | 0000000000000000 () -> Kafka.Batching.AccumulatorByTopic<Kafka.Routing.FetchMessage>.Tick
  34 | (L) @    2000 ms every     2000 ms | 0000000000000000 () -> Kafka.Batching.AccumulatorByTopicByPartition<Kafka.Cluster.ProduceMessage>.Tick
  38 | (L) @     999 ms every     1000 ms | 0000000000000000 () -> Kafka.Batching.AccumulatorByTopic<Kafka.Routing.OffsetMessage>.Tick

The first part of the output lists each instance of Timer with start/repeat information followed by the callback parameter and callback if available. The second list groups timer so you could identify cases where the same timers have been created many times.

tpq: what is in the ThreadPool queues

If no culprit is identified in timers, it is time to look at what is in the ThreadPool with the tpq command.

> tpq
 
global work item queue________________________________
0x000002AC3C1DDBB0 Work | (ASP.global_asax)System.Web.HttpApplication.ResumeStepsWaitCallback
                       ...
0x000002AABEC19148 Task | System.Threading.Tasks.Dataflow.Internal.TargetCore<System.Action>.<ProcessAsyncIfNecessary_Slow>b__3
 
local per thread work items_____________________________________
0x000002AE79D80A00 System.Threading.Tasks.ContinuationTaskFromTask
                       ...
0x000002AB7CBB84A0 Task | System.Net.Http.HttpClientHandler.StartRequest
 
   7 Task System.Threading.Tasks.Dataflow.Internal.TargetCore<System.Action>.<ProcessAsyncIfNecessary_Slow>b__3
                       ...
  84 Task System.Net.Http.HttpClientHandler.StartRequest
----
6039
 
1810 Work  (ASP.global_asax) System.Web.HttpApplication.ResumeStepsWaitCallback
----
1810

Both work items from the global queue and the local queues are displayed with each identified callback. The final summary spits work items and tasks.

Miscellaneous commands

Two other helper commands are also available.

tks: Task State

Pass a Task object reference to tks to get its human readable state.

> help tks
-------------------------------------------------------------------------------
TaskState [hexa address] [-v <decimal state value>]
 
TaskState translates a Task m_stateFlags field value into human readable format.
It supports hexadecimal address corresponding to a task instance or -v <decimal state value>.
 
> tks 000001db16cf98f0
Running
 
> tks -v 73728
WaitingToRun

dcq : Dump ConcurrentQueue

Dump elements stored in a ConcurrentQueue. Due to implementation details, more manual steps are required in case of value types.

> help dcq
-------------------------------------------------------------------------------
DumpConcurrentQueue
 
Lists all items in the given concurrent queue.
 
For simple types such as numbers, boolean and string, values are shown.
> dcq 00000202a79320e8
System.Collections.Concurrent.ConcurrentQueue<System.Int32>
   1 - 0
   2 - 1
   3 - 2
 
In case of reference types, the command to dump each object is shown.
> dcq 00000202a79337f8
System.Collections.Concurrent.ConcurrentQueue<ForDump.ReferenceType>
   1 - dumpobj 0x202a7934e38
   2 - dumpobj 0x202a7934fd0
   3 - dumpobj 0x202a7935078
 
For value types, the command to dump each array segment is shown.
The next step is to manually dump each element with dumpvc <the Element Methodtable> <[item] address>.
> dcq 00000202a7933370
System.Collections.Concurrent.ConcurrentQueue<ForDump.ValueType>
   1 - dumparray 202a79334e0
   2 - dumparray 202a7938a88

Note that for reference type items, the dumpobj command is provided to get the value of the item fields: you just copy and paste them to get instance fields.

GIT hell

Before digging into the implementation details, I want to spend some time on how to properly create a pull request. Since the Diagnostics repository is hosted on Github, you have to create a fork and push your changes on a dedicated branch to then submit a pull request.

Due to my previous Team Foundation Server experience, it seems that I have problems with Git commands: too many different ways to do simple things maybe. So I was faithfully relying on the documentation to configure/sync a fork and I ended up merging the Microsoft Master branch to our Criteo fork Master before creating my own branch dedicated to my pull request:

After a while, in the next pull requests, I started to have unrelated commits in the pull requests:

It was due to the fact that I was merging the Criteo fork master with Diagnostics master to stay in sync.

My coworker Guillaume explained to me that I should follow a different path. This time, I’m just creating a branch on the Microsoft repository (so no need to merge) and I’m passing by the Criteo fork just to push upstream:

git clone https://github.com/dotnet/diagnostics
cd diagnostics
git checkout -b PR_dotnet-dump_pstacks

Now I can change my local source code before pushing upstream to Criteo fork

git add/gui 
git commit
git remote add upstream https://github.com/criteo-forks/diagnostics
git push upstream PR_dotnet-dump_pstacks

That way, I can synchronize the Criteo fork without “impacting” the history of commits brought with my dedicated pull request branch.

The next episode will cover command implementation details.

What’s next? Read the second part of this article:

How to write commands for dotnet-dump This post describes the different steps, tips and tricks to write your own commands for dotnet-dumpmedium.com

Like what you are reading? Check out more articles from Christophe on our medium account.

The .NET Core Journey at Criteo *This post shows the challenges we faced during the migration to .NET Core on containerized Linux for our main…*medium.com

Are you interested to work with Christophe and other talented Engineers at Criteo? Take a look at our open positions:

Product, Research & Development | Criteo Careers careers.criteo.com

Introduction

When I arrived at Criteo in late 2016, I joined the .NET Core “guild” (i.e. group of people from different teams dedicated to a specific topic). The first meeting I attended included Microsoft folks led by Scott Hunter (head of .NET program management) and including David Fowler (SignalR and ASP.NET Core). The goal for Criteo was simple: Moving a set of C# applications from Windows/.NET Framework to Linux/.NET Core. I guess that for Microsoft we were a customer with workloads that could be interesting to support with .NET Core. At that time, I did not realize how strong their commitment to work with us was. Our Open Source mindset was the selling point.

How complicated could it be? Well… this post will show you the challenges that we had to face to run, monitor and debug our applications.

Try it

Once we got a build of all .NET Core assemblies (more on this in a forthcoming blog post), it was time to run a few applications. The first issues that we faced were related to missing features between .NET Framework and .NET Core. For example, we need cryptography support of 3DES and AES with cypher mode CFB but it is (still) not available in .NET Core for Linux. Thanks to the Open Source status of .NET Core, we were able to add it to CoreFx. However, since we did not implement it on MacOS/Windows as Microsoft requested for our change to be accepted as a Pull Request, we had to keep our Criteo-forked branch.

The second class of runtime problems we had to solve were due to differences between Windows and Linux but also with the “containerization” of the runtime environment. Let’s take two examples involving the .NET Garbage Collector. First, our containers were using Linux cgroups to manage quotas including memory and number of CPU cores usable by applications. However, at CLR startup, the GC was counting the total count of CPU cores to compute the number of heaps to allocate instead of the one defined at the cgroup level: We ended up with instant Out Of Memory automatic killing. This time our fix was done and merged in the CLR repository.

The second example is related to a GC optimization: During background generation 2 collections, the CLR threads working underneath are affinitized to each different CPU core to avoid locks. We were lucky enough to welcome Maoni Stephens (Lead Dev on the GC) in our Paris office early 2018 to share our weird allocation patterns that impacted the GC. During her stay, she was kind enough to help us investigate a behavior on our servers: When SysInternals ProcessExplorer was running, the garbage collections were taking more time than usual. Maoni found out ProcessExplorer had an affinitized high priority thread conflicting with GC threads. During investigations related to longer response time on Linux compared to Windows. We realized that GC threads were not affinitized like it was the case on Windows and the issue was fixed by Jan Vorlicek.

Here is our lesson: Sometimes fixes are merged into the official release and sometimes they are not. If your workloads are pushing .NET to its limits, you will probably have to build and manage your own Core fork and make it available to your deployments.

Monitor it

At Criteo, our Grafana dashboards measuring .NET Framework application health were based on metrics computed from Windows performance counters. Even without going to Linux, .NET Core is no more exposing performance counters so we had to entirely rebuild our metrics collection system!

Based on Microsoft feedbacks, we decided to listen to CLR events emitted via ETW on Windows and LTTng on Linux. In addition to work for both Operating Systems, these events are also providing accurate details about thread contention, exceptions and garbage collections not available with Performance counters. Please refer to our series of blog posts for more details and reusable code samples to integrate these events into your own systems.

Our first Linux metrics collection implementation was based on LTTng and we presented our journey during the Tracing Summit in 2017. Microsoft already built TraceEvent, an assembly allowing .NET code to parse CLR events for both Windows and Linux. Unfortunately for us, the Linux part was only able to load traces files but we needed live session like on Windows where you can listen to events emitted by running applications. Since this code is Open Source, Gregory was able to add the live session feature to TraceEvent.

With .NET Core 3.0 Microsoft provided a way to exchange events common to Linux and Windows called EventPipes. So… we moved our collection implementation from LTTng to EventPipe (look at our blog series and DotNext conference session for more details and reusable code sample). With the new EventPipe implementation in the CLR came performance issues not seen by Microsoft. The reason is simple: Some of our applications are running hundreds of threads to process thousands of requests per second and allocate memory like crazy. In that kind of context, the CLR has a lot to do and so, has a lot of events to generate and emit via LTTng or EventPipes.

The initial implementation was lacking some filtering and too many events were generated or expensive event payload was created even though the events were not emitted. Based on our feedback, the Microsoft Diagnostic team was very responsive and quickly fixed the problem.

Microsoft did not “just” move to Open Source, the teams are working deeply integrated with the issue/pull request model of GitHub. So don’t be shy and if you find a problem, create an issue with a detailed reproduction and even better, provide a pull request with the fix. Everyone in the community will benefit!

Run it

With these metrics, we started to investigate some performance differences (mostly response time) between Windows and containerized Linux.

We saw a huge performance difference on Linux: Both response time (x2) and scalability (timeout increase with QPS). Our team spent a lot of time to improve the situation up to the point where it was possible to send the applications to production.

In the new containerized environment we faced the same kind of noisy neighbor symptoms that we had with Process Explorer. If the CPU cores are not dedicated to a container (as it was for us at the beginning), this scenario happens a lot. So we updated the scheduling system to dedicate CPU cores to containers.

On a totally different area, we found out that the way .NET Core handles network I/O continuation had an impact on our main application. To give a bit of context, this application has to handle a lot of requests and is response-time driven. During the processing of a request, the current thread might have to send an HTTP request before continuing its processing. Since this is done asynchronously, the thread is now available to process more incoming requests and this is good for throughput. However, it means that when the inner HTTP request comes back, all available threads might be processing new incoming requests and it will take time to complete the old one. The net effect is to increase the median response time and this is not something we want!

The .NET Core implementation is relying on the .NET ThreadPool that shares its threads with all the async/await magic and the incoming requests processing (The .NET Framework implementation is using a totally different implementation based on I/O completion ports on Windows). To solve the issue, Kevin implemented a custom thread pool to handle network I/O and we keep on optimizing it. When you work on this kind of deep area of code-shared by so many different workloads, you realize that it is impossible to find the silver bullet.

Debug it

What would you do if something would go wrong in an application? On Windows, with Visual Studio, we are able to remote debug a rogue application to set a breakpoint, look at fields and properties or even have a high-level view of what threads are doing with the ParallelStacks view. In the worst case, SysInternals procdump allows us to take a snapshot of the application and analyze it on our developer’s machine with WinDBG or Visual Studio.

In terms of remote debugging a Linux application, Microsoft provides an SSH-based solution to attach to a running application. However, for security reasons, it is not allowed to run an SSH server in our Criteo containers. The solution was to implement the communication protocol with VsDbg for Linux on top of WebSockets.

Well… this was not enough. Hosting architecture (Marathon and Mesos in our case) ensures that applications in containers are running smoothly by sending requests to health check endpoints. If the application replies that everything is fine, then the container is safe. If the application does not answer as expected (including retries), then Marathon/Mesos kills the application and cleans up the container. Now think about what will happen if you set a breakpoint in the application and you dig into the data structures content in Visual Studio Watch/Quick Watch panels for a few minutes. Behind the scene, the debugger has to freeze all application threads, including the ones from the thread pool responsible to answer health checks. As you have probably guessed already, the debugging session will not end well.

This is why the previous figure shows an arrow between Marathon and the Remote Debugger which acts as a proxy for the application health check. When a debugging session starts (i.e. when the WebSockets code executes the protocol), the Remote Debugger knows that it should answer OK instead of calling the application endpoint that might never answer.

When remote debugging is not enough, how do you take a memory snapshot of the application? For example, if the health check does not answer after a series of retry, the Remote Debugger is calling the createdump tool installed with the .NET Core runtime to generate a dump file. Again, since the memory dump creation of 40+ GB applications could take several minutes, the same health check proxy mechanism has been put in place.

Once the dump file is created, the remote debugger let Marathon kill the application. But wait! This is not enough because in that case, the container will be cleaned up and the disk storage will disappear. Not a problem, after a dump has been generated by createdump, the file is sent to a “Dump Navigator” application (one per data center). This application is providing a simple HTML user interface to get high-level details of the application state such as thread stacks or managed heap content.

On Windows, we have built our own set of extension commands that allow us to investigate memory, threadpool starvation, thread contention, or timer leak scenarios in a Windows memory dump with WinDBG as shown during this NDC Oslo conference session. Note that they are also usable with LLDB on Linux. These commands are leveraging the ClrMD Microsoft library that gives you access to a live process or a memory dump in C#. Thanks to the Linux support that has been added to this library by Microsoft developers, it was easy to reuse the code into our Dump Navigator application. I definitively recommend to look at the API provided by ClrMD to automate and build your own tools. The long Criteo blog series is a good start in addition to my DotNext conference session.

Conclusion

Even though some of our main applications moved to .NET Core running on containerized Linux with a large set of monitoring/debugging tools, the journey is not over. We are now testing the preview of .NET Core 5.0 (like we did for 3.0) to check if it supports Criteo specific needs. If this is not the case, we will figure out why and find solutions to integrate into the code. Same for the tools: I have started to add our extension commands to Microsoft dotnet-dump CLI tool used to analyze both Windows and Linux dumps.

At least we could say that we not only helped ourselves but also Microsoft to understand how far .NET Core could go and even the whole .NET Windows and Linux community. This is where Open Source shines!

Stay tuned for the next article in our mini-series. Don’t forget to head over to our previous articles of this journey:

Migrating Arbitrage to Apache Mesos *Lessons learned from migrating our largest application to our container platform.*medium.comMoving .NET to Linux at Scale *The story of a multi-year migration: How we changed Criteo’s whole foundation.*medium.com

Interested in joining the challenge? Head over to our career site!

Product, Research & Development | Criteo Careers careers.criteo.com

This post of the series details how to look into your threads stack with ClrMD.

Introduction

It’s been a long time (see the resources at the end) since I’ve been discussing what ClrMD could bring to .NET developers/DevOps! My colleague Kevin just wrote an article about how to emulate SOS DumpStackObjects command both on Windows and Linux with ClrMD. This implementation lists the objects on the stack but without their values (like strings content for example) nor the stack frames corresponding to the method calls.

The rest of the post will show you, with ClrMD, how to get an higher view, closer to what the SOS ClrStack command could provide.

Let’s take this simple application as an example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

class Program
{
    static void Main(string[] args)
    {
        Host h = new Host();
        h.Base();
    }
}

class Host
{
    public void Base()
    {
        int iValue = Guid.NewGuid().GetHashCode();
        bool bValue = (iValue % 2) == 0;
        string parameter = Guid.NewGuid().ToString().Substring(0,1) + "_1234567890";

        Console.WriteLine(parameter + " = " + First(iValue, bValue, parameter));
    }

    private int First(int iValue, bool bValue, string parameter)
    {
        Guid guid = Guid.NewGuid();
        return Second(bValue, guid) / 2;
    }

    private int Second(bool bValue, Guid guid)
    {
        return Third(guid).Length;
    }

    private string Third(Guid guid)
    {
        Console.WriteLine($"call   procdump -ma {Process.GetCurrentProcess().Id}");
        Console.ReadLine();
        return $"{guid.GetHashCode()}#{guid.GetHashCode()}#{guid.GetHashCode()}";
    }
}

As you can see, I’ve mixed value and reference types as parameters and local variables up to the call to the Third method that displays the procdump command line to execute in order to generate a memory dump of the process.

Use WinDBG + SOS Luke!

When you open it with WinDBG and load SOS, here is the result of the dso command:

The clrstack command shows the stacked method calls:

And if you use the -a parameter, you will get methods with their parameters and local variables (or -p for parameters only and -l for local variables only):

It is weird that SOS implementation does not give the type of both the parameters and locals. But wait! While researching for this post, I looked at the SOS implementation (now in the strike.cs file moved from the coreclr to the diagnostics repository) to find this nice comment:

So I tried clrstack with -i and I got the types for parameters (and locals unlike what the comments implies):

Even though clrstack supports the -all flag to dump the call stack of all managed threads, you might need to do your own automatic analysis on hundreds of threads and this is where ClrMD shines.

Merging methods and parameters/locals

When I read Kevin’s post, I immediately thought about adding the method call on the stack based on the work I’ve done in March 2019 to implement the pstacks tool. At that time, my goal was to aggregate the call stacks of a large number of threads in order to find out pattern of blocked threads, sharing the “same” call stacks. Visual Studio provides a great “Parallel Stacks” pane but I needed it for both Windows and Linux.

To list all the call stack with ClrMD, you simply enumerate the managed threads and for each one, its StackTrace property contains the list of StackFrame objects corresponding to each method call.

The StackPointer property of each frame contains the address of the frame in the call stack, allowing a mapping of the method call with its parameters and locals:

As always with stacks, lower addresses correspond to the last things added to the stack (i.e. last called method). While checking between what is shown by SOS, the parameters/locals addresses and frame stack pointers, you realize that all objects at an address in the stack equal or below the StackPointer of a frame are either parameters or local variables of the frame method.

Even better, for non static method, you can guess what is the this* *implicit parameter if the address is the same as the frame StackPointer; shown with the green = sign in the previous screenshot and prefixed by > in Kevin’s updated code that merges the method calls to the parameters and locals:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

for (ulong ptr = stackTop; ptr <= stackLimit; ptr += (ulong)runtime.PointerSize)
{
    // look for the frame corresponding to the current position in the stack
    if (currentFrame.StackPointer <= ptr)
    {
        Console.WriteLine(FormatFrame(currentFrame));

        nFrame++;
        if (nFrame < frames.Count)
        {
            currentFrame = frames[nFrame];
        }
        else
        {
            break;
        }
    }

    ulong obj;
    if (!runtime.ReadPointer(ptr, out obj))
    {
        break;
    }

    if (!IsInHeap(heap, obj))
    {
        continue;
    }

    var type = heap.GetObjectType(obj);
    if (type == null || type.IsFree)
    {
        continue;
    }

    // try to find implicit "this" parameter in case of non-static method
    var myFrame = (nFrame == 0) ? currentFrame : frames[nFrame-1];
    separator = ((myFrame.StackPointer == ptr) &&
                 (myFrame.Method != null) &&
                 (!myFrame.Method.IsStatic))
        ? ">" 
        : " ";
    Console.Write($"{ptr:x16}   {separator} ");
    DumpObject(heap, type, (ulong)obj);

}

The FormatFrame helper method simply prefix static methods with # instead of | for instance methods:

Unfortunately, I did not find any way with ClrMD to make the difference between parameters and locals. Based on what you can see in SOS implementation of this part of the clrstack command, it relies on the EnumerateArguments and EnumetateLocalVariables methods of ICorDebugILFrame which is not exposed by ClrMD. There is another undocumented implementation based on private interfaces I could not leverage neither. For a larger discussion around stack walking in .NET, read this great post by Matt Warren.

Also, without any explicit access to specific parameter or local, I did not find a way to get the value of primitive and value type instances stored on the stack. However, it is still possible to get them for boxed ones and reference type instances such as string for example.

Getting instances from the stack

In the last code excerpt, I did not describe the DumpObject helper method used to display an object on the stack. The implementation provided by Kevin was used to show the address and the type of the object:

1

Console.WriteLine($"{objAddress:x16} {type.Name}");

The next step would be to display value for primitive types such as numbers, boolean, string and even array size:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

private static void DumpObject(ClrHeap heap, ClrType type, ulong objAddress)
{
    // get value for simple types
    string valueOrAddress = 
        (type.Name == "System.Char") ? $"{type.GetValue(objAddress),16}" :
        (type.Name == "System.String") ? $"{FormatString(type.GetValue(objAddress).ToString())}" :
        (type.Name == "System.Bool") ? $"{FormatString(((bool)type.GetValue(objAddress)).ToString())}" :
        (type.Name == "System.Byte") ? $"{FormatString(((byte)type.GetValue(objAddress)).ToString())}" :
        (type.Name == "System.SByte") ? $"{FormatString(((sbyte)type.GetValue(objAddress)).ToString())}" :
        (type.Name == "System.Decimal") ? $"{FormatString(((decimal)type.GetValue(objAddress)).ToString())}" :
        (type.Name == "System.Double") ? $"{FormatString(((double)type.GetValue(objAddress)).ToString())}" :
        (type.Name == "System.Single") ? $"{FormatString(((float)type.GetValue(objAddress)).ToString())}" :
        (type.Name == "System.Int32") ? $"{FormatString(((int)type.GetValue(objAddress)).ToString())}" :
        (type.Name == "System.SInt32") ? $"{FormatString(((uint)type.GetValue(objAddress)).ToString())}" :
        (type.Name == "System.Int64") ? $"{FormatString(((long)type.GetValue(objAddress)).ToString())}" :
        (type.Name == "System.SInt64") ? $"{FormatString(((ulong)type.GetValue(objAddress)).ToString())}" :
        (type.IsArray) ? $"{FormatString(GetArrayAsString(type, objAddress))}" :
        $"{objAddress:x16}";  // work also for IntPtr
    Console.WriteLine($"{valueOrAddress} {type.Name}");
}

Most of this code is based on the GetValue helper from ClrType: it returns the right “thing” for simple types. Look at ClrMD implementation details to get a better understanding of how the value is rebuilt.

The GetArrayAsString simply returns the number of elements in the array:

1
2
3
4
5

private static string GetArrayAsString(ClrType type, ulong objAddress)
{
    var elementCount = type.GetArrayLength(objAddress);
    return $"length = {elementCount}";
}

And the call stack is now complete!

Note that you may even get more locals or parameters than with WinDBG+SOS but don’t ask me why…

For more advanced object formatting cases such as dumping structs or enumerating fields and their value, I would highly recommend to look at the related ClrMD documentation page (just replace GCHeapType by ClrType and you’ll be safe).

Resources

• Dumping stack objects with ClrMD by Kevin Gosse
• Stack walking in the .NET Runtime by Matt Warren
• Part 1: Bootstrap ClrMD to load a dump.
• Part 2: Find duplicated strings with ClrMD heap traversing.
• Part 3: List timers by following static fields links.
• Part 4: Identify timers callback and other properties.
• Part 5: Use ClrMD to extend SOS in WinDBG.
• Part 6: Manipulate memory structures like real objects.
• Part 7: Manipulate nested structs using dynamic.
• Part 8: Spelunking inside the .NET Thread Pool
• Part 9: Deciphering Tasks and Thread Pool items

Part 1: Bootstrap ClrMD to load a dump.

Part 2: Find duplicated strings with ClrMD heap traversing.

Part 3: List timers by following static fields links.

Part 4: Identify timers callback and other properties.

Part 5: Use ClrMD to extend SOS in WinDBG.

Part 6: Manipulate memory structures like real objects.

Part 7: Manipulate nested structs using dynamic.

Introduction

The previous posts introduced the DynaMD nuget that helps navigating among type instances using a C#-like syntax “instance.field”. Let’s see how to use it to enumerate the asynchronous items queued in the .NET thread pool. As a bonus, the running tasks and work items won’t be forgotten.

ThreadPool internals

The .NET ThreadPool is keeping track of the pending work items into two different data structures:

• A global queue: stored as a ThreadPoolWorkQueue instance referenced by the workQueue static field

• several per-thread (TLS) local queues: stored in SparseArray<ThreadPoolWorkQueue+WorkStealingQueue> linked from the allThreadQueues static field

As you can see, the algorithm to list the pending tasks and work items starts from a static field and iterate on a linked list of QueueSegment for global queue and array of WorkStealingQueue for per thread queues. Both are storing arrays of IThreadPoolWorkItem that Task and QueueUserWorkItemCallback are implementing:

Too much theory… Let’s write some code!

Global ThreadPool queue

You have seen in a previous post how to access the value of a static field per application domain:

EnumerateGlobalThreadPoolItems-1.cs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

public IEnumerable<ThreadPoolItem> EnumerateGlobalThreadPoolItems()
{
    // look for the ThreadPoolGlobals.workQueue static field
    ClrModule mscorlib = GetMscorlib();
    if (mscorlib == null)
        throw new InvalidOperationException("Impossible to find mscorlib.dll");

    ClrType queueType = mscorlib.GetTypeByName("System.Threading.ThreadPoolGlobals");
    if (queueType == null)
        yield break;

    ClrStaticField workQueueField = queueType.GetStaticFieldByName("workQueue");
    if (workQueueField == null)
        yield break;

    // the CLR keeps one static instance per application domain
    foreach (var appDomain in _clr.AppDomains)
    {

For an application domain in which the threadpool is not used, we need to check against null for the expected ThreadPoolWorkQueue:

EnumerateGlobalThreadPoolItems-2.cs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

        object workQueueValue = workQueueField.GetValue(appDomain);
        ulong workQueueRef = (workQueueValue == null) ? 0L : (ulong)workQueueValue;
        if (workQueueRef == 0)
            continue;

        // should be System.Threading.ThreadPoolWorkQueue
        ClrType workQueueType = _heap.GetObjectType(workQueueRef);
        if (workQueueType == null)
            continue;
        if (workQueueType.Name != "System.Threading.ThreadPoolWorkQueue")
            continue;

        foreach (var item in EnumerateThreadPoolWorkQueue(workQueueRef))
        {
            yield return item;
        }
    }
}

The role of the EnumerateThreadPoolWorkQueue helper method is to iterate on each QueueSegment of the linked list pointed to by the queueTail field of the per appdomain ThreadPoolWorkQueue object.

At the beginning of the following code, note that dynamic allows writing C# code even though the queueTail and nodes fields are not known at compile time. Even more convenient, a foreach statement is possible when the instance behind the DynaMD proxy is an array:

EnumerateThreadPoolWorkQueue.cs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

private IEnumerable<ThreadPoolItem> EnumerateThreadPoolWorkQueue(ulong workQueueRef)
{
    // start from the tail and follow the Next
    var proxy = _heap.GetProxy(workQueueRef);
    var currentQueueSegment = proxy.queueTail;

    while (currentQueueSegment != null)
    {
        // get the System.Threading.ThreadPoolWorkQueue+QueueSegment nodes array
        var nodes = currentQueueSegment.nodes;
        if (nodes == null)
            continue;

        foreach (var item in nodes)
        {
            if (item == null)
                continue;

            yield return GetThreadPoolItem(item);
        }

        currentQueueSegment = currentQueueSegment.Next;
    }
}

The GetThreadPoolItem helper method will be described soon but first, let’s see how to get the items from the thread local queues.

Local ThreadPool queues

The same static field driven operations are needed to access the sparse array containing… more arrays:

EnumerateLocalThreadPoolItems.cs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

public IEnumerable<ThreadPoolItem> EnumerateLocalThreadPoolItems()
{
    var queueType = GetMscorlib().GetTypeByName("System.Threading.ThreadPoolWorkQueue");
    if (queueType == null)
        yield break;

    ClrStaticField threadQueuesField = queueType.GetStaticFieldByName("allThreadQueues");
    if (threadQueuesField == null)
        yield break;

    foreach (ClrAppDomain domain in _clr.AppDomains)
    {
        ulong? threadQueueRef = (ulong?)threadQueuesField.GetValue(domain);
        if (!threadQueueRef.HasValue || threadQueueRef.Value == 0)
            continue;

        var threadQueue = _heap.GetProxy((ulong)threadQueueRef);
        if (threadQueue == null)
            continue;

        var sparseArray = threadQueue.m_array;
        if (sparseArray == null)
            continue;

        foreach (var stealingQueue in sparseArray)
        {
            if (stealingQueue == null)
                continue;

            foreach (var item in EnumerateThreadPoolStealingQueue(stealingQueue))
            {
                yield return item;
            }
        }
    }
}

The spare arrays contain either null or a stealing queue that itself contains… an array:

EnumerateThreadPoolStealingQueue.cs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

private IEnumerable<ThreadPoolItem> EnumerateThreadPoolStealingQueue(dynamic stealingQueue)
{
    var array = stealingQueue.m_array;
    if (array == null)
        yield break;

    foreach (var item in array)
    {
        if (item == null)
            continue;

        yield return GetThreadPoolItem(item);
    }
}

Now that we managed to retrieve the thread pool items, we can try to decipher them.

Deciphering thread pool items

A thread pool item stored in the global or in the local queues could be a Task, a QueueUserWorkItemCallback or a simple method:

GetThreadPoolItem.cs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

private ThreadPoolItem GetThreadPoolItem(dynamic item)
{
    // get the ClrType directly from the dynamic proxy
    ClrType itemType = item.GetClrType();

    if (itemType.Name == "System.Threading.Tasks.Task")
    {
        return GetTask(item);
    }
    else if (itemType.Name == "System.Threading.QueueUserWorkItemCallback")
    {
        return GetQueueUserWorkItemCallback(item);
    }
    else
    {
        // create a raw information
        ThreadPoolItem tpi = new ThreadPoolItem()
        {
            Type = ThreadRoot.Raw,
            Address = (ulong)item,
            MethodName = itemType.Name
        };

        return tpi;
    }
}

The kind of item is computed from the ClrType of the object given by the ClrHeap.GetObjectType method. An ulong address is expected by this ClrMD method and it would be easy to get from the dynamic returned by DynaMD by just casting it to ulong. However, it is easier to simply call the GetClrType method on the dynamic proxy!

Next step…

The next and last episode of the ClrMD series will show you how to decipher tasks and thread pool items to know which of your methods will be called. As a bonus, the running tasks and work items won’t be forgotten.

Co-authored with Kevin Gosse

Part 1: Bootstrap ClrMD to load a dump.

Part 2: Find duplicated strings with ClrMD heap traversing.

Part 3: List timers by following static fields links.

Part 4: Identify timers callback and other properties.

Part 5: Use ClrMD to extend SOS in WinDBG.

Part 6: Manipulate memory structures like real objects.

Let’s start with a reminder of the object we’re manipulating via our proxy:

Values.cs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

public struct Size
{
   public int Width;
   public int Height;
}

public struct Description
{
   public int Id;
   public Size Size;
}

public class Sample
{
   public int Value;
   public string Name { get; }
   public Description Description;
   public Sample Child;
}

Accessing the value of proxy.Description.Id is a special case. Why? Because Size is a struct, and therefore is embedded directly inside of Sample outside of the responsibility of the managed heap. This scenario isn’t directly supported by ClrMD, and calling GetValue on the Size field will return null instead of the address of the struct. We need to compute this address ourselves.

What is the layout of those objects in memory? To find out, we can create a Sample object and then check how the memory is structured within WinDBG

Sample.cs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

new Sample
{
   Name = "Test",
   Value = 1,
   Child = new Sample(),
   Description = new Description
   {
      Id = 2,
      Size = new Size
      {
         Width = 3,
         Height = 4
      }
   }
}

The first address (4 bytes because the dump was taken on a 32-bit process) points to the “method table”; some metadata used internally by the CLR to identify the strong type corresponding to this instance in the managed heap. Next, we get the “1” stored by the Value field, followed by a pointer to the “Test” string stored by the Name field. The field values of the Description structure (in dark blue) are then embedded directly inside of the Sample object. And inside of that structure, we find the values of the fields of the Size structure (in light blue). Finally, the reference to the Child field is stored last in the memory of the Sample object.

To sum it up, to get the address of the Description field, we need to take the address of the Sample instance, add 4 bytes for the method table (8 bytes for 64-bit memory dumps), then add the offset of the field (which counts the size of the previous fields and the padding if any):

DynamicProxy.cs

1
2
3
4
5
6

private DynamicProxy LinkToStruct(ClrField field)
{
   var childAddress = _address + (ulong)field.Offset + (ulong)_heap.PointerSize;

   return new DynamicProxy(_heap, childAddress);
}

We call this method from TryGetMember when we have a struct that isn’t a primitive type. ClrMD gives that information thanks to the HasSimpleValue property of ClrField:

DynamicProxy.cs

1
2
3
4
5
6

if (!field.HasSimpleValue)
{
   result = LinkToStruct(field);

   return true;
}

There’s another subtlety though. As we’ve seen in the layout, the Description struct, embedded inside of the Sample class, doesn’t have a method table of its own. The consequence is that we can’t find out its type by using ClrHeap.GetObjectType. As a workaround, we add a constructor allowing us to manually set the underlying ClrMD type of the object impersonated by the proxy:

DynamicProxy.cs

1
2
3
4
5

private DynamicProxy(ClrHeap heap, ulong address, ClrType overrideType)
    : this(heap, address)
{
   _type = overrideType;
}

This constructor is called when we still have the type information (taken from the ClrField):

DynamicProxy.cs

1
2
3
4
5
6

private DynamicProxy LinkToStruct(ClrField field)
{
   var childAddress = _address + (ulong)field.Offset + (ulong)_heap.PointerSize;

   return new DynamicProxy(_heap, childAddress, field.Type);
}

Now we’re able to read the value of a field of a nested struct:

Program.cs

1

Console.WriteLine(sample.Description.Id);

Are we done now? Almost. There is one last case to handle, the Size struct that is nested inside of Description, itself a struct nested inside of a Sample instance. How is that an issue? If we use the same code to compute the address of the Size struct, then we will be adding the 4/8 bytes for the method table. Except that, as we’ve seen, the nested Size struct doesn’t have a method table! We need to add a condition to the code to know whenever we are inside of a nested struct and handle this corner case:

DynamicProxy.cs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

private DynamicProxy LinkToStruct(ClrField field)
{
   var childAddress = _address + (ulong)field.Offset;

   if (!_interior)
   {
      childAddress += (ulong)_heap.PointerSize;
   }

   return new DynamicProxy(_heap, childAddress, field.Type);
}

The Boolean flag _interior is set in the second constructor that accepts a ClrType:

DynamicProxy.cs

1
2
3
4
5
6

private DynamicProxy(ClrHeap heap, ulong address, ClrType overrideType)
    : this(heap, address)
{
   _type = overrideType;
   _interior = true;
}

We’re finally able to read the value of the nested-nested struct:

Program.cs

1

Console.WriteLine(sample.Description.Size.Width * sample.Description.Size.Height);

Next time, we’ll see how to use the same mechanisms to manipulate arrays in a convenient way.

Co-authored with Kevin Gosse

Part 1: Bootstrap ClrMD to load a dump.

Part 2: Find duplicated strings with ClrMD heap traversing.

Part 3: List timers by following static fields links.

Part 4: Identify timers callback and other properties.

Part 5: Use ClrMD to extend SOS in WinDBG.

As we’ve seen in the previous articles of the series, exploring a complex data structure using ClrMD can quickly become tedious.

Let’s take a concrete example. Imagine we have those types declared:

Values.cs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

public struct Size
{
   public int Width;
   public int Height;
}

public struct Description
{
   public int Id;
   public Size Size;
}

public class Sample
{
   public int Value;
   public string Name { get; }
   public Description Description;
   public Sample Child;
}

Given the address of the Sample object in the memory dump, even with the GetFieldValue helper method to make it simpler, the code to navigate these recursive data types is still… verbose:

Program.cs

1

Console.WriteLine((uint)GetFieldValue(heap, currentSampleRef, "Value"));

And now, how to get the value of the Name property?

Same question for the inner Child fields or deeper Size field of its Description?

Wouldn’t that be great if we could navigate just like through real strongly typed instances? In short, to be able to write:

Program.cs

1
2
3
4
5
6
7

var sample = heap.GetProxy(0x00001000); // Some address obtained by other ways

Console.WriteLine(sample.Value);
Console.WriteLine(sample.Name);
Console.WriteLine(sample.Child.Name);
Console.WriteLine(sample.Description.Id);
Console.WriteLine(sample.Description.Size.Width * sample.Description.Size.Height);

Instead of:

Program.cs

1
2
3
4

Console.WriteLine(GetFieldValue(heap, sampleRef, "Value"));
Console.WriteLine(GetFieldValue(heap, sampleRef, "<Name>k__BackingField"));
Console.WriteLine(GetFieldValue(heap, GetFieldValue(heap, sampleRef, "Child"), "Name"));
// and so on...

The first issue is: what is the GetProxy method going to return? Since we don’t know at compilation time the properties of the object the code is going to manipulate, we need a way to support some kind of late-binding. Fortunately, this scenario is supported in C# through the usage of the dynamic keyword. As you will see in the rest of this post, this is not only a keyword but also an extensible mechanism that perfectly fits our need to define fields at runtime instead of compile time.

We start by creating a class inheriting from System.Dynamic.DynamicObject.

DynamicProxy.cs

1

internal class DynamicProxy : DynamicObject

This base class provides all the facilities needed for late-binding:

As you will see, only a few of these virtual methods need to be overridden to support our scenario.

To construct our proxy, we need two parameters: the ClrMD ClrHeap object, that allows us to browse the objects in the memory, and the address of the object we want to impersonate.

DynamicProxy.cs

1
2
3
4
5

public DynamicProxy(ClrHeap heap, ulong address)
{
   _heap = heap;
   _address = address;
}

We also provide an extension method for convenience:

Extensions.cs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

namespace Microsoft.Diagnostics.Runtime
{
   public static class Extensions
   {
      public static dynamic GetProxy(this ClrHeap heap, ulong address)
      {
         if (address == 0)
         {
            return null;
         }

         return new DynamicProxy(heap, address);
      }
   }
}

The next step is to override the virtual TryGetMember method, inherited from DynamicObject. It is automatically invoked whenever somebody tries to access a any member of the dynamic object, including its fields.

DynamicProxy.cs

1

public override bool TryGetMember(GetMemberBinder binder, out object result)

The Name property of the binder parameter provides the name of the accessed member and we are supposed to return the corresponding proxy object as the out result parameter.

We’re going to need the type of the object. For convenience, we store it in a property:

DynamicProxy.cs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

protected ClrType Type
{
   get
   {
      if (_type == null)
      {
         _type = _heap.GetObjectType(_address);
      }

      return _type;
   }
}

Using the binder.Name property containing the name of the field we’re trying to access, we retrieve the ClrMD field description:

DynamicProxy.cs

1

var field = Type.GetFieldByName(binder.Name);

From there, we get the value marshalled by ClrMD and assign it to the result out parameter:

DynamicProxy.cs

1

result = field.GetValue(_address);

Finally, we signal that we managed to bind the invoked member:

DynamicProxy.cs

1

return true;

This is just a handful of lines of code, but it’s enough for the simple cases where field values are primitive types.. This covers the “Value” field for our Sample type. For the auto-property “Name”, that’s trickier, because the name of the underlying field has characters that are forbidden in C#: “k__BackingField”. If we write this, it won’t compile:

Program.cs

1

Console.WriteLine(proxy.<Name>k__BackingField);

We can handle this case by guessing the name of the compiler-generated field, then accessing it:

DynamicProxy.cs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

var field = Type.GetFieldByName(binder.Name);

if (field == null)
{
   // The field wasn't found, it could be an autoproperty
   field = Type.GetFieldByName($"<{binder.Name}>k__BackingField");

   if (field == null)
   {
      // Still not found
      throw new InvalidOperationException("Field not found: " + binder.Name);
   }
}

Thanks to this trick, we can write:

Program.cs

1

Console.WriteLine(proxy.Name);

Great! The next challenge is to transparently manipulate the “Child” field as a reference to another “Sample” object. To achieve this goal, the field could simply return another DynamicProxy object that we can manipulate the same way as its parent.

First, we need a helper to find out whether a value is a reference or not:

DynamicProxy.cs

1
2
3
4

private static bool IsReference(object result, ClrType type)
{
   return !(result is string) && type.IsObjectReference;
}

We treat string as a special case, because ClrMD gives us the marshaled string rather than a reference like for all other types. That’s how we were able to retrieve the value of the Name field previously.

Now we call the helper and return a new proxy whenever we’re dealing with a reference:

DynamicProxy.cs

1
2
3
4

if (IsReference(result, field.Type))
{
   result = new DynamicProxy(_heap, (ulong)result);
}

We can now write:

Program.cs

1

Console.WriteLine(proxy.Child.Name);

That’s it for accessing a referenced object allocated on the heap. However, this won’t work for accessing an embedded struct such as proxy.Description.Id. We’ll see in the next part how to handle this specific case.

Co-authored with Kevin Gosse

Part 1: Bootstrap ClrMD to load a dump.

Part 2: Find duplicated strings with ClrMD heap traversing.

Part 3: List timers by following static fields links.

Part 4: Identify timers callback and other properties.

Introduction

Since the beginning of this series, you have seen how to use ClrMD to write your own tool to extract meaningful information from a dump file (or a live process). However, most of the time, you are also using WinDBG and SOS to navigate inside the .NET data structures.

It would be convenient if you could leverage the new .NET exploration features based on ClrMD the same way you are using SOS. This post will explain how to achieve this goal by implementing an extension that exports commands callable from within WinDBG.

Deciphering a Task status

During one of our debugging investigations, we needed to get the value of the Status property for a few Task instances. If you take a look at the implementation of the property getter in a decompiler (or from source code), you will see that it is computed based on the value of the internal m_stateFlags field.

In WinDBG, the !DumpHeap -stat command lists all types with their instance count. If the .prefer_dml 1 command has been set, you even get hyperlinks on some values such as the address or MT (for MethodTable). If you click the MT value for System.Threading.Tasks.Task, you get all instances of type Task:

Click any address and look at the value of the m_stateFlags field:

It is easy to automate the retrieval of the m_stateFlags instance field value with ClrMD as explained earlier:

GetTaskStateFromAddress.cs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

private static ulong GetTaskStateFromAddress(ulong address)
{
    var type = Runtime.GetHeap().GetObjectType(address);

    if ((type != null) && (type.Name.StartsWith("System.Threading.Task")))
    {
        // try to get the m_stateFlags field value
        ClrInstanceField field = type.GetFieldByName("m_stateFlags");
        if (field != null)
        {
            var val = field.GetValue(address);
            if (val != null)
            {
                try
                {
                    return (ulong)(int)val;
                }
                catch (InvalidCastException)
                {
                }
            }
        }
    }

    return 0;
}

The ClrType corresponding to the address is first checked to ensure that it represents a Task instance. Next, its GetFieldByname helper method returns a ClrInstanceField that provides the status via its GetValue function.

The next step is to transform this number into a TaskStatus enumeration value by simply using a decompiler and copying the logic from the Task getter code:

GetTaskState.cs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

private static string GetTaskState(ulong flag)
{
    TaskStatus rval;

    if ((flag & TASK_STATE_FAULTED) != 0) rval = TaskStatus.Faulted;
    else if ((flag & TASK_STATE_CANCELED) != 0) rval = TaskStatus.Canceled;
    else if ((flag & TASK_STATE_RAN_TO_COMPLETION) != 0) rval = TaskStatus.RanToCompletion;
    else if ((flag & TASK_STATE_WAITING_ON_CHILDREN) != 0) rval = TaskStatus.WaitingForChildrenToComplete;
    else if ((flag & TASK_STATE_DELEGATE_INVOKED) != 0) rval = TaskStatus.Running;
    else if ((flag & TASK_STATE_STARTED) != 0) rval = TaskStatus.WaitingToRun;
    else if ((flag & TASK_STATE_WAITINGFORACTIVATION) != 0) rval = TaskStatus.WaitingForActivation;
    else if (flag == 0) rval = TaskStatus.Created;
    else return null;

    return rval.ToString();
}

It would be a time saver if this translation could be done by a command right inside WinDBG instead of relying on another tool based on ClrMD in which addresses are pasted.

WinDBG extension 101

In addition of being a native Windows debugger, WinDBG supports extensions: .dll files that you load with the .load command. They are exporting commands that are callable from within WinDBG with the “!” prefix. These commands are usual native exports that can be seen with tools such as http://www.dependencywalker.com/ as shown by the next screenshot:

As you can see, all SOS commands are functions exported by the sos.dll native binary. Before digging into the extension functions implementation, notice that a few other functions could also be exported. Among them, the DebugExtensionInitialize function provides version information (i.e. which version of the debugging API is expected) and must be exported to be called by WinDBG when the dll is loaded.

Read this post for more details about how to develop a native WinDBG extension.

All extension command functions take two parameters:

• An IDebugClient instance to interact with WinDBG
• An ANSI string for the arguments (such as “-stat” for !dumpheap)

The bridge between your extension commands and WinDBG is provided by the IDebugClient COM interface. But don’t be scared: no need to manually deal with native COM interface with ClrMD! The DataTarget**.**CreateFromDebuggerInterface method takes an IDebugClient interface and returns an instance of DataTarget. As you might remember from the initial post of this series, DataTarget is the gateway to the dump (or live-debugged attached process): we are now back to the known ClrMD world.

Reuse ClrMD Samples

Hopefully, most of the glue to bind the native world to ClrMD is already available! You simply reuse the partial DebuggerExtensions class given in the samples.

You extend the class with your extension methods that take the following signature:

MyCommandSignature.cs

1

public static void MyCommand(IntPtr client, [MarshalAs(UnmanagedType.LPStr)] string args)

The first parameter is a pointer to the IDebugClient interface provided by WinDBG. The first thing to do in your extension command method is to call the InitApi static method with the interface pointer and let the magic happens.

MyCommand-2.cs

1
2
3

// Must be the first thing in our extension.
if (!InitApi(client))
    return;

After that call, the output of the Console will be redirected to WinDBG and your code is free to use the following properties to access the dump via ClrMD:

DebuggerExtensionPartial.cs

1
2
3
4
5
6
7

public partial class DebuggerExtensions
{
    public static IDebugClient DebugClient { get; private set; }

    public static DataTarget DataTarget { get; private set; }

    public static ClrRuntime Runtime { get; private set; }

The second parameter args received by your method is a string that contains the parameters added by the user after the name of your command. For example, if the user types “MyCommand param1 param2”, the args parameter will be “param1 param2”.

Exposing native functions

The last part of magic glue is how to export a native function from a .NET assembly. This is made possible by the UnmanagedExports nuget package by Robert Giesecke.

Once added to your project, decorate the functions to export with the DllExport attribute and the native name of the function that will be visible in WinDBG as a command.

There is a little trick here: the names of exported functions are case sensitive for WinDBG. If you take a look again at sos.dll in Dependency Walker and sort exports by Function column, you will notice a few duplicates such as CLRStack/ ClrStack/ clrstack as shown in the following screenshot:

For usability sake, it is a good practice to provide several syntaxes for the same command, including short version such as !dso for !DumpStackObjectin SOS. Unfortunately the DllExport attribute does not allow multiple applications on the same method with different exported names. You need to define a different method per exported name and all of them will call the same internal helper method.

MultipleDllExport.cs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

[DllExport("tks")]
public static void tks(IntPtr client, [MarshalAs(UnmanagedType.LPStr)] string args)
{
    OnTkState(client, args);
}

[DllExport("tkstate")]
public static void tkstate(IntPtr client, [MarshalAs(UnmanagedType.LPStr)] string args)
{
    OnTkState (client, args);
}

[DllExport("tkState")]
public static void tkState(IntPtr client, [MarshalAs(UnmanagedType.LPStr)] string args)
{
    OnTkState (client, args);
}

public static void OnTkState (IntPtr client, [MarshalAs(UnmanagedType.LPStr)] string args)
{
    // Must be the first thing in our extension.
    if (!InitApi(client))
        return;
    ...
}

Thanks to the GetTaskStateFromAddress and GetTaskState helper methods described earlier, the implementation of the OnTkState method is straightforward once the address or the value has been extracted from the args parameter.

Don’t forget your user: implement help

A good extension always provides an help command that (1) lists the available commands with shortcuts and (2) additional details on each command. Simply add a new file that defines the exports for help/Help and parses the string argument if needed.

DebuggerExtensionImpl.cs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

public partial class DebuggerExtensions
{
    [DllExport("Help")]
    public static void Help(IntPtr client, [MarshalAs(UnmanagedType.LPStr)] string args)
    {
        OnHelp(client, args);
    }

   [DllExport("help")]
   public static void help(IntPtr client, [MarshalAs(UnmanagedType.LPStr)] string args)
   {
        OnHelp(client, args);
   }
 
    const string _help = "...";
    const string _tksHelp = "...";

    private static void OnHelp(IntPtr client, string args)
    {
        // Must be the first thing in our extension.
        if (!InitApi(client))
            return;

        string command = args;

        if (args != null)
            command = args.ToLower();

        switch (command)
        {
            case "tks":
            case "tkstate":
                Console.WriteLine(_tksHelp);
            break;
 
            default:
                Console.WriteLine(_help);
            break;
        }
    }
}

Tips to use the extension

Don’t forget that you might need two versions of your assembly: one for the x86 version of WinDBG if your applications are 32 bit and one for the x64 version of WinDBG in the 64 bit case. If you want to be able to easily load your extension with the .load command, copy it with Microsoft.Diagnostics.Runtime.dll (i.e. ClrMD assembly) to the winext subfolder of x64/x86 WinDBG folders:

Before being able to use any of its commands, you must load SOS with the well-known .loadby sos clr mantra. But this is not enough: you also have to run at least one SOS command. You are now ready to call any of your extension commands!

Next step…

The next episodes will bring you into the mysteries under the dynamic keyword and how to simplify the syntax to leverage ClrMD.

Co-authored with Kevin Gosse

Part 1: Bootstrapping ClrMD to load a dump.

Part 2: Finding duplicated strings with ClrMD heap traversing.

Part 3: List timers by following static fields links.

Looking at my timer

In the previous post, we explained how to access a static field of TimerQueue to start iterating the list of TimerQueueTimer wrapping the created timers. Now that the currentPointer variable contains the address of each TimerQueueTimer, it is time to extract the details of the timer we have created.

The following code extracts the value of the TimerQueueTimer fields corresponding to each Timer thanks to the GetFieldValue helper presented in the previous post:

TimerQueueTimerFields.cs

1
2
3
4
5
6
7
8

var val = GetFieldValue(heap, currentTimerQueueTimerRef, "m_dueTime");
ti.DueTime = (uint)val;

val = GetFieldValue(heap, currentTimerQueueTimerRef, "m_period");
ti.Period = (uint)val;

val = GetFieldValue(heap, currentTimerQueueTimerRef, "m_canceled");
ti.Cancelled = (bool)val;

Note that the value for m_dueTime is always the same as the value of m_period. This is not a bug but it seems that .NET is only keeping the due time during construction but use the corresponding field for other purpose after.

The m_state field case is a little bit more complicated to decipher because the type of the object passed to the timer needs to be figured out in addition to its address, if the latter is not null:

TimerQueueTimerState.cs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

val = GetFieldValue(heap, currentTimerQueueTimerRef, "m_state");
ti.StateTypeName = "";
if (val == null)
{
   ti.StateAddress = 0;
}
else
{
   ti.StateAddress = (ulong)val;
   var stateType = heap.GetObjectType(ti.StateAddress);
   if (stateType != null)
   {
      ti.StateTypeName = stateType.Name;
   }
}

As usual with ClrMD, you need to get the ClrType corresponding to the object referenced by an address before being able to access its fields or to get its name. However, instead of looking into a module as it has been done for TimerQueue, it is easier and more efficient to call the GetObjectType from ClrHeap. Remember that the mandatory test against a null value for the ClrType might seem overkill but the ClrMD implementation states that it is possible that the internal CLR state could be corrupted.

What is the timer callback?

The last piece of information to retrieve is the callback the timer will call when it triggers. The _timerCallback field references a TimerCallback instance that stores these details.

GetTimerCallBackDetails.cs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

// decypher the callback details
val = GetFieldValue(heap, currentTimerQueueTimerRef, "m_timerCallback");
if (val != null)
{
   ulong elementAddress = (ulong)val;
   if (elementAddress == 0)
      continue;

   var elementType = _heap.GetObjectType(elementAddress);
   if (elementType != null)
   {
      if (elementType.Name == "System.Threading.TimerCallback")
      {
         ti.MethodName = BuildTimerCallbackMethodName(runtime, elementAddress);
      }
      else
      {
         ti.MethodName = "<" + elementType.Name + ">";
      }
   }
   else
   {
      ti.MethodName = "{no callback type?}";
   }
}
else
{
   ti.MethodName = "???";
}
yield return ti;

currentPointer = GetFieldValue(heap, currentTimerQueueTimerRef, "m_next");

But how to get the name of the method just with the address of a TimerCallback object? Again, open up your favorite decompiler and look at the type hierarchy:

Here are the two fields of the Delegate type that are interesting:

The _methodPtr field stores the pointer to the method. By chance, the ClrRuntime GetMethodByAddress method takes this address and returns the name of the method!

If this method is static, the _target fields is null. Otherwise, it stores the value of this, the hidden parameter received by all instance methods. In case of type inheritance, it is interesting to know which override will be called. All these steps are wrapped in the following helper function:

BuildTimerCallbackMethodName.cs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

private string BuildTimerCallbackMethodName(ClrRuntime runtime, ulong timerCallbackRef)
{
   var heap = runtime.GetHeap();
   var methodPtr = GetFieldValue(heap, timerCallbackRef, "_methodPtr");
   if (methodPtr != null)
   {
      ClrMethod method = runtime.GetMethodByAddress((ulong)(long)methodPtr);
      if (method != null)
      {
         // figure out the real callback implementor type thanks to _target
         string thisTypeName = "?";
         var thisPtr = GetFieldValue(heap, timerCallbackRef, "_target");
         if ((thisPtr != null) && ((ulong) thisPtr) != 0)
         {
            ulong thisRef = (ulong) thisPtr;
            var thisType = heap.GetObjectType(thisRef);
            if (thisType != null)
            {
               thisTypeName = thisType.Name;
            }
         }
         else
         {
            thisTypeName = (method.Type != null) ? method.Type.Name : "?";
         }
         return string.Format("{0}.{1}", thisTypeName, method.Name);
      }
   }
   return string.Empty;
}

Building a usable summary

Even though the EnumerateTimers helper provides a way to list all timers, you often don’t want to show them all; especially when thousands exist and most of them are duplicates. The sample code associated to this post lists the different timers, count the duplicates and sort the result by duplicate count as shown in the following screenshot:

Next step…

After timers, the next post will show how to integrate your ClrMD-based code into an extension for WinDBG to help decyphering Task state.

Co-authored with Kevin Gosse

Part 1: Bootstrapping ClrMD

Post 2: Finding duplicated strings with ClrMD

Marshaling data from a dump

Beyond heap navigation shown in the previous post, the big thing to understand about ClrMD is that the retrieved information is often an address. An address from another address space because the dump is seen as another process just like if you were debugging it live. Your code will need to access the other process memory corresponding to this address; not directly with a pointer/reference indirection or with the raw Win32 ReadProcessMemory API function but via APIs like GetObjectType or GetValue.

To illustrate how to navigate into the dump address space with ClrMD, we will show how to list the timers that have been started. This can be useful to investigate various issues, such as leaks or timers being stuck.

Know your framework

A naive implementation, like the string example of the previous post, would try to list all object instances in the CLR heap and look at Timer instances only. However, as it has been mentioned already, this is very inefficient in terms of performance; especially for 10+ GB dumps…

It is time to figure out what happens in the .NET runtime when your code creates a new timer. If the source code of the version of the CLR you are using is not available, start your favorite IL decompiler and look at the System.Threading.Timer implementation details. The parameters given to the constructors (such as the due time, period, and callback method, in addition to its optional parameter if any) are not stored in the class itself but in the TimerQueueTimer helper class.

The Timer constructor code, after a few sanity checks, calls the TimerSetup method to wrap a TimerQueueTimer in a TimerHolder that is stored in the Timer m_timer field.

This is where things start to become interesting: this TimerQueueTimer class adds each new instance into a linked list kept by a singleton object stored in the static s_queue field of the TimerQueue class. The following figure shows the relation between instances after three timers are created:

So… a fast way to list the timers would be to get the unique static instance of TimerQueue, look at its m_timers field and iterate on each TimerQueueTimer by following their m_next field until it contains null. The rest of the post details the following operations with ClrMD:

• quickly getting a ClrType
• reading a static field
• reading an instance field

to fill up a collection of our own TimerInfo used to easily create a summary:

TimerInfo.cs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

public class TimerInfo
{
   public ulong TimerQueueTimerAddress { get; set; }
   public uint DueTime { get; set; }
   public uint Period { get; set; }
   public bool Cancelled { get; set; }
   public ulong StateAddress { get; set; }
   public string StateTypeName { get; set; }
   public ulong ThisAddress { get; set; }
   public string MethodName { get; set; }
}

This is wrapped inside a helper method described in the next few sections:

EnumerateTimers-1

1
2
3
4
5

public IEnumerable<TimerInfo> EnumerateTimers(ClrRuntime runtime)
{
   ClrHeap heap = runtime.GetHeap();
   if (!heap.CanWalkHeap)
      yield break;

As explained in the previous post, you need to ensure that the process was not in the middle of a garbage collection when the dump was taken by checking the value of the ClrHeap.CanWalkHeap property.

Standing on the shoulders of giants

I have found the different steps to get access to the static fields of classes in the ClrMD implementation from GitHub. In addition, I highly recommend that you take a look at the samples.

Let’s go back to our first goal: getting the value of the static s_queue field of the TimerQueue class. One of the very efficient optimization found in the ClrMD implementation is to directly get a ClrType from a module and call its GetTypeByName method instead of iterating the heap until an instance of the type is found. In our case, we need to access TimerQueue which is a type from mscorlib. Here is the code of the helper function from Desktop\threadpool.cs to get a ClrModule for mscorlib:

GetMscorlib.cs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

private ClrModule GetMscorlib(ClrRuntime runtime)
{
    foreach (ClrModule module in runtime.Modules)
        if (module.AssemblyName.Contains("mscorlib.dll"))
            return module;

    // Uh oh, this shouldn't have happened.  Let's look more carefully (slowly).
    foreach (ClrModule module in runtime.Modules)
        if (module.AssemblyName.ToLower().Contains("mscorlib"))
            return module;

    // Ok...not sure why we couldn't find it.
    return null;
}

The following line sets timerQueueType with the ClrType corresponding to TimerQueue:

EnumerateTimers-2.cs

1

var timerQueueType = GetMscorlib(runtime).GetTypeByName("System.Threading.TimerQueue");

Next, get the ClrStaticField corresponding to the static field s_queue:

EnumerateTimers-3.cs

1

ClrStaticField staticField = timerQueueType.GetStaticFieldByName("s_queue");

The staticField variable is not the static instance but rather a way to access it… or them.

But where are my statics!

Let’s take some time to explain a “detail” of the .NET Framework to help you understand how to get the static TimerQueue instance. Unlike previous Windows frameworks, .NET allows a process to contain several running environments called application domains (a.k.a. AppDomains). For a better isolation, each AppDomain has its own set of static variables: this is why you need to iterate on each AppDomain with ClrMD to access the static instances:

EnumerateTimers-4.cs

1
2
3
4
5

foreach (ClrAppDomain domain in runtime.AppDomains)
{
    ulong? timerQueue = (ulong?)staticField.GetValue(domain);
    if (!timerQueue.HasValue || timerQueue.Value == 0)
        continue;

The address returned by ClrStaticField.GetValue is nullable because, in an AppDomain where no TimerQueue has ever been used, its fields won’t be initialized.

We don’t really need to map this address from the dump address space into something usable in the tool. Instead, only the value of the m_timers field is interesting to be able to start iterating on the list of timers.

How to get the values of instance fields?

Now that we have an address in the dump and the ClrType describing the type of the corresponding object (TimerQueue here), it is easy to retrieve the value of one of its instance fields. Since this action is needed again and again to move from one TimerQueueTimer object to the next, it is valuable to create a helper method:

GetFieldValue.cs

1
2
3
4
5
6
7
8
9

private object GetFieldValue(ClrHeap heap, ulong address, string fieldName)
{
    var type = heap.GetObjectType(address);
    ClrInstanceField field = type.GetFieldByName(fieldName);
    if (field == null)
        return null;

    return field.GetValue(address);
}

The address of the object in the dump is used to get its ClrType. The ClrInstanceField (instead of a ClrStaticField as for the s_queue case) describing the property exposes the expected GetValue method. Note that the return value of GetValue is defined as System.Object but you should understand it as the numeric value stored in the dump (or the other process address space) at the given address. For the simple value types such as boolean, number and even ulong address, a cast will be enough to transparently marshal the value into the tool with ClrMD.

Let’s go back to writing the code to access to head of the TimerQueueTimer list from the TimerQueue static instance:

EnumerateTimers-5.cs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

// m_timers is the start of the list of TimerQueueTimer
var currentPointer = GetFieldValue(heap, timerQueue.Value, "m_timers");

while ((currentPointer != null) && (((ulong)currentPointer) != 0))
{
    // currentPointer points to a TimerQueueTimer instance
    ulong currentTimerQueueTimerRef = (ulong)currentPointer;

    TimerInfo ti = new TimerInfo()
    {
        TimerQueueTimerAddress = currentTimerQueueTimerRef
    };

    ...

 currentPointer = GetFieldValue(heap, currentTimerQueueTimerRef, "m_next");
}

currentPointer holds the address of each TimerQueueTimer in the list kept by the static TimerQueue.

Note the ((ulong)currentPointer) != 0) test in the while loop to detect the end of the list when the m_next field is null.

Next step…

After enumerating each timer, the next post will show how to extract details such as the due time, the period, and even which method is called when it ticks.

Co-authored with Kevin Gosse

Part 1: Bootstrapping ClrMD to load a dump.

From ClrRuntime to ClrHeap or how to traverse the managed heap

In the previous post, we have boostrapped the code needed to load a memory dump and get an instance of ClrRuntime. This type is the starting point for accessing the content of a managed process with ClrMD:

Most of the memory and heap management is well described in the ClrMD documentation, you are able to:

• list the application domains with AppDomains,
• dig into the memory regions with EnumerateMemoryRegions,
• access the managed heap with GetHeap

As shown in the ClrMD samples, the ClrHeap type helps you traversing the managed memory:

Before doing any heap exploration with ClrHeap, you need to ensure that the process was not in the middle of a garbage collection when the dump was taken. This can be done by checking CanWalkHeap:

clrmd.cs

1
2
3
4

var heap = clr.GetHeap();

if (!heap.CanWalkHeap)
   return;