C++ on Welcome to Christophe Nasarre's Blog

How to support .NET Framework PDB format and source line with ISymUnmanagedReader

Wed, 11 Feb 2026 09:16:11 +0000

In my previous posts, I explained how to use DIA and DbgHelp to map a method to its line in source code. I forgot to mention that it was correct for .NET Core but not for the “old” .NET Framework Windows PDB format. Instead of encoding the method token in the name, the symbol file contains the name of the methods. So, how to do the mapping for .NET Framework assemblies? You will find the answer (plus some tricks) in this article.

When I started to work on the support of the old Windows PDB format, I looked at what existed to parse the raw format and… I decided to try DbgHelp instead. With this first implementation, I realized that some token where missing and source code information was not retrieved for most of the methods.

So, I looked for another API to use and I found ISymUnmanagedReader. The usage philosophy is totally different from DIA or DbgHelp.

A little bit of magic

This interface is implemented in diasymreader.dll that comes with every .NET Framework installation. But you need to do COM magic to get it. After having called CoInitialize to setup COM, you ask for an instance of ISymUnmanagedBinder from CLSID_CorSymBinder_SxS:

1
2
3
4
5
6
7
8


CComPtr<ISymUnmanagedBinder> pBinder;
hr = CoCreateInstance(
    CLSID_CorSymBinder_SxS,
    NULL,
    CLSCTX_INPROC_SERVER,
    IID_ISymUnmanagedBinder,
    (void**)&pBinder
);

From the binder, you can get the ISymUnmanagedReader interface corresponding to the assembly you are interested in with GetReaderForFile. However, there are two tiny details to consider.

First, one parameter expects the path to the assembly, not to the .pdb file. That symbol file has to be stored in the same folder but note that the documentation states that you could have more flexible search with ISymUnmanagedBinder2::GetReaderForFile2 but I did not test it.

The second detail is the first parameter: an instance of IMetaDataImport for the same assembly. The steps to get it are… complicated.

Hosting the CLR

The idea is to host the .NET Framework and get the corresponding ICLRMetaHost interface:

1
2


CComPtr<ICLRMetaHost> pMetaHost;
HRESULT hr = CLRCreateInstance(CLSID_CLRMetaHost, IID_ICLRMetaHost, (void**)&pMetaHost);

Calling the CLRCreateInstance API allows you to get an instance of ICLRMetaHost from which you could enumerate installed version of .NET Framework. In my case, I know which version I want:

1
2
3


// Get the installed .NET Framework runtime (v4.0+)
CComPtr<ICLRRuntimeInfo> pRuntimeInfo;
hr = pMetaHost->GetRuntime(L"v4.0.30319", IID_ICLRRuntimeInfo, (void**)&pRuntimeInfo);

The ICLRRuntimeInfo interface allows you to get access to runtime services via GetInterface:

1
2


CComPtr<IMetaDataDispenser> pDispenser;
hr = pRuntimeInfo->GetInterface(CLSID_CorMetaDataDispenser, IID_IMetaDataDispenser, (void**)&pDispenser);

The service I’m interested in is the IMetadataDispenser interface that allows you to “open a scope” on the assembly you are interested in:

1
2
3
4
5
6


hr = pDispenser->OpenScope(
    wModulePath.c_str(),
    ofRead,
    IID_IMetaDataImport,
    (IUnknown**)&_pMetaDataImport
);

Note that the first parameter is the path to the assembly not to the .pdb file. The scope is abstracted by an IMetadataImport interface I have already described and that is needed to call GetReaderForFile: and get the ISymUnmanagedReader:

hr = pBinder->GetReaderForFile(_pMetaDataImport, wModulePath.c_str(), nullptr, &_pReader);

The road to get symbol details for a method

The ISymUnmanagedReader interface implements GetMethod to get details about a given method token via an ISymUnmanagedMethod interface. So, the next question is how to get these tokens. If you remember the previous article, these tokens are from the 06 MethodDef table in the assembly metadata; starting from 06000001 to the last one.

This means that you could write a simple loop starting from 1 up to a hardcoded maximum value, call TokenFromRid(index, mdtMethodDef) to get the corresponding token. However, since you are a professional developer, you would search for the exact number of tokens from IMetadataTables retrieved from IMetadataImport:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


ULONG cRows = 0;

// Get IMetaDataTables interface to query the MethodDef table
CComPtr<IMetaDataTables> pTables;
hr = _pMetaDataImport->QueryInterface(IID_IMetaDataTables, (void**)&pTables);
if (FAILED(hr) || pTables == nullptr)
{
    cRows = LAST_METHODDEF_TOKEN;
}
else
{
    // Get the number of rows in the MethodDef table (table index 0x06 = Method)
    hr = pTables->GetTableInfo(
        0x06,           // MethodDef table
        NULL,           // cbRow (not needed)
        &cRows,         // pcRows (number of methods)
        NULL,           // pcCols (not needed)
        NULL,           // piKey (not needed)
        NULL            // ppName (not needed)
    );

    if (FAILED(hr))
    {
        cRows = LAST_METHODDEF_TOKEN;
    }
}

Now that you have the number of rows (i.e. number of methods defined in the metadata), it is easy and safe to get method information from symbols:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


for (uint32_t i = 1; i <= cRows; i++)
{
    mdMethodDef token = TokenFromRid(i, mdtMethodDef);

    CComPtr<ISymUnmanagedMethod> pMethod;
    hr = _pReader->GetMethod(token, &pMethod);
    if (SUCCEEDED(hr) && pMethod != nullptr)
    {
        MethodInfo info;
        if (GetMethodInfoFromSymbol(pMethod, info))
        {
            _methods.push_back(info);
        }
    }
}

Note that GetMethod might fail (returning E_FAIL) for P/Invoked functions, abstract methods, or methods decorated with DebuggerHidden attribute.

Give me line and source code!

For the other methods with symbol information, you can get its token via the GetToken method. The ISymUnmanagedMethod interface allows low level access to line/column mapping that is beyond the scope of this article. At a high level, positions in source file are named sequence points. Call GetSequencePointCount to get… the number of sequence points for a given method.

The next step is to call GetSequencePoints with the number of points you want and the corresponding arrays of offsets, lines, columns, end lines, end columns and ISymUnmanagedDocument. In my case, I’m only interested in where the method starts so the first sequence point is good enough:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


// Get sequence points (source line information)
ULONG32 cPoints = 0;
hr = pMethod->GetSequencePointCount(&cPoints);
if (SUCCEEDED(hr) && (cPoints > 0))
{
    cPoints = 1; // We only need the first sequence point for start line
    std::vector<ULONG32> offsets(cPoints);
    std::vector<ULONG32> lines(cPoints);
    std::vector<ULONG32> columns(cPoints);
    std::vector<ULONG32> endLines(cPoints);
    std::vector<ULONG32> endColumns(cPoints);
    std::vector<ISymUnmanagedDocument*> documents(cPoints);

    ULONG32 actualCount = 0;
    hr = pMethod->GetSequencePoints(
        cPoints,
        &actualCount,
        &offsets[0],
        &documents[0],
        &lines[0],
        &columns[0],
        &endLines[0],
        &endColumns[0]
    );

    if (SUCCEEDED(hr) && (actualCount > 0))
    {

The source file is described by ISymUnmanagedDocument that provides its name when GetURL is called:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


// Get the first sequence point's document and line
        ISymUnmanagedDocument* pDoc = documents[0];
        if (pDoc != nullptr)
        {
            // Get document URL (file path)
            ULONG32 urlLen = 0;
            hr = pDoc->GetURL(0, &urlLen, NULL);
            if (SUCCEEDED(hr) && (urlLen > 0))
            {
                std::vector<WCHAR> url(urlLen);
                hr = pDoc->GetURL(urlLen, &urlLen, &url[0]);
                if (SUCCEEDED(hr))
                {
                    // Convert wide string to narrow string
                    int len = WideCharToMultiByte(CP_UTF8, 0, &url[0], urlLen, NULL, 0, NULL, NULL);
                    std::string narrowUrl(len, '\0');
                    WideCharToMultiByte(CP_UTF8, 0, &url[0], urlLen, &narrowUrl[0], len, NULL, NULL);
                    info.sourceFile = narrowUrl;
                }
            }

            // NOTE: 0xFEEFEE is a special value indicating hidden lines
            info.lineNumber = lines[0];
            pDoc->Release();
        }
    }
}

The final interesting trick is that the line number might have the special 0xFEEFEE value. It means that the line is hidden. I have seen it for methods generated by the C# compiler such as MoveNext for async state machines or anonymous methods:

The source code is available from my Github repository.

Happy coding!

References

But where is my method code? DbgHelp comes to the rescue

Fri, 16 Jan 2026 08:21:30 +0000

Introduction

In our Datatog continuous .NET profiler implementation, we collect the call stack of a thread when something interesting happens such as an exception is thrown for example. In addition to the method name we would like to figure out at what line in which source code file this method is implemented.

This information is usually stored in the program database (.pdb) file that is generated by the compiler when the assembly is generated from the source code. The type and the name of the method are stored in the metadata of the assembly itself but I already told this story before. The .NET compilers support two formats of .pdb: the Portable format for .NET Core and the Windows format for .NET Framework.

I’ve explained how to use the DIA API and it is now time to show how to leverage the DbgHelp API that is available on all Windows machines (even though it is recommended to always install the latest release via the Debugging Tools for Windows.

This time, my goal is to extract from a Windows .pdb file the source code and line information for a given managed method. You can find the corresponding source code of this DumpLine tool in my Github repository.

Starting with DbgHelp

There are two major ways to get access to symbols with DbgHelp: either from a running process (i.e. to map the currently loaded .dll to their associated .pdb files) or from a tool that would explicitly load a .dll or a .pdb file.

Before anything, you tell DbgHelp which options you want by calling SymSetOptions:

1
2
3
4
5
6
7
8


DWORD options = SymGetOptions();
    options |= SYMOPT_DEBUG;
    options |= SYMOPT_LOAD_LINES;           // Load line number information
    options |= SYMOPT_UNDNAME;              // Undecorate symbol names
    //options |= SYMOPT_DEFERRED_LOADS;       // Defer symbol loading
    options |= SYMOPT_EXACT_SYMBOLS;        // Require exact symbol match
    options |= SYMOPT_FAIL_CRITICAL_ERRORS; // Don't show error dialogs
    SymSetOptions(options);

This is where I ask that line number information should be collected.

The next step is to call SymInitialize to setup DbgHelp environment. The first parameter expects a process handle (returned by GetCurrentProcess in my case). You could pass a path where to find the .pdb files for your dlls as a second parameter. In my case, since I will provide a .pdb file path, I don’t need it, and NULL will be passed. It means that, if needed, DbgHelp will use the current folder and the path set in _NT_SYMBOL_PATH and _NT_ALTERNATE_SYMBOL_PATH environment variables.

The last boolean parameter tells DbgHelp if you want that SymLoadModule64 to be called for each and every loaded .dll in the given process. Definitively not what I want so I’m passing FALSE.

1
2
3
4
5


_hProcess = GetCurrentProcess();
    if (!SymInitialize(_hProcess, NULL, FALSE))
    {
        _hProcess = NULL;
    }

At that point, I’m ready to load a .pdb file.

Loading a .pdb file

The API is straightforward: just call SymLoadModuleEx :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


_baseAddress = SymLoadModuleEx(
        _hProcess,
        NULL,
        pdbFilePath.c_str(),
        NULL,
        0x10000000, // arbitrary base address
        0,
        NULL,
        0
    );

    if (_baseAddress == 0)
    {
        return false;
    }

The important parameters are the process handle (same as for SymInitialize) and the path of the .pdb file. I’ve lost some time trying to understand why my code was not working due to a weird behavior of this function. You know that it succeeds when the returned address is not 0. Well… This is not 100% correct. If the path you provide does not exist, you won’t get 0 but the base address that you also provide. Even worth, when you call the functions I’ll detail later on, no error will happen but nothing will work as expected. So, I simply check that the file exists:

1
2
3
4
5


// BUG? : dbghelp does not fail if the .pdb file does not exist...
    if (GetFileAttributesA(pdbFilePath.c_str()) == INVALID_FILE_ATTRIBUTES)
    {
        return false;
    }

Note that it is possible to unload the symbols of a given loaded module by calling SymUnloadModule with the same process handle and its base address: this will reduce the memory consumption if you don’t need the symbols anymore.

In case of deferred load symbols, it is needed to call SymGetModuleInfo64 before trying to access the symbols:

1
2
3
4
5
6


IMAGEHLP_MODULE64 moduleInfo = { 0 };
    moduleInfo.SizeOfStruct = sizeof(IMAGEHLP_MODULE64);
    if (!SymGetModuleInfo64(_hProcess, _baseAddress, &moduleInfo))
    {
        return false;
    }

In addition, this will fill up an IMAGEHLP_MODULE64 structure with possibly interesting details:

The .pdb signature and age could be useful to build urls to communicate with symbol servers; but this is another story:

1
2
3
4
5
6
7
8
9


_age = moduleInfo.PdbAge;
    GUID guid = moduleInfo.PdbSig70;
    char strGUID[80];
    sprintf_s(strGUID, 80, "%08x%04x%04x%02x%02x%02x%02x%02x%02x%02x%02x",
        guid.Data1, guid.Data2, guid.Data3,
        guid.Data4[0], guid.Data4[1], guid.Data4[2], guid.Data4[3],
        guid.Data4[4], guid.Data4[5], guid.Data4[6], guid.Data4[7]
        );
    _guid = strGUID;

You also know if line numbers are available or not thanks to the LineNumbers field.

Note that if you asked for deferred symbols option, you won’t get any interesting details:

Only the module name and path are provided but nothing else.

Enumerating the methods

It is now time to iterate on the symbols in the loaded .pdb thanks to SymEnumSymbols:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


if (!SymEnumSymbols(
            _hProcess,
            _baseAddress,
            "*!*",  // Mask (all symbols)
            EnumMethodSymbolsCallback,
            this    // User context to store the methods in _methods instance field
    ))
    {
        return false;
    }

In addition to the obvious parameters, this function expects a callback function that will be called for each symbol in the module specified by the process handle and the base address. Note that you can pass any context as the last parameter. In my case, the instance of my DbgHelpParser class is passed to be able to store the methods in a dedicated _methods field:

1
2
3
4
5
6
7
8
9


struct MethodInfo
{
    std::string name;
    uint64_t address;
    uint32_t size;
    std::string sourceFile;
    uint32_t lineNumber;
};
std::vector<MethodInfo> _methods;

The “*!*” mask tells DbgHelp to look for symbols in all modules. This might sound counter intuitive, but the syntax is similar to what you find in WinDBG or Visual Studio: !. This could be useful if you load more than one .pdb.

The job of the callback function is to detect the symbols you are interested in from the SYMBOL_INFO structure passed for each matching symbol:

1
2
3
4
5
6
7
8
9


BOOL CALLBACK DbgHelpParser::EnumMethodSymbolsCallback(PSYMBOL_INFO pSymInfo, ULONG SymbolSize, PVOID UserContext)
{
    DbgHelpParser* parser = reinterpret_cast<DbgHelpParser*>(UserContext);

    if (
        (pSymInfo->Tag == SymTagFunction) &&
        ((pSymInfo->Flags & (SYMFLAG_CLR_TOKEN | SYMFLAG_METADATA)) == (SYMFLAG_CLR_TOKEN | SYMFLAG_METADATA))
        )
    {

The Tag field contains a value from SymTagEnum but, for a managed .pdb file, you will only get SymTagFunction. Also, the Flags field should contain SYMFLAG_CLR_TOKEN and SYMFLAG_METADATA because we are only interested in managed methods.

Next, you get the name, address and size from other fields before looking for the source file and line details by calling SymGetLineFromAddr64:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


MethodInfo info;
        info.name = pSymInfo->Name;
        info.address = pSymInfo->Address;
        info.size = pSymInfo->Size;

        // Try to get source file and line information
        IMAGEHLP_LINE64 line = { 0 };
        line.SizeOfStruct = sizeof(IMAGEHLP_LINE64);
        DWORD displacement = 0;

        if (SymGetLineFromAddr64(parser->_hProcess, pSymInfo->Address, &displacement, &line))
        {
            info.sourceFile = line.FileName ? line.FileName : "";
            info.lineNumber = line.LineNumber;
        }
        else {
            info.sourceFile = "";
            info.lineNumber = 0;
        }

        parser->_methods.push_back(info);
    }

    return TRUE; // Continue enumeration
}

The callback returns TRUE to continue the enumeration. You could return FALSE if you would look for specific symbols and wanted to speed up the processing.

The managed side of the story

When the tool is run on a managed assembly, you get the following kind of output:

The name of the method does not match at all with the names in my test assembly! Why do all methods have this generic Method.# format?

Well… the number corresponds to the RID of the corresponding method in the metadata of the assembly. Let’s have a look at what the MethodDef metadata table of this assembly looks like in ILSpy:

The RID column corresponds to the number in the name in the tool output. So, the Method.#3 is the get_Records property getter in PDBFormatReaderTest.cs:28. And this is exactly what I can see in the test source code:

You can also check that the lines look correct compared to what is listed by the tool:

FilePath getter at line 19
FilePath setter at line 20
Records getter at line 28
Records setter at line 29

Feel free to look at the source code from my Github repository.

DbgHelp provides many more services to look into symbols but that’s all for today!

How to dump function symbols from a .pdb file

Mon, 08 Dec 2025 10:12:20 +0000

During this R&D week at Datadog, I wanted to implement a tool accepting a .pdb file and generate a .sym file listing functions symbols with their address, size, name with signature and if they are public or private. This post dig into the implementation details of using Microsoft Debug Interface Access (DIA) COM API to achieve these objectives. If you want to see what my vibe coding experience in Cursor was, read this other post instead.

One self-contained tool please!

I would like the tool to be self-contained but since DIA is based on a COM server, it would require registering msdia40.dll on the machine. Not a good idea. In case the dll is in the same folder as the tool, one could “emulate” the magic done by CoCreateInstance to get an instance of IDiaDataSource (more on this interface soon) by:

Call LoadLibrary to load the dll in memory
Call GetProcAddress to get the DllGetClassObject implementation
Call this function to get the IClassFactory implementation
Call its CreateInstance method to get an object implementing IDiaDataSource

Here is the corresponding code (without error checking for readability)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


// Create DIA data source without registration using DLL loading
HRESULT PdbSymbolExtractor::NoRegCoCreate(const std::wstring& dllPath, REFCLSID rclsid, REFIID riid, void** ppv) {
    HMODULE hDll = LoadLibraryW(dllPath.c_str());

    typedef HRESULT(__stdcall* DllGetClassObjectFunc)(REFCLSID, REFIID, LPVOID*);
    DllGetClassObjectFunc pDllGetClassObject = (DllGetClassObjectFunc)GetProcAddress(hDll, "DllGetClassObject");

    CComPtr<IClassFactory> pClassFactory;
    HRESULT hr = pDllGetClassObject(rclsid, IID_IClassFactory, (void**)&pClassFactory);

    hr = pClassFactory->CreateInstance(NULL, riid, ppv);

    // Note: We intentionally don't call FreeLibrary here because the DLL needs to stay loaded
    // The COM object references will keep it alive
    return S_OK;

This function is called with the path of msdia40.dll and the UUID of the expected IDiaDataSource:

1
2


// Create DIA data source without registration
hr = NoRegCoCreate(dllPath, CLSID_DiaSource, __uuidof(IDiaDataSource), (void**)&_pDiaDataSource);

But still, I don’t want to have two binaries!

The trick is to embed the msdia40.dll inside the tool as a Windows resource. In the .rc file, add an RCDATA entry that points to the dll:

1

IDR_MSDIA_DLL      RCDATA      "x64\\Release\\msdia140.dll"

You should see it in the Resource View in Visual Studio:

Here is the code that extracts it as a file on disk is straightforward (error checking has been removed for readability):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


// Extract embedded msdia140.dll from resources
bool PdbSymbolExtractor::ExtractEmbeddedDll(const std::wstring& outputPath) {
    // Find the resource
    HMODULE hModule = GetModuleHandle(NULL);
    HRSRC hResource = FindResource(hModule, MAKEINTRESOURCE(IDR_MSDIA_DLL), RT_RCDATA);

    // Load the resource
    HGLOBAL hLoadedResource = LoadResource(hModule, hResource);

    // Lock the resource to get a pointer to the data
    LPVOID pResourceData = LockResource(hLoadedResource);

    // Get the size of the resource
    DWORD resourceSize = SizeofResource(hModule, hResource);

    // Write the DLL to disk
    std::ofstream outFile(outputPath, std::ios::binary);
    outFile.write(static_cast<const char*>(pResourceData), resourceSize);
    outFile.close();

    return true;
}

Where are my function symbols?

After calling NoRegCoCreate(), _pDiaDataSource stores a reference to the entry point into the DIA APIs. Here are the steps to follow before being able to list the symbols:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


HRESULT PdbSymbolExtractor::ExtractSymbolsFromPdb(const std::wstring& pdbPath, std::vector<FunctionSymbol>& symbols) 
{
    // Load the PDB file
    HRESULT hr = _pDiaDataSource->loadDataFromPdb(pdbPath.c_str());

    // Open a session
    CComPtr<IDiaSession> pSession;
    hr = _pDiaDataSource->openSession(&pSession);

    // Get the global scope
    CComPtr<IDiaSymbol> pGlobal;
    hr = pSession->get_globalScope(&pGlobal);

Now, you have the global scope of the symbols, you can ask for an enumerator for the type of symbols you are interested in; SymTagFunction in my case:

1
2
3
4
5
6
7


// Enumerate all function symbols
    CComPtr<IDiaEnumSymbols> pEnumSymbols;
    hr = pGlobal->findChildren(SymTagFunction, NULL, nsNone, &pEnumSymbols);

    LONG count = 0;
    pEnumSymbols->get_Count(&count);
    std::wcout << L"Found " << count << L" function symbols" << std::endl;

The pEnumSymbols iterator allows you to loop on each SymTagFunction symbol and get its name:

1
2
3
4
5
6
7
8
9


while (SUCCEEDED(pEnumSymbols->Next(1, &pSymbol, &celt)) && celt == 1) {
        FunctionSymbol func;

        // Get function name
        BSTR bstrName;
        if (pSymbol->get_name(&bstrName) == S_OK) {
            func.name = bstrName;
            SysFreeString(bstrName);
        }

Note that each symbol details are stored in a FunctionSymbol instance:

1
2
3
4
5
6
7
8


struct FunctionSymbol {
    std::wstring name;
    ...
    std::wstring signature; // Function signature (parameters only, no return type)
    DWORD rva;              // Relative Virtual Address
    ULONGLONG length;
    bool isPublic;
};

with the rest of the code in the while() loop:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


// Get function signature (parameters only, no return type)
    func.signature = ExtractFunctionSignature(pSymbol);

    // Get relative virtual address
    DWORD rva;
    if (pSymbol->get_relativeVirtualAddress(&rva) == S_OK) {
        func.rva = rva;
    }

    // Get function length
    ULONGLONG length;
    if (pSymbol->get_length(&length) == S_OK) {
        func.length = length;
    }

    // Determine if function is public or private
    // Check access level - default to private
    func.isPublic = false;
    DWORD access;
    if (pSymbol->get_access(&access) == S_OK) {
        func.isPublic = (access == CV_public);
    }

    pSymbol.Release();

I did not have the time to do more trial for private/public state, but I should have tried by enumerating SymTagPublicSymbol or SymTagExport that could be considered as public.

Better with a signature

The final step is to figure out the signature of each function. This is where the genericity of DIA could be confusing because so many things are represented by IDiaSymbol: a symbol, a function, the type of a function, or the type of a parameter…

So, the type of the function is retrieved as an IDiaSymbol by calling getType() on the function symbol. From that IDiaSymbol, findChildren() lets you iterate on the parameters:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


// Extract function signature (parameters only, no return type)
std::wstring PdbSymbolExtractor::ExtractFunctionSignature(IDiaSymbol* pSymbol) {
    if (!pSymbol) {
        return L"()";
    }

    // Get function type
    CComPtr<IDiaSymbol> pFunctionType;
    if (pSymbol->get_type(&pFunctionType) != S_OK || !pFunctionType) {
        return L"()";
    }

    // Enumerate function arguments
    CComPtr<IDiaEnumSymbols> pEnumArgs;
    if (FAILED(pFunctionType->findChildren(SymTagFunctionArgType, NULL, nsNone, &pEnumArgs))) {
        return L"()";
    }

    LONG argCount = 0;
    pEnumArgs->get_Count(&argCount);

    if (argCount == 0) {
        return L"()";
    }

Now, the same Next() method is called on the enumerator to iterate on each parameter:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


// Build signature string
    std::wstring signature = L"(";
    CComPtr<IDiaSymbol> pArg;
    ULONG argCelt = 0;
    bool first = true;

    while (SUCCEEDED(pEnumArgs->Next(1, &pArg, &argCelt)) && argCelt == 1) {
        if (!first) {
            signature += L", ";
        }
        first = false;

        // Get the argument type
        CComPtr<IDiaSymbol> pArgType;
        if (pArg->get_type(&pArgType) == S_OK && pArgType) {
            signature += GetTypeName(pArgType);
        } else {
            signature += L"?";
        }

        pArg.Release();
    }

    signature += L")";
    return signature;
}

The final step is to get the name of the type from the IDiaSymbol returned by get_type(). If it is a custom type, call get_name() like any other symbol. Otherwise, for basic types, call get_baseType() and get_length() as shown by the code below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55


std::wstring PdbSymbolExtractor::GetTypeName(IDiaSymbol* pType) {
    if (!pType) {
        return L"?";
    }

    // Try to get type name directly
    BSTR bstrTypeName;
    if (pType->get_name(&bstrTypeName) == S_OK && bstrTypeName && wcslen(bstrTypeName) > 0) {
        std::wstring typeName = bstrTypeName;
        SysFreeString(bstrTypeName);
        return typeName;
    }

    // For basic types or unnamed types, try getting basic type info
    DWORD baseType = 0;
    ULONGLONG length = 0;

    if (pType->get_baseType(&baseType) == S_OK) {
        pType->get_length(&length);

        // Map basic types to names
        switch (baseType) {
            case btVoid: return L"void";
            case btChar: return L"char";
            case btWChar: return L"wchar_t";
            case btBool: return L"bool";

            case btInt:
            case btLong:
                if (length == 1) return L"char";
                else if (length == 2) return L"short";
                else if (length == 4) return L"int";
                else if (length == 8) return L"__int64";
                else return L"int" + std::to_wstring(length * 8);

            case btUInt:
            case btULong:
                if (length == 1) return L"unsigned char";
                else if (length == 2) return L"unsigned short";
                else if (length == 4) return L"unsigned int";
                else if (length == 8) return L"unsigned __int64";
                else return L"uint" + std::to_wstring(length * 8);

            case btFloat:
                if (length == 4) return L"float";
                else if (length == 8) return L"double";
                else return L"float" + std::to_wstring(length * 8);

            default:
                return L"?";
        }
    }

    return L"?";
}

This is a “simple” implementation that does not take pointers, addresses, arrays, and more into account. For a more complete solution, I would recommend looking at the PrintType() implementation in the DIA2Dump code sample that is installed with Visual Studio.

I hope this will get your foot in the door of symbol parsing and make you want to dig further into DIA.

References

Corresponding source code is available in my github repository.
Archived Microsoft documentation/implementation of .pdb format including a symbol dumper code.
DIA2Dump Visual Studio code sample.

Vibe coding a .pdb dumper or how I became a Product Manager

Mon, 08 Dec 2025 10:12:16 +0000

One self-contained tool please!

Call LoadLibrary to load the dll in memory
Call GetProcAddress to get the DllGetClassObject implementation
Call this function to get the IClassFactory implementation
Call its CreateInstance method to get an object implementing IDiaDataSource

Here is the corresponding code (without error checking for readability)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


// Create DIA data source without registration using DLL loading
HRESULT PdbSymbolExtractor::NoRegCoCreate(const std::wstring& dllPath, REFCLSID rclsid, REFIID riid, void** ppv) {
    HMODULE hDll = LoadLibraryW(dllPath.c_str());

    typedef HRESULT(__stdcall* DllGetClassObjectFunc)(REFCLSID, REFIID, LPVOID*);
    DllGetClassObjectFunc pDllGetClassObject = (DllGetClassObjectFunc)GetProcAddress(hDll, "DllGetClassObject");

    CComPtr<IClassFactory> pClassFactory;
    HRESULT hr = pDllGetClassObject(rclsid, IID_IClassFactory, (void**)&pClassFactory);

    hr = pClassFactory->CreateInstance(NULL, riid, ppv);

    // Note: We intentionally don't call FreeLibrary here because the DLL needs to stay loaded
    // The COM object references will keep it alive
    return S_OK;

This function is called with the path of msdia40.dll and the UUID of the expected IDiaDataSource:

1
2


// Create DIA data source without registration
hr = NoRegCoCreate(dllPath, CLSID_DiaSource, __uuidof(IDiaDataSource), (void**)&_pDiaDataSource);

But still, I don’t want to have two binaries!

The trick is to embed the msdia40.dll inside the tool as a Windows resource. In the .rc file, add an RCDATA entry that points to the dll:

1

IDR_MSDIA_DLL      RCDATA      "x64\\Release\\msdia140.dll"

You should see it in the Resource View in Visual Studio:

Here is the code that extracts it as a file on disk is straightforward (error checking has been removed for readability):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


// Extract embedded msdia140.dll from resources
bool PdbSymbolExtractor::ExtractEmbeddedDll(const std::wstring& outputPath) {
    // Find the resource
    HMODULE hModule = GetModuleHandle(NULL);
    HRSRC hResource = FindResource(hModule, MAKEINTRESOURCE(IDR_MSDIA_DLL), RT_RCDATA);

    // Load the resource
    HGLOBAL hLoadedResource = LoadResource(hModule, hResource);

    // Lock the resource to get a pointer to the data
    LPVOID pResourceData = LockResource(hLoadedResource);

    // Get the size of the resource
    DWORD resourceSize = SizeofResource(hModule, hResource);

    // Write the DLL to disk
    std::ofstream outFile(outputPath, std::ios::binary);
    outFile.write(static_cast<const char*>(pResourceData), resourceSize);
    outFile.close();

    return true;
}

Where are my function symbols?

After calling NoRegCoCreate(), _pDiaDataSource stores a reference to the entry point into the DIA APIs. Here are the steps to follow before being able to list the symbols:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


HRESULT PdbSymbolExtractor::ExtractSymbolsFromPdb(const std::wstring& pdbPath, std::vector<FunctionSymbol>& symbols) 
{
    // Load the PDB file
    HRESULT hr = _pDiaDataSource->loadDataFromPdb(pdbPath.c_str());

    // Open a session
    CComPtr<IDiaSession> pSession;
    hr = _pDiaDataSource->openSession(&pSession);

    // Get the global scope
    CComPtr<IDiaSymbol> pGlobal;
    hr = pSession->get_globalScope(&pGlobal);

Now, you have the global scope of the symbols, you can ask for an enumerator for the type of symbols you are interested in; SymTagFunction in my case:

1
2
3
4
5
6
7


// Enumerate all function symbols
    CComPtr<IDiaEnumSymbols> pEnumSymbols;
    hr = pGlobal->findChildren(SymTagFunction, NULL, nsNone, &pEnumSymbols);

    LONG count = 0;
    pEnumSymbols->get_Count(&count);
    std::wcout << L"Found " << count << L" function symbols" << std::endl;

The pEnumSymbols iterator allows you to loop on each SymTagFunction symbol and get its name:

1
2
3
4
5
6
7
8
9


while (SUCCEEDED(pEnumSymbols->Next(1, &pSymbol, &celt)) && celt == 1) {
        FunctionSymbol func;

        // Get function name
        BSTR bstrName;
        if (pSymbol->get_name(&bstrName) == S_OK) {
            func.name = bstrName;
            SysFreeString(bstrName);
        }

Note that each symbol details are stored in a FunctionSymbol instance:

1
2
3
4
5
6
7
8


struct FunctionSymbol {
    std::wstring name;
    ...
    std::wstring signature; // Function signature (parameters only, no return type)
    DWORD rva;              // Relative Virtual Address
    ULONGLONG length;
    bool isPublic;
};

with the rest of the code in the while() loop:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


// Get function signature (parameters only, no return type)
    func.signature = ExtractFunctionSignature(pSymbol);

    // Get relative virtual address
    DWORD rva;
    if (pSymbol->get_relativeVirtualAddress(&rva) == S_OK) {
        func.rva = rva;
    }

    // Get function length
    ULONGLONG length;
    if (pSymbol->get_length(&length) == S_OK) {
        func.length = length;
    }

    // Determine if function is public or private
    // Check access level - default to private
    func.isPublic = false;
    DWORD access;
    if (pSymbol->get_access(&access) == S_OK) {
        func.isPublic = (access == CV_public);
    }

    pSymbol.Release();

I did not have the time to do more trial for private/public state, but I should have tried by enumerating SymTagPublicSymbol or SymTagExport that could be considered as public.

Better with a signature

So, the type of the function is retrieved as an IDiaSymbol by calling getType() on the function symbol. From that IDiaSymbol, findChildren() lets you iterate on the parameters:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


// Extract function signature (parameters only, no return type)
std::wstring PdbSymbolExtractor::ExtractFunctionSignature(IDiaSymbol* pSymbol) {
    if (!pSymbol) {
        return L"()";
    }

    // Get function type
    CComPtr<IDiaSymbol> pFunctionType;
    if (pSymbol->get_type(&pFunctionType) != S_OK || !pFunctionType) {
        return L"()";
    }

    // Enumerate function arguments
    CComPtr<IDiaEnumSymbols> pEnumArgs;
    if (FAILED(pFunctionType->findChildren(SymTagFunctionArgType, NULL, nsNone, &pEnumArgs))) {
        return L"()";
    }

    LONG argCount = 0;
    pEnumArgs->get_Count(&argCount);

    if (argCount == 0) {
        return L"()";
    }

Now, the same Next() method is called on the enumerator to iterate on each parameter:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


// Build signature string
    std::wstring signature = L"(";
    CComPtr<IDiaSymbol> pArg;
    ULONG argCelt = 0;
    bool first = true;

    while (SUCCEEDED(pEnumArgs->Next(1, &pArg, &argCelt)) && argCelt == 1) {
        if (!first) {
            signature += L", ";
        }
        first = false;

        // Get the argument type
        CComPtr<IDiaSymbol> pArgType;
        if (pArg->get_type(&pArgType) == S_OK && pArgType) {
            signature += GetTypeName(pArgType);
        } else {
            signature += L"?";
        }

        pArg.Release();
    }

    signature += L")";
    return signature;
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55


std::wstring PdbSymbolExtractor::GetTypeName(IDiaSymbol* pType) {
    if (!pType) {
        return L"?";
    }

    // Try to get type name directly
    BSTR bstrTypeName;
    if (pType->get_name(&bstrTypeName) == S_OK && bstrTypeName && wcslen(bstrTypeName) > 0) {
        std::wstring typeName = bstrTypeName;
        SysFreeString(bstrTypeName);
        return typeName;
    }

    // For basic types or unnamed types, try getting basic type info
    DWORD baseType = 0;
    ULONGLONG length = 0;

    if (pType->get_baseType(&baseType) == S_OK) {
        pType->get_length(&length);

        // Map basic types to names
        switch (baseType) {
            case btVoid: return L"void";
            case btChar: return L"char";
            case btWChar: return L"wchar_t";
            case btBool: return L"bool";

            case btInt:
            case btLong:
                if (length == 1) return L"char";
                else if (length == 2) return L"short";
                else if (length == 4) return L"int";
                else if (length == 8) return L"__int64";
                else return L"int" + std::to_wstring(length * 8);

            case btUInt:
            case btULong:
                if (length == 1) return L"unsigned char";
                else if (length == 2) return L"unsigned short";
                else if (length == 4) return L"unsigned int";
                else if (length == 8) return L"unsigned __int64";
                else return L"uint" + std::to_wstring(length * 8);

            case btFloat:
                if (length == 4) return L"float";
                else if (length == 8) return L"double";
                else return L"float" + std::to_wstring(length * 8);

            default:
                return L"?";
        }
    }

    return L"?";
}

I hope this will get your foot in the door of symbol parsing and make you want to dig further into DIA.

References

Corresponding source code is available in my github repository.
Archived Microsoft documentation/implementation of .pdb format including a symbol dumper code.
DIA2Dump Visual Studio code sample.

Trigger your GCs with dotnet-fullgc!

Wed, 22 May 2024 18:03:17 +0000

Introduction

If you have read Microsoft documentation, you probably know that it is not recommended to trigger a garbage collection in your application code. However, in some troubleshooting cases, you might want to trigger a GC. For example, you don’t want to wait for a full gen2 compacting GC to figure out if your application is really leaking memory. For web applications, you can imagine having a hidden HTTP end point that simply call GC.Collect. What if you could simply call a command line tool to trigger a GC in any .NET application? This is exactly what my new dotnet-fullgc CLI tool is doing!

This Is The Way

If you have read a few of my past posts about the .NET diagnostics mechanisms, you know that you can send commands to another .NET process via EventPipes. Well… there is no explicit command to trigger a GC.

I have also explained how you could listen to events emitted by the CLR by enabling a provider with a set of keywords with a verbosity corresponding to the events you are interested in. This is how dotnet-trace and Perfview are collecting these events. If you want to trigger a GC, you simply need to enable the Microsoft-Windows-DotNETRuntime provider with the GCHeapCollect (= 0x800000L) keyword and an informal verbosity. Yes: it is as simple as that, and it is also working for .NET Framework!

So, you could trigger a GC with dotnet-trace via the following command line:

1

dotnet trace collect -p  - providers Microsoft-Windows-DotNETRuntime:0x800000:4 - duration 00:00:01

However, a .nettrace file would be generated and it is not possible to pass parameters (more on this later).

The rest of this post shows the C# code to obtain the same result. With Microsoft.Diagnostics.NETCore.Client and TraceEvent, you create a DiagnosticsClient with the ID of the process you are interested in:

1

var client = new DiagnosticsClient(processId);

The next step is to start an EventPipe session with the right provider, keyword and verbosity:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


var providers = new List<EventPipeProvider>()
{
    new EventPipeProvider(
        "Microsoft-Windows-DotNETRuntime",
        EventLevel.Informational,
        (long)ClrTraceEventParser.Keywords.GCHeapCollect,
        Arguments  // more on this later
        ),
};
using (var session = client.StartEventPipeSession(providers, false))
{

The source must be processed in another thread to avoid blocking the main thread:

1
2
3
4
5
6
7


Task streamTask = Task.Run(() =>
    {
        // without source to process, session.Stop() will not return
        var source = new EventPipeEventSource(session.EventStream);

        source.Process();
    });

The question to answer is how to stop the session: just create another task that waits for a second before stopping the session to exit from the Process call:

1
2
3
4
5
6
7
8


Task inputTask = Task.Run(() =>
    {
        Thread.Sleep(1000);
        session.Stop();
    });

    Task.WaitAny(streamTask, inputTask);
}

That’s all!

Pitfalls

Unfortunately, I faced a couple of issue during the implementation of the tool.

What is your number?

If you look at the TraceEvent implementation, you could imagine that is it possible to pass a GC ID as a parameter to the .NET provider:

1
2
3
4
5


//
// Summary:
//     Triggers a GC. Can pass a 64 bit value that will be logged with the GC Start
//     event so you know which GC you actually triggered.
GCHeapCollect = 0x800000L,

Unfortunately, this is not supported and the reasons are explained below.

If you check the .NET source code and look at EtwCallbackCommon(), you can indeed see that a numeric ID can be passed to ETW::GCLog::ForceGC(l64ClientSequenceNumber) and passed as ID in GCStart event instead of the one incrementally increased collection after collection.

At the diagnostics client level, you have the opportunity to pass a dictionary of key/value string pairs. This dictionary is used when defining the EventPipeProvider to be enabled:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


Dictionary<string, string> arguments = new Dictionary<string, string>();
arguments.Add("Id", "42");
var providers = new List<EventPipeProvider>()
{
    new EventPipeProvider(
        "Microsoft-Windows-DotNETRuntime",
        EventLevel.Informational,
        (long)ClrTraceEventParser.Keywords.GCHeapCollect,
        arguments
        ),
};

The diagnostics client transforms the dictionary into a string with key=value pairs separated by ‘;’ such as “Id=42;AnotherId=AnotherValue;…” and serializes it as is in the payload when sending a CollectTraces command.

The question is what identifier is expected by the CLR? To answer this question, you need to look at the code of provider_invoke_callback. The “key=value;…” string is stored into a buffer and each = and ; characters are transformed into \0. So, “Id=42” is transformed into Id\042\0.

The next step is done by ep_event_filter_desc_init() that uses that buffer to fill up the 3 fields of an EventFilterDescriptor:

1
2
3


uint64_t ptr = address of the buffer
uint32_t size = size of the buffer (=6 for id\042\0)
uint32_t type = 0

And finally, EtwCallbackCommon receives the filterdata and tries the following to get the collection id:

1
2
3
4
5
6
7


PEVENT_FILTER_DESCRIPTOR FilterData = (PEVENT_FILTER_DESCRIPTOR)pFilterData;
if ((FilterData != NULL) &&
   (FilterData->Type == 1) &&
   (FilterData->Size == sizeof(l64ClientSequenceNumber)))
{
    l64ClientSequenceNumber = *(LONGLONG *) (FilterData->Ptr);
}

As you can see, it will fail because:

the received type value is 0
the received size is not the size of a 64 bit number
the ptr field does not point to the value as 64 bit number (but as a string)

An issue has been filed with a possible fix.

Only one collection please!

When I test my dotnet-fullgc with dotnet-gcstats, I always see 2 collections!

We investigated with my colleague Kevin Gosse and he created an issue for that. The EtwCallback() function is called whenever a session is enabled or disabled. Unfortunately, the call to ForceGC is made in both cases: so, when the session is stopped, a second garbage collection is triggered.

Next steps

Feel free to install dotnet-fullgc on your machine with dotnet tool install -g dotnet-fullgc.

Next, use dotnet fullgc to trigger two gen2 full garbage collections in your running .NET processes.

View your GCs statistics live with dotnet-gcstats!

Fri, 01 Mar 2024 18:30:13 +0000

Introduction

While working on the second edition of Pro .NET Memory Management, it was needed to get statistics about each garbage collection to explain the condemned generation and other decisions taken by the GC. This post explains the different internal data structures used by the GC and how to get their value for each collection. Some require debugging the CLR and others are emitted via events. For the latter, I will show how I wrote the new dotnet-gcstats CLI tool to collect them and a personal Perfview GCStats displaying live data, garbage collection after garbage collection.

High level view of GC Internals

With regions, the GC keeps track of managed memory allocated by your application in instances of the gc_heap class. In Workstation mode, only 1 instance exists and in Server mode, by default, 1 instance is created per core. Each gc_heap keeps track of its 5 generations (gen0, gen1, gen2, Large Object Heap and Pinned Object Heap) in an array of 5 generation instances. Each generation references its dedicated regions wrapped by instances of heap_segment. These regions are reserved from a giant part of the process address space and committed as needed.

During a garbage collection, the GC code relies on global fields per gc_heap:

1
2
3
4


static gc_mechanisms settings;
gc_history_global gc_data_global;  // for non background GC including foreground GC during a background
gc_history_global bgc_data_global; // for background GC only
static dynamic_data dynamic_data_table[total_generation_count = 5];

The settings field contains a few interesting fields:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


class gc_mechanisms
{
public:
    gc_index; // starts from 1 for the first GC
    int condemned_generation;  // generation to collect
    BOOL compaction;  // true when compaction instead of sweep
    BOOL loh_compaction;  // true when LOH needs compaction
    uint32_t concurrent;  // 1 = concurrent/background GC 
    gc_reason reason;  // trigger reason
    gc_pause_mode pause_mode;  // see GCSettings.LatencyMode
    ...
};

including the trigger reason that will tell if your code called GC.Collect (i.e. induced), or if it was due to a LOH or SOH allocation for example. If compaction is true, a compacting GC will happen (instead of a sweeping one).

The gc/bgc_data_global contains almost the same information:

1
2
3
4
5
6
7
8
9


struct gc_history_global
{
    uint32_t num_heaps;  // number of gc_heap instances
    int condemned_generation;
    gc_reason reason;
    int pause_mode;
    uint32_t mem_pressure; 
    uint32_t global_mechanisms_p;
};

Most of the fields are available from different events:

GCStart: gc_index in Count, condemned_generation in Depth, reason in Reason
GCGlobalHeapHistory: pause_mode in PauseMode and some others in GlobalMechanisms

Which generation to collect = condemned generation

The computation of the condemned_generation is complicated and relies on many factors including metrics stored for each “generation” (gen0, gen1, gen2, LOH and POH) in an array of dynamic_data called dynamic_data_table. The dynamic_data class contains a few fields used by the GC to take decisions such as when a collection should be triggered and which generation to condemn:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


class dynamic_data
{
public:
    ptrdiff_t new_allocation;     // remaining budget = budget - allocated
    size_t    desired_allocation; // budget to trigger a GC

    // # of bytes taken by survived objects after mark.
    size_t    survived_size;

    // # of bytes taken by survived pinned plugs after mark.
    size_t    pinned_survived_size;

    // total object size after a GC, ie, doesn't include fragmentation
    size_t    current_size;
    size_t    promoted_size;
    size_t    fragmentation;
};

Most of these fields are found in the payload of GCPerHeapHistory or GCHeapStat events. However, the most interesting one, new_allocation is not available. Why is it interesting? Because it would give you which generation had its budget exceeded. It is initialized with the generation budget at the end of a GC and then, each time an allocation context gets created, its size is deducted from it. When it reaches 0, it means that the budget is exceeded, and a collection should happen.

Since I needed to debug the CLR to better understand all these algorithms, I added a breakpoint at the beginning of gc_heap::garbage_collect with the following action:

#{settings.gc_index}[{gc_trigger_reason}]{"\n",s8b} new_allocation(0) = {dynamic_data_table[0].new_allocation}{"\n",s8b} desired_allocation(0) = {dynamic_data_table[0].desired_allocation}{"\n",s8b} begin_data_size(0) = {dynamic_data_table[0].begin_data_size}{"\n",s8b} promoted_size(0) = {dynamic_data_table[0].promoted_size}{"\n",s8b}-{"\n",s8b} new_allocation(1) = 
...
{dynamic_data_table[4].new_allocation}{"\n",s8b} desired_allocation(4) = {dynamic_data_table[4].desired_allocation}{"\n",s8b} begin_data_size(4) = {dynamic_data_table[4].begin_data_size}{"\n",s8b} promoted_size(4) = {dynamic_data_table[4].promoted_size}{"\n",s8b}__________{"\n",s8b}

And now, each time a GC happens, I get the corresponding log in my Output pane in Visual Studio:

#2[reason_alloc_soh (0)]
 new_allocation(0) = -22728
 desired_allocation(0) = 134217728
 begin_data_size(0) = 8391376
 promoted_size(0) = 8383432
-
 new_allocation(1) = -5910416
 desired_allocation(1) = 2473016
 begin_data_size(1) = 375528
 promoted_size(1) = 353288
-
 new_allocation(2) = -91144
 desired_allocation(2) = 262144
 begin_data_size(2) = 0
 promoted_size(2) = 0
-
 new_allocation(3) = 28000088
 desired_allocation(3) = 28000088
 begin_data_size(3) = 8000024
 promoted_size(3) = 8000024
-
 new_allocation(4) = 3145728
 desired_allocation(4) = 3145728
 begin_data_size(4) = 32712
 promoted_size(4) = 32712

As you can see, gen0, gen1 and gen2 have all their budget exceeded (i.e. their new_allocation is negative) and it explains why a simple gen0 collection (from allocation in SOH = gen0) becomes a gen2 collection. If you wonder how gen1 and gen2 budgets are exceeded as your application is only allocating in gen0, you need to understand that when a GC copy surviving objects from one younger generation to the older, they are counted as allocations in the older and subtracted from its new_allocation metric.

The GC is encoding the different steps leading to the final condemned generation in a 32 bit value stored in a gen_to_condemn_tuning field that allows you to get:

initial condemned generation,
final generation to condemn,
which generation’s budget is exceeded.

The value of the last one corresponds to the highest generation for which its new_allocation was negative.

This information is available in the CondemnReasons0 field of the GCPerHeapHistory event, and you need some arithmetic to get the generation you want:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


private const int gen_initial = 0;          // indicates the initial gen to condemn.
private const int gen_final_per_heap = 1;   // indicates the final gen to condemn per heap.
private const int gen_alloc_budget = 2;     // indicates which gen's budget is exceeded.

private const int InitialGenMask = 0x0 + 0x1 + 0x2;

static int GetGen(int val, int reason)
{
    int gen = (val >> 2 * reason) & InitialGenMask;
    return gen;
}

Building your own tool

Even though I could dig into the different matrices available in the Perfview’s GCStats view or its export to Excel, I decided to write dotnet-gcstats. This CLI tool listens to the CLR events emitted by a .NET application thanks to Microsoft.Diagnostics.NETCore.Client (connect to the application EventPipe) and TraceEvent (receive and analyze the CLR events).

The code is amazingly simple:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


var providers = new List<EventPipeProvider>()
{
    new EventPipeProvider("Microsoft-Windows-DotNETRuntime",
        EventLevel.Informational, (long)ClrTraceEventParser.Keywords.GC),
};
var client = new DiagnosticsClient(processId);

using (var session = client.StartEventPipeSession(providers, false))
{
    Console.WriteLine();

    Task streamTask = Task.Run(() =>
    {
        var source = new EventPipeEventSource(session.EventStream);

        ClrTraceEventParser clrParser = new ClrTraceEventParser(source);
        clrParser.GCPerHeapHistory += OnGCPerHeapHistory;
        clrParser.GCStart += OnGCStart;
        clrParser.GCGlobalHeapHistory += OnGCGlobalHeapHistory;

        try
        {
            source.Process();
        }
        catch (Exception e)
        {
            ShowError($"Error encountered while processing events: {e.Message}");
        }
    });

Each event handler is responsible for extracting and translating the interesting fields of its event payload with a few color enhancements:

GCStart: collection count and reason (highlight induced collections).
GCGlobalHeapHistory: condemned generation, pause mode and memory pressure.
GCPerHeapHistory: starting -> final condemned generation and for each heap, budget, begin size, begin obj size, final size, promoted size and fragmentation.

The final step was to transform a simple console application into a .NET CLI tool that everyone will be able to install with dotnet tool install -g dotnet-gcstats and use with dotnet gcstats . I followed the documentation by adding the following to the project file:

1
2
3
4
5
6



    true
    dotnet-gcstats
    ./nupkg
    true
  

In addition, I provided a few additional details:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15



    dotnet-gcstats
    1.0.0
    </span>dotnet-gcstats<span class="nt">
    christophe Nasarre
    chrisnas
    https://github.com/chrisnas
    git
    https://github.com/chrisnas/GCStats
    LICENSE
    Global CLI tool to display live statistics during .NET garbage collections
    Initial release
    Copyright Christophe Nasarre 2024-$([System.DateTime]::UtcNow.ToString(yyyy))
    .NET TraceEvent CLR GC
  

Once built, I simply uploaded the generated package to nuget.org et voila!

Now, you should be able to better understand why some collections are triggered:

And if it is not enough, wait for reading the second edition of Pro .NET Memory Management ;^)

Be Aligned! Or how to investigate a stack corruption

Mon, 11 Dec 2023 11:01:45 +0000

Introduction

During the Datadog R&D week, my goal is to mimic the generation of a .gcdump from our .NET profiler. I’ve already written most of the code for a previous post and after changing the required plumbing, it is time to test the workflow.

Unfortunately, I’m facing the dreaded stack corruption dialog:

The rest of the post explains the different steps I’m following to investigate this issue.

Trying to understand the problem

This stack check is done by the debug version of the C Runtime library by basically adding some special bytes on the stack before calling a function and checking these bytes are not tampered when returning from the call.

So, the next step is to debug the application to get more details and at least where in the code the problem happened:

The failed check occurs at the end of a function that looks like the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


bool GcDumpProvider::Get(IGcDumpProvider::gcdump_t& gcDump)
{
    // trigger the GC and get the dump
    GcDump gcd(::GetCurrentProcessId());
    gcd.TriggerDump();

    auto const& dump = gcd.GetGcDumpState();
    auto& types = dump._types;
    for (auto& type : types)
    {
        auto& typeInfo = type.second;

        uint64_t instancesCount = typeInfo._instances.size();
        uint64_t instancesSize = 0;
        for (size_t i = 0; i < instancesCount; i++)
        {
            instancesSize += typeInfo._instances[i]._size;
        }

        gcDump.push_back({typeInfo._name, instancesCount, instancesSize});
    }
    return true;
}

There is much more code behind this; especially in the TriggerDump() function. I have already tested this code many times when I dug into the .gcdump generation process without facing this stack corruption. I’m spending a few hours digging back into the code because:

I’m not running inside the profiled process and not outside like in the blog post
I’m introducing a “slight” change because I need to exit the communication with the CLR when the GC ends.
I need to mention that Visual Studio is refusing to debug (Step Over or Step Into) and only Run to Cursor was possible due to mixed mode (managed and native) debugging. So, I created a simple native console application with my updated code for easier and faster debugging.

After a couple of hours, it is time to go back to the Get() implementation because I do not find anything obviously wrong.

Make it simpler and simpler and simpler again

In that type of situation, I recommend the “remove code and debug” strategy (from “divide and conquer” attributed to Julius Cesar). From the simplified console application, the code now looks like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


bool GetGcDump(int pid, IGcDumpProvider::gcdump_t& gcDump)
{
    GcDump gcd(pid);

    // no more gcd.TriggerDump()

    // no more for (auto& type : types)

    return true;
}

No more complicated call nor iteration on the vector of results. Guess what? Same stack corruption.

It is time to go one level deeper: what does this GcDump class look like?

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


class GcDump
{
public:
    GcDump(int pid);
    ~GcDump();

...

private:
    int _pid;
    DiagnosticsClient* _pClient;
    EventPipeSession* _pSession;
    HANDLE _hListenerThread;
    GcDumpState _gcDumpState;
};

The constructor is setting the fields value to zero/nullptr and the destructor is cleaning up these fields if necessary. Since TriggerDump() is no more called, these fields never change.

I’m commenting out all fields until only _gcDumpState remains and it continues to crash. When is it commented out, it is not more crashing.

Use your debugger Luke!

Let’s turn to the GcDumpState class that is even simpler:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


class GcDumpState
{
public:
    GcDumpState();
    ~GcDumpState();

public:
    // fields removed for brevity

private:
    bool _isStarted;
    bool _hasEnded;
    uint32_t _collectionIndex;
};

The code of the destructor only sends a trace to the console (removing it completely does not fix the issue) and here is the constructor code:

1
2
3
4
5
6


GcDumpState::GcDumpState()
{
    _isStarted = false;
    _hasEnded = false;
    _collectionIndex = 0;
}

Again, same strategy: remove one field after the other. This time, the code stops crashing if the two Boolean fields are removed or if the 32 bits index is removed. If the index field is not set, no more corruption!

How could this assignation corrupt the stack? It is time to use the Visual Studio debugger to better understand what is going on.

First, set a breakpoint on the assignment line and click Debug | Disassembly to see the corresponding assembly code:

The two lines of assembly code are easy to understand:

1
2


mov rax,qword ptr [this]  
mov dword ptr [rax+4],0

The this pointer is stored in the rax register
The 32 bits (mov dword) memory starting 4 bytes after the beginning of the object pointed to by this, is set to 0

I enter “this” in a Memory panel (Debug | Windows | Memory xx)

And Visual Studio gives me the corresponding address and the content of the memory there:

Pressing F10 twice to Step Over the two assembly instructions and this is confirmed:

Instead of storing the 32 bits 0 value just after the two bytes corresponding to the bool fields, it is stored 2 bytes away. It looks like a padding is added on my behalf.

I change the build settings for the GcDumpState.cpp file to enable all warnings:

The compilation confirms what has been seen in the memory:

1

GcDumpState.h(41,14): warning C4820: 'GcDumpState': '2' bytes padding added after data member 'GcDumpState::_hasEnded'

What’s next?

My understanding is that the compiler is:

adding a 2 bytes padding to align the 32 bits index field
generating the constructor code based on that padding
but the stack corruption checking code does not take it into account

The solution is to either add a uint16_t field after the 2 bool fields as an explicit padding or use #pragma pack(1) to decorate the class definition.

However, this looks really weird to me. We should have faced this issue a long time ago because we were never cautious about alignment in all the classes and structures that we allocate in our code. To validate the assumption, I’m writing a small reproduction code outside of all the .gcdump complexity. And guess what? I’m not able to reproduce the stack corruption. Another mystery of the C++ compilation optimizations probably…

This is the end of my debugging Friday at Datadog :^)

How to dig into the CLR

Sun, 12 Nov 2023 09:12:28 +0000

Introduction

When I started to work on the second edition of Pro .NET Memory Management : For Better Code, Performance, and Scalability by Konrad Kokosa, I already spent some time in the CLR code for a couple of pull requests related to the garbage collector. However, updating the book to cover 5 new versions of .NET requires looking at new APIs but also digging deep inside the CLR (and especially the GC) hundreds of thousand lines of code!

The first step is to install Visual Studio 2022 Preview that allows you to compile and run projects targeting .NET 8. Then, goto https://github.com/dotnet/runtime and git clone the tag of the .NET 8 preview version you have installed.

That way, you will be able to directly run the same version that you will debug.

And now, what are the next steps?

The goal of this post is to share with you the tips and tricks I used to navigate into the CLR implementation so you could better understand how things are working.

From C# to C++

As a .NET developer, I’m used to the APIs provided by the Base Class Library built on top of the CLR. Let’s take as an example the following code that is using the GC.AllocateArray method that allows you to allocate a pinned in memory array and available since .NET 5.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


using System;

internal class Program
{
    static void Main(string[] args)
    {
        byte[] pinned = GC.AllocateArray<byte>(90000, true);
        Console.WriteLine($"generation = {GC.GetGeneration(pinned)}");
    }
}

When you Ctrl+click the method name (or use F12), thanks to Source Link integration, you go to its implementation where you can even set breakpoint:

If you don’t use Visual Studio, you could open the generated assembly into a decompiler such as ILSpy or DnSpy. The latter even allows you to set breakpoints and debug the disassembly IL without any source.

In both cases, only the managed implementation will be available: you soon end up to an “internal call” corresponding to a native function implemented by the CLR. The managed methods are decorated with the MethodImplOptions.InternalCall attribute.

For the garbage collector code, you can look into the GC.CoreCLR.cs file where these methods are defined. You can note some methods decorated with the DllImport attribute to bind to native functions exported by a “QCall” library. There is an optimized path in P/Invoke done by the JIT to transform these calls not like a usual LoadLibrary/GetProcAddress as you could expect. Instead, they will be routed to the exported methods by coredll.dll and defined in the s_QCall array in qcallentrypoints.cpp. But where to look further for the native implementation?

Instead of searching among the thousands of files, focus on comutilnative.h that defines the signature of most exported functions. The implementation of the exported native functions is found in comutilnative.cpp. This is where you should start your journey in the native implementation of the CLR. For the list of all functions called by the libraries in the runtime, look at the ecalllist.h file (around gGCInterfaceFuncs and gGCSettingsFuncs specifically for the GC).

Note that you might also find some implementations under the classlibnative folder like in the system.cpp file for GCSettings.IsServerGC.

CLR Source code debugging

It is nice to know that the implementation of most CLR exported native functions used by the BCL is in comutilnative.cpp. For the GC, the functions are either statics from the GCInterface class or static functions prefixed by GCInterface_; I don’t know why all are not part of GCInterface…

When you look at the GC-related methods implementation, a lot are calling methods from the instance returned by GCHeapUtilities::GetGCHeap() that corresponds to the static g_pGCHeap global variable. It is interesting to follow the threads of calls like that, but I have to admit that, after a few hops, I’m starting to get lost. So, I’m drawing boxes for types on a piece of paper and arrays from their fields to other types as boxes.

However, with a code base that big, I definitively prefer to set breakpoints and write a small C# application to call the methods I’m interested in and see what data structures are used in the different layers of implementation. Don’t be scared: WinDBG is not required to achieve this goal. As this page explains, you need to type the following commands in a shell at the root of the repo:

.\build.cmd -s clr -c Debug .\build.cmd clr.nativeprereqs -a x64 -c debug .\build.cmd -msbuild

The last command generates a CoreCLR.sln solution file in artifacts\obj\coreclr\windows.x64.Debug\ide) that you can open in Visual Studio 2022 Preview.

In VS, right-click the INSTALL project, select Properties and setup the Debugging properties

Here are the details of each property:

It could be interesting to set some environment variables such as DOTNET_gcServer to 1 for a GC Server configuration instead of workstation. In that case, click the choice in the combo-box:

And update the textbox at the top:

The final step is to set this project as the startup project:

You are now able to set the breakpoint you want in the native code of the CLR and type F5/Debug in Visual Studio to step into the code!

And what about the assembly code?

Some specific data structures, such as the NonGC Heap, are used by the JIT compiler when generating the assembly code from the IL compiled from your C# code. It means that you need to look at that JITted code to fully understand what is going on.

A first way to get it is to use https://sharplab.io/, type your C# code and select x64 for Core of x86/x64 for Framework:

But as you can see from this screenshot, it is using the .NET 7 compiler. What if you would like to see the .NET 8 compilation result just in case something changed?

The solution I’m using is to generate a memory dump with procdump -ma of a test application. Before opening the dump in WinDBG, there is something you should be aware of: with the tiered compilation, you will need to call a method several times before the final optimized assembly code gets JITed. Or… decorate the method you are interested in with the [MethodImpl(MethodImplOptions.AggressiveOptimization)] attribute to instruct the JIT to directly generate the most optimized tier.

Once the dump loaded in WinDBG, the first step is to get the MethodTable pointer corresponding to the method you are interested in. For that, use the name2ee SOS command:

Click the link corresponding to MethodDesc to run the dumpmd SOS command:

The last step is to click the link corresponding to CodeAddr to run the U command and see the JITted assembly code:

If you compare this code to get the “Hello, World!” string, with the one shown by sharplab,

1
2
3
4


Program.Hello()
    L0000: mov rcx, 0x257f7cbc368
    L000a: mov rcx, [rcx]
    L000d: jmp qword ptr [0x7ff9c9bd7f48]

you might notice a tiny difference: there is one less indirection in .NET 8! But this is another story that will be told in the second edition of the “Pro .NET Memory Management: For Better Code, Performance, and Scalability” book ;^)

Crap: the application is randomly crashing!

Mon, 02 Oct 2023 09:02:26 +0000

Introduction

When you have a call with a customer who explains to you that his application is crashing when your profiler is enabled, it is never a great experience. This post is listing which steps were followed to investigate such an issue I faced last week; from the basics up to the final in analysing memory dumps in WinDbg.

Get as many setup details as possible

The situation was the following:

A web application was running fine with our Datadog .NET profiler on some non-production servers with less traffic.
The same application was crashing on production servers with more traffic.

We were lucky to be able to remote access both machines. A lot of time was spent to check the setup that is based on environment variables. Basically, for our profiler to be loaded by a .NET application, a few Microsoft related environment variables need to be set. Then, you enable the Datadog profiler by setting DD_PROFILING_ENABLED to 1 in order to get the profiling details available in our UI. Since the web application is running in IIS, things get more complicated because some environments variables must be set in… the Registry.

So, we checked the environment variables set at the machine level with the set command in a prompt and those for IIS with the Registry Editor. However, we got some inconsistencies, and we needed a way to validate what were the environment variables really seen by the web application! The Process Explorer tool from Sysinternals was downloaded and launched. After finding the process ID of the running w3wp.exe corresponding to the web application, a simple right-click to get the Properties and selecting the Environment Tab gave us the truth:

(This screenshot shows the results for one of our test applications on my development machine).

Getting a memory dump

Once the setup was checked on both machines without any too weird issues, the next step was to figure out why the application was randomly crashing. Even if the machines received different traffic loads, since applications running without our profiler enabled were not crashing, the chances were high that our C++ code was at the source of the problem. But the crashes were random… And you can’t install Visual Studio on a production server and attach to the process hoping that it will crash and start a debugging session there!

Windows Error Reporting is generating mini dumps when applications are crashing but they are usually not enough to start an investigation. Again, the other Sysinternals tools procdump was installed as a global crash handler with procdump -i c:\dumps -ma. The next time the application crashed, a memory dump was be generated in the c:\dumps folder. Don’t forget to create it manually if it does not exist.

From addresses to source code

To play with a memory dump, WinDbg is my preferred toy. I opened the memory dump and, in the case of a crash, the stack panel automatically displayed the call stack of the faulted thread:

The last frame triggering the issue (i.e., before KiUserExceptionDispatch) is Datadog_Profiler_Native!DllCanUnloadNow+0x2954b. Knowing that WinDbg transforms . in file names into _ leads to Datadog.Profiler.Native.dll which is the file where our profiler is implemented. However, WinDbg was not able to find the name of the function and only looked at the exported public symbols. With the lm command, you can see how WinDbg gets the symbols for this dll:

With DllCanUnloadNow, you could tell that we are dealing with some COM stuff but it did not really help me for the investigation: I needed to know which function was running which part of its code. Hopefully, for each release of the .NET profiler in Github, in addition to the .msi installer, the symbols and the source code are also provided.

Both files were unzipped in the folder where the dumps were copied. Then, I changed the Debugging Settings in WinDbg to point to these folders:

Let’s start with the symbols to let WinDbg match an instruction pointer to a function name. I asked WinDbg to provide details about the symbol resolution with !sym noisy. Then I forced the symbols for my module to gets reloaded with .reload /f “Datadog.Profiler.Native.dll”. In the flow of errors, I find out where the .pdb file should be stored so that WinDbg would find it:

So the problem is triggered somewhere in our Windows64BitStackFramesCollector::CollectStackSampleImplementation function. By simply double-clicking this frame, WinDbg automagically found the corresponding source file and pinpointed the culprit line:

A bit of WinDbg magic

To follow me a bit further, you need to understand what this code is doing: it is walking the stack of a thread to find the instruction pointers of each called function. This line 260 is dereferencing the address contained in context.Rsp. I looked at Locals panel to get its value:

The !address command gave me in which module this code was executed from:

It looked like a valid page with executable code…

I wanted to see why our stack walking code would break here. What if I asked WinDbg to show me this stack? To do that, I first needed to know which thread our code was trying to stack walk. I knew that Windows64BitStackFramesCollector was keeping track of the currently walked thread in a ManagedThreadInfo instance pointed to by its _pCurrentCollectionThreadInfo field:

This instance stores the thread ID in its _osThreadId field: now let’s ask WinDbg to switch to this thread.

The ~ command lists all threads:

A quick CTRL+F with “27600” stopped on the thread #72. Threads have a lot of identifiers in WinDbg and the first one allowed me to switch with ~72s.

The Stack panel was almost empty:

To be sure, I used the kp command… that told me that WinDbg was not really happy neither:

I was kind of stuck but my colleague Kevin Gosse mentioned that I could use r rip to see what would be the next instruction to be executed by this thread:

Then, the ln command (close to the !address command I used just before) allowed me to click the Browse Module link and see that, again, some code from Sentinel One was ready to execute.

This agent is part of an anti-virus (and more) solution that seems to highjack the stack of threads and our code was not dealing properly with this kind of situation. The fix was to protect our dereferencing code against access violation and stop walking the stack in that case.

Another debugging day at Datadog :^)

Raiders of the lost root: looking for memory leaks in .NET

Mon, 08 May 2023 09:05:15 +0000

Introduction

It’s been almost 12 years since I wrote LeakShell to help me automate the search of memory leaks in .NET. The idea was simple: compare 2 memory dumps of a leaking .NET application to show the types with increasing instances count.

Today, you could use Visual Studio Memory Usage tool to do the same but with a much better user interface! The additional killer feature is the ability to see the references chain that explains why a “leaky” object stays in memory.

My previous series about building your own .NET memory profiler in C# is based on CLR events and does not allow to get the references chain. This post explains how you could write your own memory profiler based on.NET profiler APIs in C++. Refer to this post for an introduction of how to implement ICorProfilerCallback to be loaded by the CLR in a .NET process.

How to detect memory leaks

From a high level view, detecting a memory leak means being able to know which objects stay alive garbage collection after garbage collection:

implement ICorProfilerCallback::ObjectAllocated to keep track of ALL objects in the heap,
use ICorProfilerCallback::MovedReferences2 to fixup the addresses when the live objects are moved during compaction garbage collections,
since** **ICorProfilerCallback::ObjectReferences is called for each surviving object, clean your list of live objects.

The first drawback of this solution is that the CLR has to disable concurrent GC to call these functions with probable impact on performances. However, if you can’t find a leak in production that leads to out of memory crashes, running one instance in this mode is perfectly acceptable. The second drawback is the complexity of keeping track of objects through compacting GCs.

This is why I implemented in .NET 7 a new set of functions in ICorProfilerInfo13 to mimic what you can do in C# with a weak reference:

CreateHandle : create a weak handle to wrap an object,
GetObjectIDFromHandle: get the address of the wrapped object or null if the object is no more in the heap,
DestroyHandle: clean up the weak handle.

Creating such a weak handle for allocated objects get rid of the address fixup complexity. However, you should not create a handle for ALL allocated objects because it will slow down the garbage collections. So the next step is to listen to the AllocationTick CLR event and create a weak handle for each sampled allocation. Even though the statistical distribution of such 100 KB threshold-based sampling is not perfect, leaking objects should appear.

After each garbage collection detected in ICorProfilerCallback2::GarbageCollectionFinished or via specific GC events, you could clean up this list of allocated objects by removing those for which GetObjectIDFromHandle returns null.

Feel free to look at the corresponding implementation in Datadog .NET profiler code.

Rebuild references chain up to a root

Even though it is possible to get the call stack that led to allocating a leaking object thanks to the AllocationTick event, it would be better to know why it stays in memory. So the next step is to rebuild the references chain up to the root.

As explained in a previous post, it is possible, for a given object, to get the list of its fields and build a graph of dependencies from a parent to its children. However, you are interested in the opposite and it would require to get these parent/children references for ALL objects in the heap. And this is not possible with the sampled AllocationTick event…

This is where ICorProfilerCallback::ObjectReferences shines:

1
2
3
4
5
6


HRESULT ObjectReferences(
   ObjectID objectId,
   ClassID  classId,
   ULONG    cObjectRefs,
   ObjectID objectRefIds[] 
);

This method is called during a garbage collection for all objects (i.e. objectId first parameter) still alive and lists its fields referencing objects in the heap (i.e. objectRefIds last parameter).

You could store each object as an ObjectNode:

1
2
3
4
5
6
7
8
9


struct ObjectNode
{
public:
    ObjectNode(ObjectID objectId);

public:
    ObjectID instance;
    std::vector<ObjectNode*> rootRefs;
};

in a vector that represents the heap.

For each fields, look for its corresponding node in the vector and add the node of its parent (given by objectId) to its rootRefs vector of parents. That way, you are building a back reference graph:

The small blue arrows show the parent/children reference given by ObjectReferences and the large purple ones are kept to build a reverse references graph you are interested in.

You know when all live objects in the heap have been listed when ICorProfilerCallback2::GarbageCollectionFinished is called. It is now time to get the build the references chain for all sampled objects still alive (thanks to GetObjectIDFromHandle returning non null address).

It is important to understand that it is a graph and not a tree because cycles exist in .NET.

This is common in situations where objects need to keep a reference to their “parents”. It means that these cycles should be detected when looking for the list of references of a given live object to avoid infinite recursion:

bool DumpNode(ObjectNode* node, std::vector& referenceStack)

The traversing DumpNode method takes a node (i.e. an object of the heap) and a stack where the parents will be added as we dig into the graph.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59


bool DumpNode(ObjectNode* node, std::vector<ObjectID>& referenceStack)
{
    // end of recursion: the node is a root
    if (node->rootRefs.size() == 0)
    {
        //  dump the root
        std::cout << std::endl;
        std::cout << std::hex << node->instance << std::dec;
        COR_PRF_GC_ROOT_KIND kind;
        COR_PRF_GC_ROOT_FLAGS flags;
        if (FindRoot(_roots, node->instance, kind, flags))
        {
            std::cout << " | ";
            DumpKind(kind);
            std::cout << " - ";
            DumpFlags(flags);
        }
        else
        {
            std::cout << " | ?";
        }
        std::cout << " = ";

        DumpObjectType(node->instance, _pCorProfilerInfo, _pFrameStore);
        std::cout << std::endl;

        // dump the references from the root
        for (int16_t i = referenceStack.size()-1; i >= 0; i--)
        {
            ObjectID reference = referenceStack[i];
            std::cout << " --> ";
            std::cout << std::hex << reference << std::dec;
            std::cout << " = ";
            DumpObjectType(reference, _pCorProfilerInfo, _pFrameStore);
            std::cout << std::endl;
        }

        return true;
    }

    // detect cycles
    if (Find(referenceStack, node->instance))
    {
        return false;
    }

    // go up into the reference chain
    referenceStack.push_back(node->instance);
    for (auto& parentNode : node->rootRefs)
    {
        if (DumpNode(parentNode, referenceStack))
        {
            return true;
        }
    }
    referenceStack.pop_back();

    return false;
}

If a parent node is already in the stack, a cycle is detected and that path is not used. Once a root is reached, the stack is dumped as shown in the following output:

OnGarbageCollectionFinished: 3859 objects in the heap.

OnRootReferences2: 90/109 roots.
            stack:40
        finalizer:1
           handle:49
            other:0
------------------

21266c00020 | H - 0 = Object[]
 --> 21268c092c8 = NativeRuntimeEventSource
 --> 21268c3fbe8 = EventSource.EventMetadata[]
 --> 21268c18010 = ParameterInfo[]
 --> 21268c17ef0 = RuntimeParameterInfo
 --> 21268c0ed58 = RuntimeMethodInfo
 --> 21268c0e708 = RuntimeType.RuntimeTypeCache
 --> 21268c0e840 = RuntimeType.RuntimeTypeCache.MemberInfoCache
 --> 21268c14978 = RuntimeMethodInfo[]
 --> 21268c10d70 = RuntimeMethodInfo
 --> 21268c29720 = Signature
 --> 21268c29770 = RuntimeType[]
=====================================

As shown in the output, it is possible to provide details about the kind of root is keeping the references chain alive thanks to ICorProfilerCallback::RootReferences2:

1
2
3
4
5
6
7


HRESULT RootReferences2(  
   ULONG  cRootRefs,  
   ObjectID rootRefIds[],  
   COR_PRF_GC_ROOT_KIND rootKinds[],  
   COR_PRF_GC_ROOT_FLAGS rootFlags[],  
   UINT_PTR rootIds[]
);

This function is called with three synchronized arrays cRootRefs long that contain for each root:

the address (**rootRefsIds **objectID),
the kind (rootKind for stack, finalizer, handle and other)
and flags (rootFlags for pinned, weak reference interior or ref counted).

These are stored in a vector of ObjectRoot:

Goodies: how to get arrays type name

I did not mention how to get the type name of either an ObjectID or a ClassID because it is explained in a previous post. However, I forgot to explain how to deal with the different kinds of arrays: single dimension (ex: byte[]), multidimensional (ex: byte[,]) or jagged (ex: byte[][]).

When you call ICorProfilerInfo::GetClassInfo on a ClassID corresponding to an array,

1
2
3


ModuleID moduleId;
mdTypeDef typeDefToken;
hr = _pCorProfilerInfo->GetClassIDInfo(classId, &moduleId, &typeDefToken);

it won’t fail but the module id and the metadata token will both be set to 0.

Instead, you have to call ICorProfilerInfo::IsArrayClass to get the rank and the item class ID of the array. This is then done recursively on the item class ID until it fails:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


std::string arrayBuilder;

CorElementType baseElementType;
ClassID itemClassId;
ULONG rank = 0;
if (_pCorProfilerInfo->IsArrayClass(classId, &baseElementType, &itemClassId, &rank) == S_OK)
{
    classId = itemClassId;
    isArray = true;
    AppendArrayRank(arrayBuilder, rank);

    // in case of matrices, it is needed to look for the last "good" item class ID
    // because all others might be array of array of ...
    for (size_t i = 0; i < rank; i++)
    {
        HRESULT hr = _pCorProfilerInfo->IsArrayClass(classId, &baseElementType, &itemClassId, &rank);
        if ((hr == S_FALSE) || FAILED(hr))
        {
            itemClassId = classId;

            break;
        }

        AppendArrayRank(arrayBuilder, rank);
        classId = itemClassId;
    }
}

Notice that the way to concatenate the possible [] / [,] / [][] could is the opposite of how the array type is defined in C#:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


void AppendArrayRank(std::string& arrayBuilder, ULONG rank)
{
    if (rank == 1)
    {
        arrayBuilder = "[]" + arrayBuilder;
    }
    else
    {
        std::stringstream builder;
        builder << "[";
        for (size_t i = 0; i < rank - 1; i++)
        {
            builder << ",";
        }
        builder << "]";

        arrayBuilder = builder.str() + arrayBuilder;
    }
}

For example, a byte[][,] is defined as an rank 2 array of array of byte.

References

Automate the search of memory leaks with LeakShell
Building your own .NET memory profiler in C#
Introduction to .NET Profiling with ICorProfilerCallback
Pull request in .NET 7 for ICorProfilerInfo13 to create weak handles
Datadog .NET Live Heap Profiler implementation

From Metadata to Event block in nettrace format

Fri, 10 Mar 2023 16:15:40 +0000

The previous episodes started the parsing of the “nettrace” format used when contacting the .NET Diagnostics IPC server, initiate the protocol to receive CLR events and start to parse stacks. This last episode covers the Metadata and Event blocks.

In terms of format, both Metadata and Event blocks share the same memory layout:

The common EventBlockHeader starts the block:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


#pragma pack(1)
struct EventBlockHeader
{
    uint16_t HeaderSize;
    uint16_t Flags;
    uint64_t MinTimestamp;
    uint64_t MaxTimestamp;

    // some optional reserved space might be following
};

The timestamp fields give the time of the first and last event in the block. The HeaderSize fields is important because additional information can be stored in the header. Since I have no idea what could be stored there, I simply skip it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


bool EventParserBase::OnParse()
{
    // read event block header
    EventBlockHeader ebHeader = {};
    if (!Read(&ebHeader, sizeof(ebHeader)))
    {
        return false;
    }

    // skip any optional content if any
    if (ebHeader.HeaderSize > sizeof(EventBlockHeader))
    {
        uint8_t optionalSize = ebHeader.HeaderSize - sizeof(EventBlockHeader);
        if (!SkipBytes(optionalSize))
        {
            return false;
        }
    }

The important piece of information to figure out how to unpack the rest of the block is kept in the Flags field. If the lowest bit is set, it means that the blobs header will be compressed:

1
2
3
4
5
6


// the rest of the block is a list of Event blobs
    //
    DWORD blobSize = 0;
    DWORD totalBlobSize = 0;
    DWORD remainingBlockSize = _blockSize - ebHeader.HeaderSize;
    bool isCompressed = ((ebHeader.Flags & 1) == 1);

The rest of the code iterates on each blob:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


// Note: in order to gain space, some fields of the header could be "inherited"
    // from the header of the previous blob --> need to pass it from blob to blob
    EventBlobHeader header = {};
    while (OnParseBlob(header, isCompressed, blobSize))
    {
        totalBlobSize += blobSize;
        blobSize = 0;

        if (totalBlobSize >= remainingBlockSize - 1) // try to detect last blob
        {
            // don't forget to check the end of block tag
            uint8_t tag;
            if (!ReadByte(tag) || (tag != NettraceTag::EndObject))
            {
                std::cout << "Missing end of block tag\n";
                return false;
            }

            return true;
        }
    }

    return true;
}

Here is the tricky part: to gain space, each blob starts with a header that could be “compressed”. The compression mechanism is simple: the first byte is a bitfield value that indicates which fields are present (i.e. their value should be read from the memory block) or skipped (i.e. their value is the same as the previous blob header). Therefore, an EventBlobHeader is passed by reference to the OnParseBlob function. My MetadataParser and EventParser implementations of OnParseBlob both starts with the same code to read the header:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


bool XXXParser::OnParseBlob(EventBlobHeader& header, bool isCompressed, DWORD& blobSize)
{
    if (isCompressed)
    {
        if (!ReadCompressedHeader(header, blobSize))
        {
            return false;
        }
    }
    else
    {
        if (!ReadUncompressedHeader(header, blobSize))
        {
            return false;
        }
    }

The implementation to read compressed and uncompressed version of the header is a direct translation of the TraceEvent C# code into C++.

The EventBlobHeader contains details of events:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


#pragma pack(1)
struct EventBlobHeader_V4
{
    uint32_t EventSize;
    uint32_t MetadataId;
    uint32_t SequenceNumber;
    uint64_t ThreadId;
    uint64_t CaptureThreadId;
    uint32_t ProcessorNumber;
    uint32_t StackId;
    uint64_t Timestamp;
    GUID ActivityId;
    GUID RelatedActivityId;
    uint32_t PayloadSize;
};

The “identity” of an event is given by the MetadataId field that refers to information defined in Metadata “object“ (for which MetadataId is 0).
The SequenceNumber field is incremented on a per thread basis each time an event is emitted. This could be used to detect if some events have been dropped (for a given CaptureThreadId, two consecutive events have a SequenceNumber incremented by more than 1 — more on dropped events in the forthcoming SequencePoint “object” description). Its value is 0 for a metadata “object”
The ThreadId and CaptureThreadId field have always the same value for Event “object”; CaptureThreadId is 0 for Metadata “object”.
In case of Event “object”, the StackId field refers to one of the stacks extracted from a Stack “object”. Its value is 0 for Metadata “object”.

The Metadata “object”

As the documentation states, each MetadataBlock holds a set of metadata records. Each metadata record has an ID and it describes one type of event. Each event has a metadataId field which will indicate the ID of the metadata record which describes that event.

The resulting mapping is stored in EventPipeSession class:

1
2


// per metadataID event metadata description
    std::unordered_map<uint32_t, EventCacheMetadata> _metadata;

However, the rest of the documentation is partially right in the case of nettrace stream received through EventPipe: Metadata includes an event name, provider name, and the layout of fields that are encoded in the event’s payload section.

First, the fields layout is simply not there. In addition, for some providers (dotnet runtime, private and rundown), the event names are empty strings. So, the data structure filled from the MetadataBlock will most of the time have an empty EventName field. Note that the “Microsoft-DotNETCore-EventPipe” provider (i.e. command events for that specific provider) and EventSource-derived classes written in C# provide the events name:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


class EventCacheMetadata
{
public:
    uint32_t     MetadataId;
    std::wstring ProviderName;
    uint32_t     EventId;
    std::wstring EventName; // empty most of the time
    uint64_t     Keywords;
    uint32_t     Version;
    uint32_t     Level;
};

In addition to the provider’s name serialized as a UTF16 string (including last ‘\0’ wide character), the EventId field is the key used to identify an event.

After these details, you will find a 4 bytes value corresponding to the number of fields in the event payload. As already mentioned, this value is always 0 so my code is skipping the rest of the metadata block payload.

The Event “object”

And at last, here comes the time to parse Event “object” payload! The MetadataId field of the EventBlobHeader is used to find the provider’s name and event id:

1
2
3
4
5


bool EventParser::OnParseBlob(EventBlobHeader& header, bool isCompressed, DWORD& blobSize)
{
    ...

    auto& metadataDef = _metadata[header.MetadataId];

So, the rest of the function reads the payload based on the expected event id:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


switch (metadataDef.EventId)
    {
        case EventIDs::AllocationTick:
            if (!OnAllocationTick(header.PayloadSize, metadataDef))
            {
                return false;
            }
            break;

        case ...
            break;

        case EventIDs::ExceptionThrown:
            if (!OnExceptionThrown(header.PayloadSize, metadataDef))
            {
                return false;
            }
            break;

        default:  // skip events we are not interested in
        {
            SkipBytes(header.PayloadSize);
        }
    }

    blobSize += header.PayloadSize;

    return true;
}

The format of each event payload is usually given by the Microsoft documentation. If not, you should look into the ClrEtwall.man file where the payload of ALL events are defined. For example, the AllocationTick event payload provides the name of the last allocated type to reach the 100 KB threshold (read this blog post for more details about how to use this event):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


//  AllocationAmount    UInt32          The allocation size, in bytes.
//                                      This value is accurate for allocations that are less than the length of a ULONG(4,294,967,295 bytes).
//                                      If the allocation is greater, this field contains a truncated value.
//                                      Use AllocationAmount64 for very large allocations.
//  AllocationKind      UInt32          0x0 - Small object allocation(allocation is in small object heap).
//                                      0x1 - Large object allocation(allocation is in large object heap).
//  ClrInstanceID       UInt16          Unique ID for the instance of CLR or CoreCLR.
//  AllocationAmount64  UInt64          The allocation size, in bytes.This value is accurate for very large allocations.
//  TypeId              Pointer         The address of the MethodTable.When there are several types of objects that were allocated during this event,
//                                      this is the address of the MethodTable that corresponds to the last object allocated (the object that caused the 100 KB threshold to be exceeded).
//  TypeName            UnicodeString   The name of the type that was allocated.When there are several types of objects that were allocated during this event,
//                                      this is the type of the last object allocated (the object that caused the 100 KB threshold to be exceeded).
//  HeapIndex           UInt32          The heap where the object was allocated.This value is 0 (zero)when running with workstation garbage collection.
//  Address             Pointer         The address of the last allocated object.
//

Based on this fields definition, the EventParser::OnAllocationTick function is reading each field after the other thanks to the ReadWord, ReadDWord, ReadLong and ReadWString :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


bool EventParser::OnAllocationTick(DWORD payloadSize, EventCacheMetadata& metadataDef)
{
    DWORD readBytesCount = 0;
    DWORD size = 0;
    std::cout << "\nAllocation Tick:\n";

    // get common fields
    uint32_t dword = 0;
    if (!ReadDWord(dword))
    {
        return false;
    }
    readBytesCount += sizeof(dword);
    std::cout << "   Amount        = " << dword << " bytes\n";

    if (!ReadDWord(dword))
    {
        return false;
    }
    readBytesCount += sizeof(dword);
    std::cout << "   Kind          = " << ((dword == 1) ? "LOH" : "small") << " bytes\n";

    uint16_t word = 0;
    if (!ReadWord(word))
    {
        return false;
    }
    readBytesCount += sizeof(word);
    std::cout << "   CLR ID        = " << word << "\n";

    uint64_t ulong = 0;
    if (!ReadLong(ulong))
    {
        return false;
    }
    readBytesCount += sizeof(ulong);
    std::cout << "   Amount64      = " << ulong << " bytes\n";

The bitness of the monitored application is important when “pointers” need to be read from the payload: use ReadDWord for 32-bit and ReadLong for 64-bit:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


// skip useless MT address
    // Note: handle 32/64 bit difference
    if (_is64Bit)
    {
        if (!ReadLong(ulong))
        {
            return false;
        }
        readBytesCount += sizeof(ulong);
    }
    else
    {
        if (!ReadDWord(dword))
        {
            return false;
        }
        readBytesCount += sizeof(dword);
    }

And if you don’t need the rest of the payload, SkipBytes is your friend:

1
2
3


// skip the rest of the payload
    return SkipBytes(payloadSize - readBytesCount);
}

I had some issues when dealing with the ExceptionThrown event payload:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


// Type             wstring     Exception type
// Message          wstring     Exception message
// EIPCodeThrow     win:Pointer Instruction pointer where exception occurred.
// ExceptionHR      win:UInt32  Exception HRESULT.
// ExceptionFlags   win:UInt16
//      0x01: HasInnerException (see CLR ETW Events in the Visual Basic documentation).
//      0x02: IsNestedException.
//      0x04: IsRethrownException.
//      0x08: IsCorruptedStateException (indicates that the process state is corrupt).
//      0x10: IsCLSCompliant (an exception that derives from Exception is CLS-compliant).
// ClrInstanceID win:UInt16 Unique ID for the instance of CLR or CoreCLR.

In case of an empty message, the field itself was not even there! Not even 0 for a ‘\0’ wide character… In fact, there is a bug in the serialization code that skips the field in that case. This has been fixed in .NET 6 by storing “NULL” as the serialized string: I would have preferred ‘\0’ but it seems to be compatible with the ETW implementation.

To support .NET Core 3+ and .NET 5, my code is comparing the size of the remaining of the payload after reading the exception type with the expected size of the 4 remaining fields after the exception message. If it is greater then it means that there is a string for the message. If not, I know that the message is empty:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


bool EventParser::OnExceptionThrown(DWORD payloadSize, EventCacheMetadata& metadataDef)
{
    DWORD readBytesCount = 0;
    DWORD size = 0;

    // read exception type
    ...

    // Size of the ExceptionThrown payload AFTER the Message field
    uint16_t exceptionRemainingPayloadSize = (_is64Bit ? 8 : 4) + 4 + 2 + 2;

    // In case of "empty" message, it might not be even visible as "\0" before .NET Core 6 (and after, will be "NULL")
    // so it is needed to check if the remaining payload contains such a string
    if ((payloadSize - readBytesCount) == _exceptionRemainingPayloadSize)
    {
        std::wcout << L"   message = ''\n";
    }
    else
    {
        if (!ReadWString(strBuffer, size))
        {
            return false;
        }
        readBytesCount += size;

        // handle empty string case (check for "NULL" in case of .NET 6+)
        if (strBuffer.empty() || (wcscmp(strBuffer.c_str(), L"NULL") == 0))
            std::wcout << L"   message = ''\n";
        else
        {
            std::wcout << L"   message = " << strBuffer.c_str() << L"\n";
        }
    }

    // skip the rest of the payload
    return SkipBytes(payloadSize - readBytesCount);
}

The SequencePointBlock “object”

The last “object” type is the sequence point block that contains the following fields:

In addition to these fields, it also implicitly tells you that new stack “object” will be received (with stack id restarting from 1) to match next Event “objects”. For example, the following trace shows how a sequence point block resets the stacks by restarting at 1:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44


Event block (140 bytes)

blob header:
   StackId           = 3
Contention

blob header:
   StackId           = 4
Event = 81

blob header:
   StackId           = 3
Contention
...
------------------------------------------------ 

________________________________________________
SequencePoint block (217 bytes)
...
------------------------------------------------

________________________________________________
Stack block (105 bytes)

Stack block header:
   FirstID: 1
   Count  : 2
------------------------------------------------

________________________________________________
Event block (92 bytes)

blob header:
   StackId           = 1
Contention

blob header:
   StackId           = 2
Event = 81

blob header:
   StackId           = 1
Contention:
------------------------------------------------

So, the stacks you might have cached based on the already received stack “objects” should now be invalidated like what I’m doing in SequencePointParser::OnParse:

1
2
3
4
5
6


bool SequencePointParser::OnParse()
{
    // reset stack caches
    _stacks32.clear();
    _stacks64.clear();
    ...

You now have all the elements you need to listen to CLR events on Windows and Linux for .NET Core 3+ and .NET 5+. If you are still running applications with .NET Framework, you will need to use ETW but this is another story.

Resources

Episode 1 — Digging into the CLR Diagnostics IPC Protocol in C#
Episode 2 — .NET Diagnostic IPC protocol: the C++ way
[Episode 3 ](/posts/2022-10-23_clr-events-go-for/ CLR events: go for the nettrace file format!
[Episode 4 ](/posts/2022-11-27_parsing-the-nettrace-stream/ Parsing the “nettrace” steam
[Episode 5 ](/posts/2023-01-15_reading-object-in-memory/ Reading “object” in memory — starting with stacks
Source code for the C++ implementation of CLR events listener
Diagnostics IPC protocol documentation

Reading “object” in memory — starting with stacks

Sun, 15 Jan 2023 16:38:45 +0000

The previous episodes started the parsing of the “nettrace” format used when contacting the .NET Diagnostics IPC server and initiate the protocol to receive CLR events. It is now time to see how to get the payload of each “object” type, especially how stacks are stored.

We have seen that the stream starts with a TraceObject that describes the rest of the stream followed by a sequence of “object”:

The remaining of each “object” is a 32 bit block size followed by the payload.

Well… not only. One thing I missed when I started to work on the nettrace format is the fact that all “object” payloads must be 4-bytes aligned on the beginning of the stream!

This is why I’m keeping track of the current position in the EventPipeSession class:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


private:
    IIpcEndpoint* _pEndpoint;
    bool _stopRequested;

    // parsers
    MetadataParser _metadataParser;
    EventParser _eventParser;
    StackParser _stackParser;
    SequencePointParser _sequencePointParser;

    // Keep track of the position since the beginning of the "file"
    // i.e. starting at 0 from the first character of the NettraceHeader
    //      Nettrace
    uint64_t _position;
    ...
};

So each ParseXXXBlock function checks the minimum reader version in the header before reading the “object” payload as a memory block. The idea is being able to support backward compatibility:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


bool EventPipeSession::ParseMetadataBlock(ObjectHeader& header)
{
    if (header.MinReaderVersion != 2) return false;

    uint32_t blockSize = 0;

    // read the block and send it to the corresponding parser
    uint64_t blockOriginInFile = 0;
    if (!ExtractBlock("Metadata", blockSize, blockOriginInFile))
        return false;

    return _metadataParser.Parse(_pBlock, blockSize, blockOriginInFile);
}

The ExtractBlock function reads the size of the payload (and skips the padding if any) with ReadBlockSize:

1
2
3
4
5
6
7
8
9


bool EventPipeSession::ExtractBlock(const char* blockName, uint32_t& blockSize, uint64_t& blockOriginInFile)
{
    // get the block size
    if (!ReadBlockSize(blockName, blockSize))
        return false;

    // skip the block + final EndOfObject tag
    blockSize++;
    ...

The block name is only used for error messages if needed.

The next step is to read the payload in a memory block using these two EventPipeSession fields:

1
2
3
4
5


...
    // buffer used to read each block that will be then parsed
    uint8_t* _pBlock;
    uint32_t _blockSize;
    ...

In the session constructor, _blockSize is set to 4 KB and _pBlock points to an allocated memory buffer of that size.

The rest of ExtractBlock deals with payload size: if the current payload to parse is larger than _blockSize, then these fields are updated up to a maximum of 100 KB (i.e. max block size sent by the CLR).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


// check if it is needed to resize the block buffer
    if (_blockSize < blockSize)
    {
        // don't expect blocks larger than 100KB
        if (blockSize > MAX_BLOCK_SIZE)
            return false;

        delete [] _pBlock;
        _pBlock = new uint8_t[blockSize];
        ::ZeroMemory(_pBlock, blockSize);
        _blockSize = blockSize;
    }

    // keep track of the current position in file for padding
    blockOriginInFile = _position;
    if (!Read(_pBlock, blockSize))
    {
        Error = ::GetLastError();
        std::cout << "Error while extracting " << blockName << " block: 0x" << std::hex << Error << std::dec << "\n";
        return false;
    }
    std::cout << "\n" << blockName << " block (" << blockSize << " bytes)\n";
    DumpBuffer(_pBlock, blockSize);

    return true;
}

For debugging sake, I’m displaying each “object” payload

thanks to the DumpBuffer helper.

To ease the memory access to the memory block content, my BlockParser will be used as a base class for each dedicated parsers:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


class BlockParser
{
public:
    BlockParser();
    bool Parse(uint8_t* pBlock, uint32_t bytesCount, uint64_t blockOriginInFile);
    void SetPointerSize(uint8_t pointerSize);

public:
    uint8_t PointerSize;

protected:
    virtual bool OnParse() = 0;

    // Access helpers
    bool Read(LPVOID buffer, DWORD bufferSize);
    bool ReadByte(uint8_t& byte);
    bool ReadWord(uint16_t& word);
    bool ReadDWord(uint32_t& dword);
    bool ReadLong(uint64_t& ulong);
    bool ReadDouble(double& d);
    bool ReadVarUInt32(uint32_t& val, DWORD& size);
    bool ReadVarUInt64(uint64_t& val, DWORD& size);
    bool ReadWString(std::wstring& wstring, DWORD& bytesRead);
    bool SkipBytes(uint32_t byteCount);

// shared fields
protected:
    bool _is64Bit;
    uint32_t _blockSize;
    uint32_t _pos;

private:
    uint8_t* _pBlock;
    uint64_t _blockOriginInFile;
};

The Parse function accepts the memory buffer containing an “object” payload, its size and its position since the beginning of the stream. The derived class will have to implement the OnParse function using the ReadXXX helpers.

The two ReadVarUintXXX functions are different from the other direct read helpers because they deal with some simple compression mechanisms used by the serialization of 32-bit and 64-bit numbers.

In the different types of “object” payloads, the strings are serialized as UTF16 strings ending with a “\0” wide character. Here is the implementation of the helper function used to read a std::wstring from a memory block:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


bool BlockParser::ReadWString(std::wstring& wstring, DWORD& bytesRead)
{
    uint16_t character;
    bytesRead = 0;  // in case of empty string
    while (true)
    {
        if (!ReadWord(character))
        {
            return false;
        }

        // protect against invalid UNICODE character (due to missing fields in ExceptionThrown event)
        if (character > 256)
        {
            // rewind the character
            _pos = _pos - sizeof(character);

            // this is only covering a missing string
            return (bytesRead == 0);
        }

        bytesRead += sizeof(character);

        // Note that an empty string contains only that \0 character
        if (character == 0) // \0 final character of the string
            return true;

        wstring.push_back(character);
    }
}

Note the check for character content in the loop: this is due to a serialization issue I will discuss later when the event “object” block will be detailed.

The Stack “object”

If you remember my previous post about retrieving call stacks for CLR events with TraceEvent, you might be wondering why there is a specific stack object since a ClrStackWalk event should contain the frames if the Stack keyword is enabled for the .NET provider. In fact, the current TraceEvent implementation is not using the stack object sent by the CLR (maybe to have the same code between ETW and EventPipe).

One stack “object” received in a nettrace stream contains one or more stacks. Each stack is identified by an id (more about this soon) and contains a list of instruction pointer addresses.

In the previous screenshot, the id of the first stack is 1 and the second is 2. In the next stack “object”, the FirstId field will be 3 and so on. This avoids storing the id in each call stack and saves space.

Note that even if this does not seem to make any sense, it might happen that the addresses list is empty.

These call stacks are stored in EventPipeSession as a per id cache:

1
2
3
4


// per stackID stack
    // only one will be used depending on the bitness of the monitored application
    std::unordered_map<uint32_t, EventCacheStack32> _stacks32;
    std::unordered_map<uint32_t, EventCacheStack64> _stacks64;

The frames are stored as addresses in a vector:

1
2
3
4
5
6


class EventCacheStack32
{
public:
    uint32_t Id;
    std::vector<uint32_t> Frames;
};

The CLR is sending one stack per unique callstack (i.e. at least one frame is different). As you will soon see, each event “object” contains a stack id corresponding to the chain of code from which it is sent.

The next episode will detail the Metadata and Event blocks to end the series.

Resources

Episode 1 — Digging into the CLR Diagnostics IPC Protocol in C#
Episode 2 — .NET Diagnostic IPC protocol: the C++ way
[Episode 3 ](/posts/2022-10-23_clr-events-go-for/ CLR events: go for the nettrace file format!
[Episode 4 ](/posts/2022-11-27_parsing-the-nettrace-stream/ Parsing the “nettrace” steam
Source code for the C++ implementation of CLR events listener
Diagnostics IPC protocol documentation

Parsing the “nettrace” stream of (not only) events

Sun, 27 Nov 2022 13:59:44 +0000

The previous episodes explained how to contact the .NET Diagnostics IPC server and initiate the protocol to receive CLR events. It is now time to dig into the “nettrace” stream format!

As the IPC command documentation states, the response to the CollectTracing command is followed by an Optional Continuation of a nettrace format stream of events. In fact, before .NET Core 3, the netperf format was used but I will focus on the nettrace format also used in .NET 5+.

From a high-level view, it is a header followed by a stream of “objects”; each described by a header and ending with a byte with 6 as value:

Let’s start with the nettrace header:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


#pragma pack(1)
struct NettraceHeader
{
    uint8_t Magic[8];               // "Nettrace" with not '\0'
    uint32_t FastSerializationLen;  // 20
    uint8_t FastSerialization[20];  // "!FastSerialization.1" with not '\0'
};

const char* NettraceHeaderMagic = "Nettrace";
const char* FastSerializationMagic = "!FastSerialization.1";

It can be used to check the format and version of the received data (in case of format evolution over time):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


bool CheckNettraceHeader(NettraceHeader& header)
{
    if (!IsSameAsString(header.Magic, sizeof(header.Magic), NettraceHeaderMagic))
        return false;

    if (header.FastSerializationLen != strlen(FastSerializationMagic))
        return false;

    if (!IsSameAsString(header.FastSerialization, sizeof(header.FastSerialization), FastSerializationMagic))
        return false;

    return true;
};

In memory, the “strings” are stored as an array of UTF8 characters without trailing ‘\0’

This drives the implementation of the comparison helper:

1
2
3
4


bool IsSameAsString(uint8_t* bytes, uint16_t length, const char* characters)
{
    return memcmp(bytes, characters, length) == 0;
}

Everything is an “object”

After the header, data is represented as “objects” whose description is stored in an ObjectHeader:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


#pragma pack(1)
struct ObjectHeader
{
    NettraceTag TagTraceObject;         // 5
    NettraceTag TagTypeObjectForTrace;  // 5
    NettraceTag TagType;                // 1
    uint32_t Version;                   //
    uint32_t MinReaderVersion;          //
    uint32_t NameLength;                // length of UTF8 name that follows
};

followed by the name of the object type in UTF8. Note that, like for “!FastSerialization.1” in NettraceHeader, its length is provided in the NameLength field of the ObjectHeader. For example, here is how the initial TraceObject header is stored in memory:

and its equivalent in code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


#pragma pack(1)
struct TraceObjectHeader : ObjectHeader
{
  //NettraceTag TagTraceObject;         // 5
  //NettraceTag TagTypeObjectForTrace;  // 5
  //NettraceTag TagType;                // 1
  //uint32_t Version;                   // 4
  //uint32_t MinReaderVersion;          // 4
  //uint32_t NameLength;                // 5
    uint8_t Name[5];                    // 'Trace
    NettraceTag TagEndTraceObject;      // 6
};

Note that after the “object” type name, an “end of object” byte (i.e. = 6) appears before the payload.

So, each kind of “object” shares the same ObjectHeader followed by a UTF8 type name:

“EventBlock” : contains one or more events
“MetadataBlock” : contains partial description of events (no name nor payload fields)
“StackBlock” : contains call stacks (i.e. arrays of instruction pointers)
“SPBlock” : contains check point inside the stream inside the stream (used for drop message detection and callstack cache invalidation)

It means that you need to compare the strings to figure out the “object” type:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58


const char* EventBlockName = "EventBlock";
const char* MetadataBlockName = "MetadataBlock";
const char* StackBlockName = "StackBlock";
const char* SequencePointBlockName = "SPBlock";

ObjectType EventPipeSession::GetObjectType(ObjectHeader& header)
{
    // check validity
    if (header.TagTraceObject != NettraceTag::BeginPrivateObject) return ObjectType::Unknown;
    if (header.TagTypeObjectForTrace != NettraceTag::BeginPrivateObject) return ObjectType::Unknown;
    if (header.TagType != NettraceTag::NullReference) return ObjectType::Unknown;

    // figure out which type it is based on the name:
    //   EventBlock -> "EventBlock"  (size = 10)
    //   MetadataBlock -> "MetadataBlock" (size = 13)
    //   StackBlock -> "StackBlock" (size = 10)
    //   SequencePointBlock -> "SPBlock" (size = 7)
    if (header.NameLength == 13)
    {
        uint8_t buffer[13];
        if (!Read(buffer, 13))
            return ObjectType::Unknown;

        if (IsSameAsString(buffer, 13, MetadataBlockName))
            return ObjectType::MetadataBlock;

        return ObjectType::Unknown;
    }
    else
    if (header.NameLength == 10)
    {
        uint8_t buffer[10];
        if (!Read(buffer, 10))
            return ObjectType::Unknown;

        if (IsSameAsString(buffer, 10, EventBlockName))
            return ObjectType::EventBlock;
        else
        if (IsSameAsString(buffer, 10, StackBlockName))
            return ObjectType::StackBlock;

        return ObjectType::Unknown;
    }
    else
    if (header.NameLength == 7)
    {
        uint8_t buffer[7];
        if (!Read(buffer, 7))
            return ObjectType::Unknown;

        if (IsSameAsString(buffer, 7, SequencePointBlockName))
            return ObjectType::SequencePointBlock;

        return ObjectType::Unknown;
    }

    return ObjectType::Unknown;
}

The first object that appears in the stream contains details about the whole stream:

After the header, some fields follow as a payload:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


#pragma pack(1)
struct ObjectFields
{
    uint16_t Year;
    uint16_t Month;
    uint16_t DayOfWeek;
    uint16_t Day;
    uint16_t Hour;
    uint16_t Minute;
    uint16_t Second;
    uint16_t Millisecond;
    uint64_t SyncTimeQPC;
    uint64_t QPCFrequency;
    uint32_t PointerSize;
    uint32_t ProcessId;
    uint32_t NumProcessors;
    uint32_t ExpectedCPUSamplingRate;
};

Beyond the timestamp information and the pointer size (required when call stack instruction pointers will be read as 8 (64-bit) or 4 (32-bit) bytes addresses) the other fields are not really interesting.

Let’s go back to a higher-level view

Here is the code to listen to the nettrace stream:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32


bool EventPipeSession::Listen()
{
    if (!ReadHeader())
        return false;

    if (!ReadTraceObjectHeader())
        return false;

    ObjectFields ofTrace;
    if (!ReadObjectFields(ofTrace))
        return false;

    // use the "trace object" fields to figure out the bitness of the application
    Is64Bit = ofTrace.PointerSize == 8;
    _stackParser.SetPointerSize(ofTrace.PointerSize);
    _metadataParser.SetPointerSize(ofTrace.PointerSize);
    _eventParser.SetPointerSize(ofTrace.PointerSize);

    // don't forget to check the end object tag
    uint8_t tag;
    if (!ReadByte(tag) || (tag != NettraceTag::EndObject))
        return false;

    // read one "object" after the other
    while (ReadNextObject())
    {
        std::cout << "------------------------------------------------\n";
        std::cout << "\n________________________________________________\n";
    }

    return _stopRequested;
}

I will come back to the different _XXXparser fields soon.

The ReadNextObject helper is responsible for reading the expected ObjectHeader and the string that follows to figure out what is the type of this “object” and what payload to expect:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


bool EventPipeSession::ReadNextObject()
{
    // get the type of object from the header
    ObjectHeader header;
    if (!Read(&header, sizeof(ObjectHeader)))
    {
        Error = ::GetLastError();
        std::cout << "Error while reading Object header: 0x" << std::hex << Error << std::dec << "\n";
        return false;
    }

    ObjectType ot = GetObjectType(header);
    if (ot == ObjectType::Unknown)
    {
        std::cout << "Invalid object header type:\n";
        DumpObjectHeader(header);
        return false;
    }

    // don't forget to check the end object tag
    uint8_t tag;
    if (!ReadByte(tag) || (tag != NettraceTag::EndObject))
    {
        std::cout << "Missing end of object tag: " << (uint8_t)tag << "\n";
        return false;
    }
    ...

The GetObjectType function checks the header validity and extracts the “object” type name:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53


ObjectType EventPipeSession::GetObjectType(ObjectHeader& header)
{
    // check validity
    if (header.TagTraceObject != NettraceTag::BeginPrivateObject) return ObjectType::Unknown;
    if (header.TagTypeObjectForTrace != NettraceTag::BeginPrivateObject) return ObjectType::Unknown;
    if (header.TagType != NettraceTag::NullReference) return ObjectType::Unknown;

    // figure out which type it is based on the name:
    //   EventBlock -> "EventBlock"  (size = 10)
    //   MetadataBlock -> "MetadataBlock" (size = 13)
    //   StackBlock -> "StackBlock" (size = 10)
    //   SequencePointBlock -> "SPBlock" (size = 7)
    if (header.NameLength == 13)
    {
        uint8_t buffer[13];
        if (!Read(buffer, 13))
            return ObjectType::Unknown;

        if (IsSameAsString(buffer, 13, MetadataBlockName))
            return ObjectType::MetadataBlock;

        return ObjectType::Unknown;
    }
    else
    if (header.NameLength == 10)
    {
        uint8_t buffer[10];
        if (!Read(buffer, 10))
            return ObjectType::Unknown;

        if (IsSameAsString(buffer, 10, EventBlockName))
            return ObjectType::EventBlock;
        else
        if (IsSameAsString(buffer, 10, StackBlockName))
            return ObjectType::StackBlock;

        return ObjectType::Unknown;
    }
    else
    if (header.NameLength == 7)
    {
        uint8_t buffer[7];
        if (!Read(buffer, 7))
            return ObjectType::Unknown;

        if (IsSameAsString(buffer, 7, SequencePointBlockName))
            return ObjectType::SequencePointBlock;

        return ObjectType::Unknown;
    }

    return ObjectType::Unknown;
}

The same IsSameAsString helper is used to compare the read “object” name with the known types.

The end of ReadNextObject simply parses the “object” payload as a memory block based on its type:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


    ...
    switch (ot)
    {
        case ObjectType::EventBlock:
            return ParseEventBlock(header);
        case ObjectType::MetadataBlock:
            return ParseMetadataBlock(header);
        case ObjectType::StackBlock:
            return ParseStackBlock(header);
        case ObjectType::SequencePointBlock:
            return ParseSequencePointBlock(header);

        default:
            return false;
    }
}

The next step will be to look into each “object” type payload.

Resources

Episode 1 — Digging into the CLR Diagnostics IPC Protocol in C#
Episode 2 — .NET Diagnostic IPC protocol: the C++ way
[Episode 3 ](/posts/2022-10-23_clr-events-go-for/ CLR events: go for the nettrace file format!
Source code for the C++ implementation of CLR events listener
Diagnostics IPC protocol documentation

CLR events: go for the nettrace file format!

Sun, 23 Oct 2022 16:55:52 +0000

As shown in the previous post, the processing of ProcessInfo diagnostic commands is easy because you send a request and read the different fields from the response. This is different if you want to receive events from the CLR via EventPipe. In C#, the TraceEvent nuget package wraps everything under a nice event handler based model as shown in many of my previous posts.

Behind the scene, a StartSession command is sent (more details about the parameters later) and the response contains the numeric ID of the session. Then, the events will be read from the IPC channel as a binary stream of data with the “nettrace“ file format. The collection ends when the StopTracing command is sent.

The source code is available from my github repository.

Hidding the transport layer: IIpcEndoint

Unlike the previous post, to send the command and read the response back from the CLR , I’m wrapping the transport layer with the IIpcEndpoint interface:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


class IIpcEndpoint
{
public:
    virtual bool Write(LPCVOID buffer, DWORD bufferSize, DWORD* writtenBytes) = 0;
    virtual bool Read(LPVOID buffer, DWORD bufferSize, DWORD* readBytes) = 0;
    virtual bool ReadByte(uint8_t& byte) = 0;
    virtual bool ReadWord(uint16_t& word) = 0;
    virtual bool ReadDWord(uint32_t& dword) = 0;
    virtual bool ReadLong(uint64_t& ulong) = 0;

    virtual bool Close() = 0;
    virtual ~IIpcEndpoint() = default;
};

It abstracts the write and read accesses to the underlying transport layer. In addition, the base class accepts a “recorder” that allows me to store what is received from the CLR into any kind of storage (today only a file-based recorder that helped a lot to reproduce specific situations without the need to have a running process to connect to):

The PidEndpoint class accepts the process id of the running .NET application to monitor its CLR events and an optional recorder implementing the IIpcRecorder interface. The Create static factory implementation creates the expected named pipe on Windows (or the domain socket on Linux) and stores the handle into its _handle field:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42


PidEndpoint* PidEndpoint::CreateForWindows(int pid, IIpcRecorder* pRecorder)
{
    PidEndpoint* pEndpoint = new PidEndpoint(pRecorder);

    // build the pipe name as described in the protocol
    wchar_t pszPipeName[256];
    int nCharactersWritten = -1;
    nCharactersWritten = wsprintf(
        pszPipeName,
        L"\\\\.\\pipe\\dotnet-diagnostic-%d",
        pid
    );

    // check that CLR has created the diagnostics named pipe
    if (!::WaitNamedPipe(pszPipeName, 200))
    {
        auto error = ::GetLastError();
        std::cout << "Diagnostics named pipe is not available for process #" << pid << " (" << error << ")" << "\n";
        return nullptr;
    }

    // connect to the named pipe
    HANDLE hPipe;
    hPipe = ::CreateFile(
        pszPipeName,    // pipe name
        GENERIC_READ |  // read and write access
        GENERIC_WRITE,
        0,              // no sharing
        NULL,           // default security attributes
        OPEN_EXISTING,  // opens existing pipe
        0,              // default attributes
        NULL);          // no template file

    if (hPipe == INVALID_HANDLE_VALUE)
    {
        std::cout << "Impossible to connect to " << pszPipeName << "\n";
        return nullptr;
    }

    pEndpoint->_handle = hPipe;
    return pEndpoint;
}

The next step is to open a tracing session by sending the StartSession command.

The Trace diagnostic commands

Following the same object model provided by the Microsoft.Diagnostics.NETCore.Client nuget, my DiagnosticsClient class hides the transport layer. It also exposes high level functions such as OpenEventPipeSession to initiate a trace event session with the CLR:

1

EventPipeSession* OpenEventPipeSession(uint64_t keywords, EventVerbosityLevel verbosity);

If you remember from TraceEvent, you need a few parameters to create a session:

size of circular buffers used by the CLR to cache events (same as Perfview, use 16 MB as default)
netttrace format (i.e. value of 1)
if rundown events are needed
a list of providers (“Microsoft-Windows-DotNETRuntime” for the CLR in my case)
keywords
verbosity level
possible arguments (none here)

Here is the corresponding C++ description of the command type:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


const uint8_t DotnetProviderMagicLength = 32;
struct MagicProvider
{
    wchar_t Magic[DotnetProviderMagicLength];
};

// 32 wchar_t (including \0)
const MagicProvider DotnetProviderMagic = { L"Microsoft-Windows-DotNETRuntime" };

const uint32_t CircularBufferMBSize = 16;
const uint32_t NetTraceFormat = 1;

#pragma pack(1)
struct StartSessionMessage : public IpcHeader
{
    uint32_t CircularBufferMB;  // 16 MB
    uint32_t Format;            // 1 for NetTrace format
    uint8_t RequestRundown;     // 0 because don't want rundown

    // array of provider configuration
    uint32_t ProviderCount;     // 1 only: Microsoft-Windows-DotNETRuntime
    uint64_t Keywords;          // from EventKeyword
    uint32_t Verbosity;         // from EventPipeEventLevel
    uint32_t ProviderStringLen; // number of UTF16 characters = 32 (including last \0)
    union                       // dotnet provider name
    {
        MagicProvider _magic;
        uint8_t Provider[2 * DotnetProviderMagicLength];
    };
    uint32_t Arguments;         // 0 for empty string (no argument)
};

The code to fill up the command is straightforward:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


StartSessionMessage* CreateStartSessionMessage(uint64_t keywords, EventVerbosityLevel verbosity)
{
    auto message = new StartSessionMessage();
    ::ZeroMemory(message, sizeof(message));
    memcpy(message->Magic, &DotnetIpcMagic_V1, sizeof(message->Magic));
    message->Size = sizeof(StartSessionMessage);  
    message->CommandSet = (uint8_t)DiagnosticServerCommandSet::EventPipe;
    message->CommandId = (uint8_t)EventPipeCommandId::CollectTracing2;
    message->Reserved = 0;
    message->CircularBufferMB = CircularBufferMBSize;
    message->Format = NetTraceFormat;
    message->RequestRundown = 0;
    message->ProviderCount = 1;
    message->Keywords = (uint64_t)keywords;
    message->Verbosity = (uint32_t)verbosity;
    message->ProviderStringLen = DotnetProviderMagicLength;
    memcpy(message->Provider, &DotnetProviderMagic, sizeof(message->Provider));
    message->Arguments = 0;

    return message;
}

The provider list is defined with the ProviderCount field and string containing the list (only one here) follows the Verbosity field. To start the session, it is needed to send the StartSession message and read the session id from the response:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


bool EventPipeStartRequest::Process(IIpcEndpoint* pEndpoint, uint64_t keywords, EventVerbosityLevel verbosity)
{
    // send an StartSessionMessage and parse the response
    StartSessionMessage* pMessage = CreateStartSessionMessage(keywords, verbosity);

    DWORD writtenBytes = 0;
    if (!pEndpoint->Write(pMessage, pMessage->Size, &writtenBytes))
    {
        return false;
    }

    // analyze the response
    IpcHeader response = {};
    DWORD bytesReadCount = 0;
    if (!pEndpoint->Read(&response, sizeof(response), &bytesReadCount))
    {
        return false;
    }

    if (response.CommandId != (uint8_t)DiagnosticServerResponseId::OK)
    {
        return false;
    }

    // get the session ID from the payload
    uint16_t payloadSize = response.Size - sizeof(response);
    if (payloadSize < sizeof(uint64_t))
    {
        return false;
    }

    if (!pEndpoint->ReadLong(SessionId))
    {
        return false;
    }

    return true;
}

Once the StartSession command has been sent, the events corresponding to the given provider/keywords/verbosity (here the CLR runtime/gc+exception+contention/verbose)

1
2
3
4
5
6


auto pSession = pClient->OpenEventPipeSession(
        EventKeyword::gc |
          EventKeyword::exception |
          EventKeyword::contention,
        EventVerbosityLevel::Verbose  // required for AllocationTick
        );

will be read from the event pipe. Since this action will be synchronous, it is recommended to dedicate a thread to read and process the events:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


    if (pSession != nullptr)
    {
        DWORD tid = 0;
        auto hThread = ::CreateThread(nullptr, 0, ListenToEvents, pSession, 0, &tid);

        std::cout << "Press ENTER to stop listening to events...\n\n";
        std::string line;
        std::getline(std::cin, line);

        std::cout << "Stopping session\n\n";
        pSession->Stop();
        std::cout << "Session stopped\n\n";

        // test if it works
        ::Sleep(1000);

        ::CloseHandle(hThread);
    }

The ListenToEvents callback executed by the new thread is “simply” listening to the event pipe of the session:

1
2
3
4
5
6
7
8


DWORD WINAPI ListenToEvents(void* pParam)
{
    EventPipeSession* pSession = static_cast<EventPipeSession*>(pParam);

    pSession->Listen();

    return 0;
}

Before describing how to read the events, it is important to understand how to stop the flow. First, inside the EventPipeSession, the internal loop that reads events needs to exit thanks to the _stopRequested boolean:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


bool EventPipeSession::Stop()
{
    _stopRequested = true;

    if (_pid == -1)
        return true;

    // it is neeeded to use a different ipc connection to stop the Session
    DiagnosticsClient* pStopClient = DiagnosticsClient::Create(_pid, nullptr);
    pStopClient->StopEventPipeSession(SessionId);
    delete pStopClient;

    return true; 
}

In addition, a message with StopTracing command id from the EventPipe command set needs to be sent to tell the CLR to stop sending the events. This message must be sent through a different IPC channel (hence the pStopClient variable used in the previous code. The StopEventPipeSession helper function uses the EventPipeStopRequest wrapper:

1
2
3
4
5


bool DiagnosticsClient::StopEventPipeSession(uint64_t sessionId)
{
    EventPipeStopRequest request;
    return request.Process(_pEndpoint, sessionId);
}

The StopSession command accepts the session ID as single parameter:

1
2
3
4
5


#pragma pack(1)
struct StopSessionMessage : public IpcHeader
{
    uint64_t SessionId;
};

The processing of the stop request is to create such a message and send it through the IPC channel:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32


StopSessionMessage* CreateStopMessage(uint64_t sessionId)
{
    StopSessionMessage* message = new StopSessionMessage();
    ::ZeroMemory(message, sizeof(message));
    memcpy(message->Magic, &DotnetIpcMagic_V1, sizeof(message->Magic));
    message->Size = sizeof(StopSessionMessage);
    message->CommandSet = (uint8_t)DiagnosticServerCommandSet::EventPipe;
    message->CommandId = (uint8_t)EventPipeCommandId::StopTracing;
    message->Reserved = 0;
    message->SessionId = sessionId;

    return message;
}

bool EventPipeStopRequest::Process(IIpcEndpoint* pEndpoint, uint64_t sessionId)
{
    StopSessionMessage* pMessage = CreateStopMessage(sessionId);
    DWORD writtenBytes;
    if (!pEndpoint->Write(pMessage, pMessage->Size, &writtenBytes))
    {
        Error = ::GetLastError();
        std::cout << "Error while sending EventPipe Stop message to the CLR: 0x" << std::hex << Error << std::dec << "\n";
        delete pMessage;
        return false;
    }
    
    delete pMessage;

    ... // handle the response 

    return true;
}

When the stop command is received by the CLR, the remaining “data” (more on this in the next episode) is sent through the first IPC channel before being closed. This is how the code knows that the session can stop listening to the EventPipe.

The next episode will start to parse the nettrace stream of events.

Resources

Episode 1 — Digging into the CLR Diagnostics IPC Protocol in C#
Episode 2 — .NET Diagnostic IPC protocol: the C++ way
Source code for the C++ implementation of CLR events listener
Diagnostics IPC protocol documentation
Microsoft.Diagnostics.NETCore.Client source code

.NET Diagnostic IPC protocol: the C++ way

Sun, 18 Sep 2022 14:36:31 +0000

The previous post was describing the C# helpers to communicate with the diagnostic server in the CLR of a running .NET application.

If, like me, you must write native code (i.e not in C#), you will need to implement the transport and protocol yourself. And, as you will see, it is not that complicated thanks to the documentation but also by using the available C# code of the Microsoft.Diagnostics.NETCore.Client implementation as a guide.

EventPipe transport layer

The first step is to connect to the CLR of a running .NET process. On Linux, you connect to a domain socket named “{$TMPDIR}/dotnet-diagnostic-{%d:PID}-{%llu:disambiguation key}-socket”. For Windows, a named pipe called \.\pipe\dotnet-diagnostic-{%d:PID} needs to be accessed.

Here is the Windows implementation code to connect to the IPC named pipe:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


int BasicConnection(DWORD pid)
{
    wchar_t pszPipeName[256];

    // build the pipe name as described in the protocol
    int nCharactersWritten = -1;
    nCharactersWritten = wsprintf(
        pszPipeName,
        L"\\\\.\\pipe\\dotnet-diagnostic-%d",
        pid
    );

    // check that CLR has created the diagnostics named pipe
    if (!::WaitNamedPipe(pszPipeName, 200))
    {
        auto error = ::GetLastError();
        std::cout << "Diagnostics named pipe is not available for process #" << pid << " (" << error << ")" << "\n";
        return -1;
    }

    // connect to the named pipe
    HANDLE hPipe;
    hPipe = ::CreateFile(
        pszPipeName,    // pipe name
        GENERIC_READ |  // read and write access
        GENERIC_WRITE,
        0,              // no sharing
        NULL,           // default security attributes
        OPEN_EXISTING,  // opens existing pipe
        0,              // default attributes
        NULL);          // no template file

    if (hPipe == INVALID_HANDLE_VALUE)
    {
        std::cout << "Impossible to connect to " << pszPipeName << "\n";
        return -2;
    }

        // ... send a command...

    // don't forget to close the named pipe
    ::CloseHandle(hPipe);

    return 0;
}

Dig into the Command protocol

Once the connection is created, a command can be sent by writing to the pipe (or socket on Linux). The answer will be received by reading from the pipe (or socket on Linux). There is something important to remember: if you need to send different commands, it is needed to create one connection for each. You should not try to reuse a given connection that might also be closed after a command has been processed.

Let’s start with the IpcHeader

Both the command and the response share the same header format:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


#pragma pack(1)
struct IpcHeader
{
    union
    {
        MagicVersion _magic;
        uint8_t  Magic[14];  // Magic Version number; a 0 terminated ANSI char array
    };
    uint16_t Size;       // The size of the incoming packet, size = header + payload size
    uint8_t  CommandSet; // The scope of the Command.
    uint8_t  CommandId;  // The command being sent
    uint16_t Reserved;   // reserved for future use
};

const MagicVersion DotnetIpcMagic_V1 = { "DOTNET_IPC_V1" };

The Size field stores the size of the header (= 20) plus the size of the payload (if any). Next comes the CommandSet field that identifies the groups in which the command belongs to:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


enum class DiagnosticServerCommandSet : uint8_t
{
    // reserved = 0x00,
    Dump        = 0x01,
    EventPipe   = 0x02,
    Profiler    = 0x03,
    Process     = 0x04,

    Server      = 0xFF,
};

It is then followed by the ID of a command in that CommandSet :

For example, here is the memory layout of a ProcessInfo command:

Since there is no additional parameter that would need to be encoded in a payload, the Size field is set to 20 (= 0x14 in hexadecimal) which is the size of the header alone.

To make command handling easier, I have defined the corresponding headers:

1
2
3
4
5
6
7
8


const IpcHeader ProcessInfoMessage =
{
    { DotnetIpcMagic_V1 },
    (uint16_t)sizeof(IpcHeader),
    (uint8_t)DiagnosticServerCommandSet::Process,
    (uint8_t)DiagnosticServerResponseId::OK,
    (uint16_t)0x0000
};

All that makes the the code to send such a ProcessInfo command straightforward:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


bool ProcessInfoRequest::Send(HANDLE hPipe)
{
    // send the request
    IpcHeader message = ProcessInfoMessage;
    DWORD bytesWrittenCount = 0;
    if (!::WriteFile(hPipe, &message, sizeof(message), &bytesWrittenCount, nullptr))
    {
        Error = ::GetLastError();
        std::cout << "Error while sending ProcessInfo message to the CLR: 0x" << std::hex << Error << std::dec << "\n";
        return false;
    }

The next step is to read from the pipe to get the answer from the CLR. As mentioned earlier, the same IpcHeader is received; always with the Server (0xFF) CommandSet value. The CommandId field is 0 for a success and 0xFF in case of an error.

1
2
3
4
5
6


enum class DiagnosticServerResponseId : uint8_t
{
    OK = 0x00,
    // future
    Error = 0xFF,
};

For example, here is the memory layout of a ProcessInfo answer:

Note that numbers are little-endian encoded (hence the 0001 for the Size field: it means 0x0100 = 256 bytes)

The code to analyze the response follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


    // analyze the response
    // 1. get the header to know how large the buffer should be to get the payload
    message = {};
    DWORD bytesReadCount = 0;
    if (!::ReadFile(hPipe, &message, sizeof(message), &bytesReadCount, nullptr))
    {
        Error = ::GetLastError();
        std::cout << "Error while getting ProcessInfo response from the CLR: 0x" << std::hex << Error<< std::dec << "\n";
        return false;
    }

    if (message.CommandId != (uint8_t)DiagnosticServerResponseId::OK)
    {
        Error = message.CommandId;
        std::cout << "Error returned by the CLR in ProcessInfo response: 0x" << std::hex << Error<< std::dec << "\n";
        return false;
    }

In case of success, the size of the response payload is obtained from the Size field by subtracting the size of the header. The next step is to allocate a buffer and read the payload from the pipe into that buffer:

1
2
3
4
5
6
7
8
9


    uint16_t payloadSize = message.Size - sizeof(message);
    _buffer = new uint8_t[payloadSize];
    if (!::ReadFile(hPipe, _buffer, payloadSize, &bytesReadCount, nullptr))
    {
        Error = ::GetLastError();
        std::cout << "Error while getting ProcessInfo payload: 0x" << std::hex << Error << std::dec << "\n";
        return false;
    }
    // Note: bytesReadCount == payloadSize

How to access the response fields

Now that the response payload has been read into a memory buffer, it is time to look at the expected fields as described in the documentation. Here is the encoding of the different field types:

To continue with the ProcessInfo example, here are the expected fields:

So here is the memory layout for the response payload for my monitored Windows 64-bit simulator.exe test application:

To simplify the implementation, the class in charge of a command allocates a buffer corresponding to the payload and provides fields with values either copied from it or, in case of strings, pointing to the buffer (just after the 32-bit length):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


bool ProcessInfoRequest::ParseResponse(DWORD payloadSize)
{
    uint32_t index = 0;
    
    memcpy(&Pid, &_buffer[index], sizeof(Pid));
    index += sizeof(Pid);
    if (payloadSize < index) return false;

    memcpy(&RuntimeCookie, &_buffer[index], sizeof(RuntimeCookie));
    index += sizeof(RuntimeCookie);
    if (payloadSize < index) return false;

    PointToString(_buffer, index, CommandLine);
    if (payloadSize < index) return false;

    PointToString(_buffer, index, OperatingSystem);
    if (payloadSize < index) return false;

    PointToString(_buffer, index, Architecture);

    return true;
}

The PointToString helper code is straightforward:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


void PointToString(uint8_t* buffer, uint32_t& index, wchar_t*& string)
{
    // strings are stored as:
    //    - characters count as uint32_t
    //    - array of UTF16 characters followed by "\0"
    // Note that the last L"\0" IS COUNTED 
    uint32_t count;
    memcpy(&count, &buffer[index], sizeof(count));

    // skip characters count 
    index += sizeof(count);

    // empty string case
    // Note: could make it point to the "count" which is 0 in the buffer
    //       instead of returning nullptr
    if (count == 0)
    {
        string = nullptr;
        return;
    }

    string = (wchar_t*)&buffer[index];

    // skip the whole string (including last UTF16 '\0')
    index += count * (uint32_t)sizeof(wchar_t);
}

You now know how to send a command without parameter and analyze the expected response. The next post will show you how to listen to CLR events like dotnet trace.

Resources

Episode 1 — Digging into the CLR Diagnostics IPC Protocol in C#
Diagnostics IPC protocol documentation
Microsoft.Diagnostics.NETCore.Client source code

Digging into the CLR Diagnostics IPC Protocol in C#

Thu, 28 Jul 2022 08:59:40 +0000

Introduction

As I explained during a DotNext conference session, the .NET CLI tools such as dotnet-trace, dotnet-counter or dotnet-dump are communicating with the CLR thanks to Named Pipe on Windows and Domain Socket on Linux. Within the CLR, a diagnostic server thread is responsible for answering requests. A communication protocol allows a tool to send commands and expect responses. This Diagnostic IPC Protocol is pretty well documented in the dotnet Diagnostics repository.

Before going into the protocol details, here is a list of the available commands and their effect:

This series will detail how to communicate with a CLR using this protocol both in C# and in C++. Also note that processing CLR events thanks to EventPipe will also be covered.

Make it simple: use Microsoft.Diagnostics.NETCore.Client nuget

With TraceEvent nugget package, Microsoft provided a great library to easily listen to CLR events in C#. If you want to easily send CLR diagnostic IPC protocol commands to a CLR in a .NET process, Microsoft.Diagnostics.NETCore.Client nuget package is for you. Remember that EventPipe is implemented by .NET Core and .NET 5+ (so no .NET Framework support)

The Swiss knife class DiagnosticsClient gives you access to most of the commands plus a way to list .NET processes as a bonus:

If you want to get the pid of all supported running .NET applications, call the static GetPublishedProcesses() method. Beware that the pid of your own application will also be included.

1
2
3
4
5
6
7
8
9


private static void ListProcesses()
{
    var selfPid = Process.GetCurrentProcess().Id;
    foreach (var pid in DiagnosticsClient.GetPublishedProcesses())
    {
        var process = Process.GetProcessById(pid);
        Console.WriteLine($"{pid,6}{GetSeparator(pid == selfPid)}{process.ProcessName}");
    }
}

Otherwise, create an instance passing the process ID of the .NET application you are interested in. With this object, call the method corresponding to the command you want to send. For example, the following code is calling GetProcessEnvironment() to list the environment variables:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


private static void ListEnvironmentVariables(int pid)
{
    // get environment variables via existing wrapper in DiagnosticsClient
    var client = new DiagnosticsClient(pid);
    var envVariables = client.GetProcessEnvironment();
    foreach (var variable in envVariables.Keys.OrderBy(k => k))
    {
        Console.WriteLine($"{variable,26} = {envVariables[variable]}");
    }
}

Note that the value “ExitCode=00000000” is associated to the “” (empty) key for reason unknown to me…

Even though the undocumented command to set an environment variable is available via the SetEnvironmentVariable() method, there is no helper method wrapping the ProcessInfo command. In fact, a GetProcessInfo() method exists but it is internal! The PidIpcEndpoint type in charge of the transport and the IpcMessage, IpcResponse and IpcClient types dealing with commands are also internal. It means that the nuget will not help if you need to send the ProcessInfo command.

Still easy: use Microsoft.Diagnostics.NETCore.Client source code

The .NET team spends some extra time testing, documenting, and verifying they are happy with the APIs in NetCore.Client before making them public, so sometimes you will see types that they used in their own tools that are still internal. But wait, if the CLI tools need some of these types, how will it work? Well…

The C# project corresponding to the MicrosoftDiagnostics.NETCore.Client assembly is part of the dotnet Diagnostic repository where the tools are implemented. If you look at the .csproj file, you will see InternalsVisibleTo attributes to allow the tools to access the internal types:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


  
     Include="dotnet-counters" />
     Include="dotnet-dsrouter" />
     Include="dotnet-monitor" />
     Include="dotnet-trace" />
     Include="Microsoft.Diagnostics.Monitoring" />
     Include="Microsoft.Diagnostics.Monitoring.EventPipe" />
    
     Include="Microsoft.Diagnostics.Monitoring.WebApi" />
     Include="Microsoft.Diagnostics.NETCore.Client.UnitTests" />
  

The great thing about OSS is that you can compile your own fork to make these types public. Of course you will be on your own to support these custom builds of the library and it is possible there will be changes to the API before .NET makes it public.

So what you could do to use these internal types in your code is the following:

copy the folder from the Diagnostics repository
add the name of your assembly that needs to access the internal types and members into the .csproj
replace the reference to the nuget package by a project reference to the copied project

And now GetProcessInfo and the other internal types are public for you:

1
2
3
4
5
6
7
8
9


private static void ListProcessInfo(int pid)
{
    var client = new DiagnosticsClient(pid);
    var info = client.GetProcessInfo();  // this method is internal
    Console.WriteLine($"              Command Line = {info.CommandLine}");
    Console.WriteLine($"              Architecture = {info.ProcessArchitecture}");
    Console.WriteLine($"      Entry point assembly = {info.ManagedEntrypointAssemblyName}");
    Console.WriteLine($"               CLR Version = {info.ClrProductVersionString}");
}

Note that during my tests, I was able to get a value for the ManagedEntrypointAssemblyName or ClrProductVersionString properties only with .NET 6+: the ProcessInfo2 (0x404) command does not seem to be implemented in previous versions.

The next episode of the series will start to explain the EventPipe IPC protocol from a native C++ developer perspective.

Resources

Microsoft.Diagnostics.NETCore.Client nuget package
TraceEvent nugget package
Diagnostics IPC protocol documentation

Reading parameters value with the .NET Profiling APIs

Tue, 16 Nov 2021 09:02:44 +0000

Introduction

From the list of arguments with their type, it becomes possible to figure out their value when a method gets called. The rest of this post describes how to access method call parameters and get the value of numbers and strings.

Where are my parameters?

When you pass COR_PRF_ENABLE_FUNCTION_ARGS to ICorProfilerInfo::SetEventMask, the runtime prepares a COR_PRF_FUNCTION_ARGUMENT_INFO structure before your enter callback is called:

1
2
3
4
5


typedef struct _COR_PRF_FUNCTION_ARGUMENT_INFO {  
    ULONG numRanges;  
    ULONG totalArgumentSize;  
    COR_PRF_FUNCTION_ARGUMENT_RANGE ranges[1];  
} COR_PRF_FUNCTION_ARGUMENT_INFO;

I have to admit that the Microsoft Docs did not really help me to figure out what is the meaning of each field of this structure because the word “range” is very confusing here…

Based on my experiments, numRanges gives you the number of parameters; including the implicit this parameter in case of a non-static method. It is different from the signature that we have already parsed from the metadata where this is not mentioned. The ranges fields is an array of COR_PRF_FUNCTION_ARGUMENT_RANGE ; one per parameter:

1
2
3
4


typedef struct _COR_PRF_FUNCTION_ARGUMENT_RANGE {  
    UINT_PTR startAddress;  
    ULONG length;  
} COR_PRF_FUNCTION_ARGUMENT_RANGE;

The startAddress points to where the parameter value is stored in memory.

However, in addition to the FunctionID, you only receive a COR_PRF_ELT_INFO in your enter callback. You need to call ICorProfilerInfo3:: GetFunctionEnter3Info to get the corresponding COR_PRF_FUNCTION_ARGUMENT_INFO you are interested in. As often with COM, you need to call a first time to get the size of the buffer to allocate and a second time to fill it up:

1
2
3
4
5
6


ULONG argumentInfoSize = 0;
COR_PRF_FRAME_INFO frameInfo;
_pInfo->GetFunctionEnter3Info(functionId, eltInfo, &frameInfo, &argumentInfoSize, NULL);
byte* pBuffer = new byte[argumentInfoSize];
_pInfo->GetFunctionEnter3Info(functionId, eltInfo, &frameInfo, &argumentInfoSize, (COR_PRF_FUNCTION_ARGUMENT_INFO*)pBuffer);
COR_PRF_FUNCTION_ARGUMENT_INFO* pArgumentInfo = (COR_PRF_FUNCTION_ARGUMENT_INFO*)pBuffer;

Before iterating on the parameters, you need to deal with non-static method and their implicit this parameter stored in pArgumentInfo->ranges[0]:

1
2
3
4
5
6
7
8


ULONG hiddenThisParameterIndexOffset = 0;
if (!pSignature->IsStatic)
{
   hiddenThisParameterIndexOffset++;

   // deal with the "this" hidden parameter for non static method
   // ex: show its address (i.e. pArgumentInfo->ranges[0].startAddress)
}

Next, write a loop to iterate on each parameter based on the parameter count obtained previously from the metadata:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


char value[128];
for (ULONG currentParameterInSignature = 0; currentParameterInSignature < parameterCount; currentParameterInSignature++)
{
   // Note: pParameter contains detail extracted from the metadata signature

   UINT_PTR pStartValue = pArgumentInfo->ranges[currentParameterInSignature + hiddenThisParameterIndexOffset].startAddress;
   ULONG length = pArgumentInfo->ranges[currentParameterInSignature + hiddenThisParameterIndexOffset].length;

   if (IsPdOut(pParameter->Attributes))
   {
      // if [out] parameter, nothing to get from it
   }
   else
   {
      value[0] = '\0';
      // call a helper function to extract the value of the parameter
      // a string from its address and type 
      pHelpers->GetObjectValue(pStartValue, length, pParameter->ElementType , pParameter->TypeToken, pSignature->ModuleId, value, ARRAY_LEN(value) - 1);
   }
}

Simple type parameters case

The GetObjectValue() helper function looks like the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


void CorProfilerHelpers::GetObjectValue(UINT_PTR address, ULONG length, ULONG elementType, mdToken elementTypeToken, ModuleID moduleId, char* value, ULONG charCount)
{
   ULONG numberValue;
   strcpy_s(value, charCount, "???");
   switch(elementType)
   {
      ...

      default:
         sprintf_s(value, charCount, "unknown type 0x%x", elementType);
      break;
   }

The way to get the value of a parameter really depends on its type. I know that a length is provided by the COR_PRF_FUNCTION_ARGUMENT_INFO structure but I did not used it except for sanity check.

The value for simple types are easy to compute because they are mostly stored at the given address :

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101


case ELEMENT_TYPE_BOOLEAN:
{
   bool* pBool = (bool*)address;
   if (*pBool)
      strcpy_s(value, charCount, "true");
   else
      strcpy_s(value, charCount, "false");
}
break;

case ELEMENT_TYPE_CHAR:
{
   WCHAR* pChar = (WCHAR*)address;
   sprintf_s(value, charCount, "%C", *pChar);
}
break;

case ELEMENT_TYPE_I1:
// int8
{
   char* pNumber = (char*)address;
   numberValue = *pNumber;
   sprintf_s(value, charCount, "%d", numberValue);
}
break;

case ELEMENT_TYPE_U1:
// unsigned int8
{
   unsigned char* pNumber = (unsigned char*)address;
   numberValue = *pNumber;
   sprintf_s(value, charCount, "%d", numberValue);
}
break;

case ELEMENT_TYPE_I2:
// int16
{
   short int* pNumber = (short int*)address;
   numberValue = *pNumber;
   sprintf_s(value, charCount, "%d", numberValue);
}
break;

case ELEMENT_TYPE_U2:
// unsigned int16
{
   short unsigned int* pNumber = (short unsigned int*)address;
   numberValue = *pNumber;
   sprintf_s(value, charCount, "%d", numberValue);
}
break;

case ELEMENT_TYPE_I4:
// int32
{
   __int32* pNumber = (__int32*)address;
   numberValue = *pNumber;
   sprintf_s(value, charCount, "%d", numberValue);
}
break;

case ELEMENT_TYPE_U4:
// unsigned int32
{
   unsigned __int32* pNumber = (unsigned __int32*)address;
   numberValue = *pNumber;
   sprintf_s(value, charCount, "%d", numberValue);
}
break;

// NOTE: %lld might not work on linux
case ELEMENT_TYPE_I8:
// int64
{
   __int64* pNumber = (__int64*)address;
   sprintf_s(value, charCount, "%lld", *pNumber);
}
break;

case ELEMENT_TYPE_U8:
{
// unsigned int64
   unsigned __int64* pNumber = (unsigned __int64*)address;
   sprintf_s(value, charCount, "%lld", *pNumber);
}
break;
And guess what? It is the same for float and double because it is stored by the CLR the same way as in C++:
case ELEMENT_TYPE_R4:
{
   float* pFloat = (float*)address;
   sprintf_s(value, charCount, "%f", *pFloat);
}
break;

case ELEMENT_TYPE_R8:
{
   double* pDouble = (double*)address;
   sprintf_s(value, charCount, "%g", *pDouble);
}
break;

The other types require more knowledge about how the CLR stores their value.

The string case

This is the first reference type we meet and, as for all reference types, the given address points to the memory where the reference (i.e. address of the object in the managed heap) is stored. It allows you to check against null parameter before looking at the “real” managed reference:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


case ELEMENT_TYPE_STRING:
{
   // look at the reference stored at the given address
   unsigned __int64* pAddress = (unsigned __int64*)address;
   byte* managedReference = (byte*)(*pAddress);

   // easily check for null string
   if (managedReference == NULL)
   {
      strcpy_s(value, charCount, "null string");
      break;
   }

At that point, you need to know how an instance of a reference type instance is stored by the CLR in the managed heap. Hopefully, Sergey Tepliakov, a software engineer at Microsoft, has provided a lot of details about that, especially where does the address stored by a managed reference point to:

It means that you have to skip the Method Table pointer pointed to by the address you have. This applies to any reference types instance!

But for our string current case, you still need to know how a string is stored (i.e. its length followed by the buffer of UTF16 characters). I recommend that you read the post from Matt Warren about the subject because it also covers a lot of interesting details related to string implementation. However, you should simply rely on the implementation details provided by the CLR via ICorProfilerInfo3:: GetStringLayout2:

1
2
3


ULONG _stringLengthOffset;
ULONG _stringBufferOffset;
hr = _pProfilerInfo->GetStringLayout2(&_stringLengthOffset, &_stringBufferOffset);

These two variables give you the offsets to use to access both the string size and the beginning of the array of WCHAR storing each character.

1
2
3
4
5
6
7
8
9


// -----------------------------------------------------------------
//    | MethodTable address | string length | buffer 
// -----------------------------------------------------------------
// 64       8 bytes              4 bytes      (length+1) x WCHAR
// off                           8            12
// -----------------------------------------------------------------
// 32       4 bytes              4 bytes      (length+1) x WCHAR
// off                           4            8
// -----------------------------------------------------------------

As shown in this table, you need to skip 8/4 bytes to read the length. It is just the confirmation that you need to jump over the address of the Method Table stored as a 64/32 bit value (i.e. an address in x64/x86). The length itself is stored as a 32 bit number (4 bytes) both in x64 an x86. So the array containing the consecutive UTF16 characters just follows (i.e. its offset from the reference address is 12/8 bytes). For example, here is what you get in Visual Studio Memory panel with the reference to the 3 characters “CLR” string for a 64 bit application:

With this information in hand, it is easy to detect empty strings or copy the UNICODE string into a simple char* buffer:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


byte* pLength = GetPointerAfterNBytes(managedReference, _stringLengthOffset);
ULONG stringLength = *pLength;
if (stringLength == 0)
{
   strcpy_s(value, charCount, "empty string");
   break;
}

byte* pBuffer = GetPointerAfterNBytes(managedReference, _stringBufferOffset);
//                                                               V-- to copy the trailing \0
::WideCharToMultiByte(CP_ACP, 0, (WCHAR*)pBuffer, stringLength + 1, value, charCount, NULL, NULL);

The GetPointerAfterNBytes function simply helps me dealing with pointer arithmetic in C++

1
2
3
4


byte* GetPointerAfterNBytes(void* pAddress, ULONG byteCount)
{
   return ((byte*)pAddress) + byteCount;
}

The next post will describe how to get the value of array parameters and the basics of extracting fields value from a reference type instance.

References

Sample: A Signature Blob Parser for your Profiler
Strings and the CLR — a Special Relationship by Matt Warren
Episode 1: Start a journey into the .NET Profiling APIs
Episode 2: Dealing with Modules, Assemblies and Types with CLR profiling API
Episode 3: Decyphering methods signature with .NET Profiling APIs

Start a journey into the .NET Profiling APIs

Sat, 07 Aug 2021 12:21:39 +0000

Introduction

When I want to dig into a new API, I implement a real world scenario. This is exactly what I did for the .NET native profiling API. I want to know how to get parameters and return value of any method call during any .NET application life. The expected result would be something like :

Enter PublicClass::ClassParamReturnClass
this = 0x6f97e190 (8)
ClassType obj = 0x6f97e488 (8)
| int32 k__BackingField = 84
| int32 intField = 42
| String stringField = 43
ClassType obj = 0x0000023A475BBAD8

Leave PublicClass::ClassParamReturnClass
| int32 k__BackingField = 170
| int32 intField = 85
| String stringField = 86
returns 0x0000023A475BBBB0

when the following method is executed:

1
2
3
4


public ClassType ClassParamReturnClass(ClassType obj)
{
   return new ClassType(obj.IntProperty + 1);
}

You will have to write native C/C++ code to leverage the .NET Profiling API as John Robbins explained in his 2003 Debugging Applications book. Even though Microsoft is providing a code sample for that, it does just show how to get notified when a function is called or exited but nothing about its type, its name, what are its parameters value and its return value. This series will detail both the .NET profiling API and the metadata API (i.e. the native reflection API).

Let’s start with the basics of .NET Profiling. As shown in the following figure from the Microsoft Profiling documentation, with the right environment configuration, the CLR will load a COM-like object implementing ICorProfilerCallback interface to notify almost everything happening in a .NET application from startup to shutdown.

Since .NET was launched, more and more notifications have been added by versioning the interface up to ICorProfilerCallback9. Welcome to the usual COM world! I recommend that you watch Pavel Yosifovich session about writing a CLR Profiler in an hour to get an overview. I will use his sample solution as a starting point.

For performance sake, you tell the runtime which events you are interested in; i.e. which ICorProfilerCallback functions will be called and you simply have to return S_OK from all other functions. This setup is done in your implementation of ICorProfilerCallback::Initialize. This first function called by the CLR provides a parameter from which you need to QueryInterface a version of ICorProfilerInfo interface (the current one is ICorProfilerInfo10). This interface provides functions to query information about parameters passed to your ICorProfilerCallback functions (such as AppDomain, assembly, type, function, thread and so on).

The first ICorProfilerInfo function you will use is SetEventMask to filter the ICorProfilerCallback functions that will be called by the runtime. It accepts a flag combination of values from the COR_PRF_MONITOR enumeration.

Assembly code is needed for Enter/Leave/TailCall

To get notified when a managed method is called or exits, you should pass:

COR_PRF_MONITOR_ENTERLEAVE | 
COR_PRF_ENABLE_FUNCTION_ARGS | 
COR_PRF_ENABLE_FUNCTION_RETVAL | 
COR_PRF_ENABLE_FRAME_INFO

to ICorProfilerInfo::SetEventMask. The first flag tells the runtime to call static callbacks (i.e. not exposed as functions of ICorProfilerCallback) when a managed method gets executed or returns. The other three flags ensure that these callbacks will receive enough information to extract method arguments and return value.

Unlike the other notifications that end up calling your ICorProfilerCallback functions, you need to register three special callbacks to the runtime via ICorProfilerInfo3::SetEnterLeaveFunctionHooks3WithInfo. For performance reasons, the .NET team asks you to write the prolog and epilog (i.e. saving/restoring CPU registers on/from the stack) yourself in assembly code instead of relying on well-defined calling conventions supported by the C/C++ compiler.

This is why you have to call ICorProfilerInfo3:: SetEnterLeaveFunctionHooks3WithInfo and pass pointers to these “naked” functions. The Microsoft ELTProfiler sample implements the stubs both for x86 (as inlined assembly embedded in a C++ file) and x64 (defined in .asm file). In x64, you need to update your project file to add the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


 Label = "ExtensionSettings">
    Project = "$(VCTargetsPath)\BuildCustomizations\masm.props" / >


 Include = "../DotNext.Profiler.Shared/asm/windows/nakedcallbacks.asm"
	Condition = "'$(Platform)' == 'x64'" / >

 Label = "ExtensionTargets">
     Project = "$(VCTargetsPath)\BuildCustomizations\masm.targets" / >

The nakedcallbacks.asm file contains the assembly code to call stub functions wrapped by the expected prolog and epilog written in assembly code. Here is the signature of the functions from where you will be able to start working in C++:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


PROFILER_STUB EnterStub(FunctionIDOrClientID functionId, COR_PRF_ELT_INFO eltInfo)
{
   ...
}

PROFILER_STUB LeaveStub(FunctionID functionId, COR_PRF_ELT_INFO eltInfo)
{
   ...
}

PROFILER_STUB TailcallStub(FunctionID functionId, COR_PRF_ELT_INFO eltInfo)
{
   ...
}

A function is identified by a FunctionID and this is where you start your adventure (I will come back later to the COR_PRF_ELT_INFO parameter). Note that for the EnterStub function, you need to get the FunctionID from the FunctionIDOrClientID.functionId field.

Once your hook callbacks have been registered via ICorProfilerInfo3:: SetEnterLeaveFunctionHooks3WithInfo, it is still possible to decide whether or not a managed method call should trigger them. For that, it is needed to register a “mapper” function that is called once per managed method with one of these functions:

HRESULT ICorProfilerInfo::SetFunctionIDMapper([in] FunctionIDMapper* pFunc); with UINT_PTR __stdcall Mapper(FunctionID functionId, BOOL* pHookFunction)
HRESULT ICorProfilerInfo3::SetFunctionIDMapper2([in] FunctionIDMapper2* pFunc, [in] void* clientData); with UINT_PTR __stdcall Mapper2(FunctionID functionId, void* clientData, BOOL* pHookFunction)

The latter allows you to pass some “client data” to the mapper function such as a helper class to manipulate the received FunctionID or your profiler state.

If pHookFunction is set to TRUE, your enter/leave functions will be called and the returned UINT_PTR will be passed as the FunctionID parameter. This allows you to handle function name or signature computation at one single place outside of the real profiling work done each time a method is called. If pHookFunction is set to FALSE, the enter/leave functions will never be called for that FunctionID. The mapper callback is called only once per FunctionID: this could be a good way to avoid performance impact if you just want to profile a small subset of methods.

How to debug your profiler

Before going any further, it is needed to know how to debug your C++ profiler code with Visual Studio. The first step is to write a simple .NET application to execute the method calls you want to intercept. The second natural step would be to setup the environment variables needed to inject your profiler:

and also check the Enable native code debugging option:

If you start a debug session, the Visual Studio debugger with use both managed and native debugging APIs. Unfortunately, the managed debugging API does not allow breakpoints sets in ICorProfilerCallback functions.

Instead, you need to Debug | Start Without Debugging (CTRL+F5), not Debug | Start Debugging (F5) the C# test application with the same environment variables and attach the debugger via Debug | Attach to Process. Select the process and click Select button in the Attach to section:

To make attachment simple, add a Console.ReadLine in a console application before starting the method calls to test.

The next post will show you how to extract information from a FunctionID.

References

Writing a .NET Core cross platform profiler in an hour video and the corresponding source code from Pavel Yosifovich
Microsoft Profiling documentation
Microsoft Enter/Leave code sample

Build your own .NET memory profiler in C# — call stacks (2/2–1)

Mon, 18 May 2020 12:07:01 +0000

In the previous episode of this series, you have seen how to get a sampling of .NET application allocations thanks to the AllocationTick and GCSampleObjectAllocation(High/Low) CLR events. However, this is often not enough to investigate unexpected memory consumption: you would need to know which part of the code is triggering the allocations. This post explains how to get the call stack corresponding to the allocations, again with CLR events.

Introduction

If you look carefully at the payload of the TraceEvent object mapped by Microsoft TraceEvent library (not my fault if they have the same name) for each CLR event, you won’t see anything related to a call stack. However, in the TraceEvent sample 41, the following line looks promising:

var callStack = data.CallStack();

with data being a TraceEvent object received for each CLR event!

This CallStack method is an extension method provided by the TraceLog special kind of event source. You might not have noticed but I have used it in the AllocationTick code sample from the previous post. This class (and many more helper classes) is doing a lot of work to :

“attach” a call stack to each CLR event; i.e. a list of addresses of assembly code
to translate addresses into string symbols (method names or full signatures), listen to a bunch of JIT related events for managed methods (more on this later), using COM-based Debug Interface Access (a.k.a. DIA) and MetadataReaderProvider** **for native functions

Notice that since events from all managed processes on the machine are handled by TraceLog, the internal cache for JITted methods description could consume a lot of memory. During my tests with two Visual Studio running, my test profiler consumed more than 500 MB before even handling call stacks. If you are in such an environment with multiple .NET processes, I will show how to “manually” get the same stacks (+ symbols in the next episode) with CLR events and a few methods from dbghelp.dll in a cheaper way.

The new provider (more on ClrRundown later), keywords and events need to be received to make all this work:

TraceLog: the easy way

As you have seen in the previous posts, the TraceEventSession class exposes a Source property of ETWTraceEventSource type. This source has event parsers properties from which you register handler methods that will be called when CLR events are received. Instead of directly using this source, you should wrap it with a TraceLogEventSource object that provides the same event parsers.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


await Task.Factory.StartNew(() =>
{
    using (_session)
    {
        SetupProviders(_session);

        using (TraceLogEventSource source = TraceLog.CreateFromTraceEventSession(_session))
        {
            SetupListeners(source);

            source.Process();
        }
    }
});

What’s new with providers?

The code for mySetupProviders method is a little bit different from the previous post even though no new event listeners are needed:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


private void SetupProviders(TraceEventSession session)
{
    // Note: the kernel provider MUST be the first provider to be enabled
    // If the kernel provider is not enabled, the callstacks for CLR events are still received 
    // but the symbols are not found (except for the application itself)
    // TraceEvent implementation details triggered when a module (image) is loaded
    session.EnableKernelProvider(
        KernelTraceEventParser.Keywords.ImageLoad |
        KernelTraceEventParser.Keywords.Process,
        KernelTraceEventParser.Keywords.None
    );

    session.EnableProvider(
        ClrTraceEventParser.ProviderGuid,
        TraceEventLevel.Verbose,    // this is needed in order to receive AllocationTick_V2 event
        (ulong)(
        // required to receive AllocationTick events
        ClrTraceEventParser.Keywords.GC |
        ClrTraceEventParser.Keywords.Jit |                      // Turning on JIT events is necessary to resolve JIT compiled code 
        ClrTraceEventParser.Keywords.JittedMethodILToNativeMap |// This is needed if you want line number information in the stacks
        ClrTraceEventParser.Keywords.Loader |                   // You must include loader events as well to resolve JIT compiled code.
        ClrTraceEventParser.Keywords.Stack
        )
    );

    // this provider will send events of already JITed methods
    session.EnableProvider(ClrRundownTraceEventParser.ProviderGuid, TraceEventLevel.Informational,
    (ulong)(
        ClrTraceEventParser.Keywords.Jit |              // We need JIT events to be rundown to resolve method names
        ClrTraceEventParser.Keywords.JittedMethodILToNativeMap | // This is needed if you want line number information in the stacks
        ClrTraceEventParser.Keywords.Loader |           // As well as the module load events.  
        ClrTraceEventParser.Keywords.StartEnumeration   // This indicates to do the rundown now (at enable time)
        ));

}

The kernel provider needs to be enabled with the ImageLoad and Process keywords in order to let TraceEvent detect when a process loads “images” (i.e. dlls) and at which address (needed to convert Relative Virtual Addresses (RVA) to addresses in the address space). Note that this provider must be enabled before any other provider or your code will trigger an exception.
The CLR provider needs to be enabled with Jit, JittedMethodILToNativeMap, and Loader (in addition to the usual GC one).
The Stack keyword has to be set on the same CLR provider to receive call stacks events for “normal” CLR event (more on this later)
The CLR Rundown provider is enabled with the same Jit, JittedMethodILToNativeMap, and Loader keywords. That way, JIT events corresponding to already JITted methods will be received (not only the new ones). This is important because otherwise, you won’t be able to map these methods with the address in memory of their JITted native code in the case of processes that have been started before the profiler. This is the case for my AllocationTickProfiler sample.

Callstacks and symbols

Now, when an AllocationTick event is received, calling the CallStack extension method on the GCAllocationTickTraceData argument returns a TraceCallStack object. This class is a linked list of TraceCodeAddress representing each stack frame (i.e. address in assembly code). These classes are at the heart of TraceEvent and Perfview callstack management. The method names and signatures are retrieved behind the scene thanks to JIT events and the SymbolReader class that digs into .pdb files.

You first need to initialize a SymbolReader instance:

Set the path to find the .pdb; including the Microsoft HTTP endpoint for public .NET versions symbols,
Allow pdb next to the executable to be loaded.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


// By default a symbol Reader uses whatever is in the _NT_SYMBOL_PATH variable.  However you can override
// if you wish by passing it to the SymbolReader constructor.  Since we want this to work even if you 
// have not set an _NT_SYMBOL_PATH, so we add the Microsoft default symbol server path to be sure/
var symbolPath = new SymbolPath(SymbolPath.SymbolPathFromEnvironment).Add(SymbolPath.MicrosoftSymbolServerPath);
_symbolReader = new SymbolReader(_symbolLookupMessages, symbolPath.ToString());

// By default the symbol reader will NOT read PDBs from 'unsafe' locations (like next to the EXE)  
// because hackers might make malicious PDBs. If you wish ignore this threat, you can override this
// check to always return 'true' for checking that a PDB is 'safe'.  
_symbolReader.SecurityCheck = (path => true);

Then, displaying a TraceCallStack from a received CLR event in a human-readable format is simple:

Get one frame after the other from the linked list,
If the CodeAddress field is not cached yet, load the symbols for its module,
Display the FullMethodName field of the frame (or the address if not found).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


private void DumpStack(TraceCallStack callStack)
{
    while (callStack != null)
    {
        var codeAddress = callStack.CodeAddress;
        if (codeAddress.Method == null)
        {
            var moduleFile = codeAddress.ModuleFile;
            if (moduleFile == null)
            {
                Debug.WriteLine($"Could not find module for Address 0x{codeAddress.Address:x}");
            }
            else
            {
                codeAddress.CodeAddresses.LookupSymbolsForModule(_symbolReader, moduleFile);
            }
        }
        if (!string.IsNullOrEmpty(codeAddress.FullMethodName))
            Console.WriteLine($"     {codeAddress.FullMethodName}");
        else
            Console.WriteLine($"     0x{codeAddress.Address:x}");
        callStack = callStack.Caller;
    }
}

Note that the first frame in the linked list is the last on the stack (i.e. last executed method).

As I mentioned at the beginning of the post, I have been facing OutOfMemory errors due to the TraceEvent symbols management large memory usage when a few other .NET applications were running. Let’s see how to get the call stacks in a less memory consuming way.

Manually rebuilding the allocations call stack

Instead of using the call stack and symbol management provided by TraceLog in TraceEvent, I would prefer to manually get them. If you remember the last post, thanks to GCSampledObjectAllocation CLR events, it is possible to have a sampling of the allocation size and count per process and per type. What I would like to add to the type picture is the list of call stacks leading to these allocations.

How to manually get CLR events call stack

The first step is to understand how to get the CLR events call stacks. If you use the TraceLog-based code just presented, you should see the following kind of call stack:

The ETWCallout CLR helper function is in charge of sending a special event containing the call stack of other normal events from the four supported CLR providers. If you set the Stack keyword to the CLR provider, each time an event is sent by a thread, a ClrStackWalk event will be sent just after. It means after each SampleObjectAllocation event, a ClrStackWalk event containing the call stack will be immediately received. In fact, since an application will probably be using more than one thread, it is required to do the mapping between the two events on a per-thread basis.

Each allocation event received by the OnSampleObjectAllocation handler contains the ThreadID property so it is easy to keep track of the last received allocation event per thread. In my case, the ProcessAllocations class stores this information in its _perThreadLastAllocation field:

1
2
3
4
5


public class ProcessAllocations
{
    ...
    private readonly Dictionary<string, AllocationInfo> _allocations;
    private readonly Dictionary<int, AllocationInfo> _perThreadLastAllocation;

Now, each time a SampleObjectAllocation event is received, the id of the sending thread is passed to the updatedProcessAllocations.AddAllocation() method:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


public AllocationInfo AddAllocation(int pid, ulong size, ulong count, string typeName)
{
    if (!_allocations.TryGetValue(typeName, out var info))
    {
        info = new AllocationInfo(typeName);
        _allocations[typeName] = info;
    }

    info.AddAllocation(size, count);

    // the last allocation is still here without the corresponding stack
    if (_perThreadLastAllocation.TryGetValue(pid, out var lastAlloc))
    {
        Console.WriteLine("no stack for the last allocation");
    }

    // keep track of the allocation for the given thread
    // --> will be used when the corresponding call stack event will be received
    _perThreadLastAllocation[pid] = info;

    return info;
}

The _perThreadLastAllocation dictionary stores the AllocationInfo per thread. If an allocation happens, it is added into the dictionary. When a ClrStackWalk event is received for a given thread, the stack will be associated with the last AllocationInfo and removed from the dictionary. If some events are missed (it never happens during my tests but who knows), error message could be logged.

The ClrStackWalkTraceData argument received by the ClrStackWalk listener has a FrameCount property that returns the number of frames in the call stack. In addition, its InstructionPointer() method takes a frame position in the stack (starting at 0) and returns the address (in assembly code) at this position on the call stack.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


private void OnClrStackWalk(ClrStackWalkTraceData data)
{
    if (FilterOutEvent(data)) return;

    var callstack = BuildCallStack(data);
    GetProcessAllocations(data.ProcessID).AddStack(data.ThreadID, callstack);
}
private AddressStack BuildCallStack(ClrStackWalkTraceData data)
{
    var length = data.FrameCount;
    AddressStack stack = new AddressStack(length);

    // frame 0 is the last frame of the stack (i.e. last called method)
    for (int i = 0; i < length; i++)
    {
        stack.AddFrame(data.InstructionPointer(i));
    }

    return stack;
}

The AddressStack class returned by BuildCallStack stores the frames as a list of addresses so it can be stored in AllocationInfo.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36


public class AddressStack
{
    // the first frame is the address of the last called method 
    private readonly List<ulong> _stack;

    public AddressStack(int capacity)
    {
        _stack = new List<ulong>(capacity);
    }

    // No need to override GetHashCode because we don't want to use it as a key in a dictionary
    public override bool Equals(object obj)
    {
        if (obj == null) return false;

        var stack = obj as AddressStack;
        if (stack == null) return false;

        var frameCount = _stack.Count;
        if (frameCount != stack._stack.Count) return false;

        for (int i = 0; i < frameCount; i++)
        {
            if (_stack[i] != stack._stack[i]) return false;
        }

        return true;
    }

    public IReadOnlyList<ulong> Stack => _stack; 

    public void AddFrame(ulong address)
    {
        _stack.Add(address);
    }
}

This class overrides the Equals method for a single reason: I want to be able to detect when the “same” stack (i.e. with the exact same frame addresses) is received for a given type allocation. That way, I just need to keep a counter for each different AddressStack and not all call stacks in AllocationInfo. Remember that AllocationInfo is used to keep track of allocations per type:

1
2
3
4
5
6


public class AllocationInfo
{
    private readonly string _typeName;
    private ulong _size;
    private ulong _count;
    private List<StackInfo> _stacks;

The StackInfo class contains an AddressStack and how many times it led to this type of allocation.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


public class StackInfo
{
    private readonly AddressStack _stack;
    public ulong Count;

    internal StackInfo(AddressStack stack)
    {
        Count = 0;
        _stack = stack;
    }

    public AddressStack Stack => _stack;
}

So, when a stack event is received, AddStack is called on the last AllocationInfo for the same thread:

1
2
3
4
5
6
7
8
9


public void AddStack(int tid, AddressStack stack)
{
    if (_perThreadLastAllocation.TryGetValue(tid, out var lastAlloc))
    {
        lastAlloc.AddStack(stack);
        _perThreadLastAllocation.Remove(tid);
        return;
    }
}

The job of AllocationInfo.AddStack() the method is to check if a previous allocation was made with the same call stack (hence the Equals override). If this is the case, just increment the corresponding StackInfo count. Otherwise, create a new StackInfo for this call stack with a count set to 1.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


internal void AddStack(AddressStack stack)
{
    var info = GetInfo(stack);
    if (info == null)
    {
        info = new StackInfo(stack);
        _stacks.Add(info);
    }

    info.Count++;
}

private StackInfo GetInfo(AddressStack stack)
{
    for (int i = 0; i < _stacks.Count; i++)
    {
        var info = _stacks[i];
        if (stack.Equals(info.Stack)) return info;
    }

    return null;
}

Knowing the address in code of each frame for all events call stack is nice but it would be much more useful to translate them into method names… You have to deal with two different cases: managed and native methods. I will cover these topics in the next episode.

Resources

Source code available on Github.
TraceEvent sample 41 source code.

Missed the first part of this story? Check this out:

Build your own .NET memory profiler in C# *This post explains how to collect allocation details by writing your own memory profiler in C#.*medium.com

Interested in joining our journey? Check this out:

Product, Research & Development | Criteo Careers careers.criteo.com

Build your own .NET memory profiler in C# — Allocations (1/2)

Sat, 18 Apr 2020 09:48:53 +0000

In a previous post, I explained how to get statistics about the .NET Garbage Collector such as suspension time or generation sizes. But what if you would need more details about your application allocations such as how many times instances of a given type were allocated and for what cumulated size or even the allocation rate? This post explains how to get access to such information by writing your own memory profiler. The next one will show how to collect each sampled allocation stack trace.

Introduction

I have already used commercial tools to get detailed information about allocated type instances in an application; Visual Studio Profiler, dotTrace, ANTS memory profiler, or Perfview to name a few. With these tools in mind, I started to look at the .NET profiler API documentation and it reminded me the first time I read about the .NET profiler API. It was in December 2001 in Matt Pietrek’s MSDN Magazine article (I still have the paper version). When your application is starting, based on an environment variable, the .NET Framework (and now .NET Core) runtime is loading a profiler COM object that implements a specific ICorProfilerCallback interface (today, runtimes are supporting the 9th version ICorProfilerCallback9 interface). The methods of this interface will be called by the runtime at specific moments during the application lifetime. For example, the ObjectAllocated method is called each time an instance of a class is allocated: perfect for the job but it requires going back to COM and writing native code. Don’t be scared: I won’t go that way :^)

However, if you would like to get more details about writing your own .NET profiler in C or C++, I would recommend looking at the Microsoft ClrProfiler initial .NET Framework implementation and also Pavel Yosifovich DotNext session about Writing a .NET Core cross platform profiler in an hour with the corresponding (more recent and cross platform) source code.

Instead, several events that are emitted by the CLR are providing interesting details:

The GCSampledObjectAllocation events payload provides a type ID instead of a plain text type name. In order to retrieve the type name given its ID, we need to listen to TypeBulkType event that contains the mapping as I described in my post about finalizers. This is why the last two GCHeapAndTypeNames and Type keywords are needed.

Remember that if both GCSampledObjectAllocationLow and GCSampledObjectAllocationHigh keywords are set, an event will be received for EACH allocation. This could be a performance issue both for the monitored application and the profiler. I would recommend starting with either low or high (more on this later).

Last but not least, enabling at least one of these keywords is also switching the CLR to use “slower” allocators. This is why you should check that it does not impact your application performance. These slower allocators are also used when your ICorProfilerCallback.Initialize method calls SetEventMask with COR_PRF_ENABLE_OBJECT_ALLOCATED flag to receive allocation notifications.

When you use Perfview for memory investigation, you are relying on these events without knowing it. In the Collect/Run dialog, three checkboxes are defining how to get the memory profiling details:

.NET Alloc: use a custom native C++ ICorProfilerCallback implementation (noticeable impact on the profiled application performance).
.NET SampAlloc: use the same custom native profiler but with sampled events.
ETW .NET Alloc: use GCSampledObjectAllocationHigh events

In all cases, the profiled application needs to be started after the collection begins.

How to listen to allocation events

As I have already explained in previous posts, the Microsoft TraceEvent nuget helps you listening to CLR events. First, you create a TraceEventSession and setup the providers you want to receive events from:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


session.EnableProvider(
    ClrTraceEventParser.ProviderGuid,
    TraceEventLevel.Verbose,    // this is needed in order to receive AllocationTick_V2 event
    (ulong)(

    // required to receive AllocationTick events
    ClrTraceEventParser.Keywords.GC |

    // the CLR source code indicates that the provider must be set before the monitored application starts
    ClrTraceEventParser.Keywords.GCSampledObjectAllocationLow | 
    //ClrTraceEventParser.Keywords.GCSampledObjectAllocationHigh | 

    // required to receive the BulkType events that allows 
    // mapping between the type ID received in the allocation events
    ClrTraceEventParser.Keywords.GCHeapAndTypeNames |   
    ClrTraceEventParser.Keywords.Type |
);

Second, you set up the handlers for the events you are interested in:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


private void SetupListeners(TraceLogEventSource source)
{
    source.Clr.GCAllocationTick += OnAllocationTick;

    source.Clr.GCSampledObjectAllocation += OnSampleObjectAllocation;

    // required to receive the mapping between type ID (received in GCSampledObjectAllocation)
    // and their name (received in TypeBulkType)
    source.Clr.TypeBulkType += OnTypeBulkType;
}

And lastly, the processing of received events is done in a dedicated thread until the session is disposed of:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


await Task.Factory.StartNew(() =>
{
    using (_session)
    {
        SetupProviders(_session);
        SetupListeners(_session.Source);
        
        _session.Source.Process();
    }
});

Now let’s see the difference between the two sets of events.

The AllocationTick way

My first idea was to use the AllocationTick event because it seemed easy: one sampled event with a size, a type name, and LOH/ephemeral kind. However, how this sampling works makes it impossible to get an exact per type allocated size. Let’s have a look at this list of events received from a WPF test application:

Small | 105444 : FreezableContextPair[]
Small | 111908 : FreezableContextPair[]
Small | 106720 : System.String
Small | 102488 : System.String
Small | 107028 : System.TimeSpan[]
Small | 106100 : System.String

All allocations were small (i.e. not in the LOH: < 85.000 bytes) and the second column gives the cumulated size of all allocations to reach the 100 KB threshold but not for this particular type! There is no easy way to make a valid guess of the specific last allocation size for which we get the type name.

For example, the first array of FreezableContextPair triggered the event for a cumulated size of 105.444 bytes. But how big was this array? We don’t know: could have been 100.000 because only 5444 bytes were allocated before or only 10444 bytes because 95.000 were allocated before. It would have been so useful that the size of the last allocated object would be passed in the event payload…

It is a little bit different (but not that better) for objects allocated in LOH because they have to be at least 85.000 bytes long. For example, allocate 4-byte arrays, each one 85.000 bytes long and let’s see the corresponding events:

Large | 170064 : System.Byte[]
Large | 170064 : System.Byte[]

Two AllocationTick events are received with 170064 as cumulated size. Still hard to figure out what was the size of the last allocated array: the only thing we know is that it was larger (or equal) to 85.000 bytes because it was allocated in LOH.

For larger objects, it might seem a little bit more accurate. Let’s allocate 2 byte arrays, each one 110.000 bytes long:

Large | 195064 : System.Byte[]
Large | 110032 : System.Byte[]

There are ~85.000 bytes difference between the two events even though the same 110.000 bytes were allocated. You could remove 85.000 bytes from the value and have an approximation of the LOH allocated object: the larger the allocation the less the error. But still: could be 85.000 size error…

So we won’t be able to rely on the size provided by the AllocationTick event; only the type name. In addition, you get a view of objects allocated in LOH. Maybe the other events will provide better results.

The GCSampledObjectAllocation way

When an object is allocated by the GC allocator, a GCSampledObjectAllocation event is emitted under certain conditions:

Both GCSampledObjectAllocationLow and GCSampledObjectAllocationHigh keywords are set on the CLR provider,
The object size is larger than 10.000 bytes,
At least 1000 instances of the type have been allocated,
Just before the application exits, current statistics for all types are flushed,
A complicated piece of code decides based on time since the last event and the type allocation rate.

Picking one or the other keyword changes the maximum number of milliseconds between two events for a given type:

High (10 ms) : 100 events / second
Low (200 ms) : 5 events / second

You should use low or high depending on the monitored application memory allocation workload to avoid impacting too much the profiler (and even the monitored application performance)

The interesting feature of these events is that, for a given type, the payload contains both the number of allocated instances since the last event and the cumulated size of these instances. Let’s take the same allocation of 4 arrays of byte, each 85000 long:

226 | 103616 : System.Byte[]
  1 |  85012 : System.Byte[]
  1 |  85012 : System.Byte[]
  1 |  85012 : System.Byte[]

This time, we get the exact count in the first column (ObjectCountForTypeSample) and the exact cumulated size in the second column (TotalSizeForTypeSample). If the count is 1, we have the exact size of that allocation and if it is bigger than 85000 bytes, we know it has been allocated in the LOH. Same accuracy for the 2-byte array of 110.000 elements:

198 | 123552 : System.Byte[]
  1 | 110012 : System.Byte[]

Sounds good. However, you have to remember that profiled applications need to be started after the session was created: it means that you can’t write a tool that will listen to a specific process ID like with AllocationTick. Three dictionaries are used by PerProcessProfilingState to keep track of per type allocations, type ID mappings, and process names:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


public class PerProcessProfilingState
{
    private readonly Dictionary<int, string> _processNames = new Dictionary<int, string>();
    private readonly Dictionary<int, ProcessTypeMapping> _perProcessTypes = new Dictionary<int, ProcessTypeMapping>();
    private readonly Dictionary<int, ProcessAllocationInfo> _perProcessAllocations = new Dictionary<int, ProcessAllocationInfo>();

    public Dictionary<int, string> Names => _processNames;
    public Dictionary<int, ProcessTypeMapping> Types => _perProcessTypes;
    public Dictionary<int, ProcessAllocationInfo> Allocations => _perProcessAllocations;
}

The SampledObjectAllocationMemoryProfiler class uses it for the events processing:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


public class SampledObjectAllocationMemoryProfiler
{
    private readonly TraceEventSession _session;
    private readonly PerProcessProfilingState _processes;
    
    // because we are not interested in self monitoring
    private readonly int _currentPid;

    private int _started = 0;

    public SampledObjectAllocationMemoryProfiler(TraceEventSession session, PerProcessProfilingState processes)
    {
        _session = session;
        _processes = processes;
        _currentPid = Process.GetCurrentProcess().Id;
    }

The constructor of the profiler keeps track of its own process ID in _currentPid to skip its own events.

Gathering type mapping

The processing of TypeBulkType events is quite straightforward: store the type ID/name association into a per-process dictionary:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


private void OnTypeBulkType(GCBulkTypeTraceData data)
{
    if (FilterOutEvent(data)) return;

    ProcessTypeMapping mapping = GetProcessTypesMapping(data.ProcessID);
    for (int currentType = 0; currentType < data.Count; currentType++)
    {
        GCBulkTypeValues value = data.Values(currentType);
        mapping[value.TypeID] = value.TypeName;
    }
}

private ProcessTypeMapping GetProcessTypesMapping(int pid)
{
    ProcessTypeMapping mapping;
    if (!_processes.Types.TryGetValue(pid, out mapping))
    {
        AssociateProcess(pid);

        mapping = new ProcessTypeMapping(pid);
        _processes.Types[pid] = mapping;
    }
    return mapping;
}

Remember that I choose to skip events from the current process detected by FilterOutEvent().

How to get process names

Even though each event contains the ID of the emitting process, it would be better to display its name instead. You could use Process.GetProcessById(pid).ProcessName when analyzing the details but the process might be long gone at that time.

Another solution would be to enable the Kernel ETW provider and listen to the ProcessStart event. The ImageFileName field of the payload contains the process filename with the extension. However, it is obviously not working on Linux.

The easiest solution is to use GetProcessById but just when you receive the first type mapping for a given process. This is the role of the AssociateProcess method called in GetProcessTypesMapping shown previously:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


private void AssociateProcess(int pid)
{
    try
    {
        _processes.Names[pid] = Process.GetProcessById(pid).ProcessName;
    }
    catch (Exception)
    {
        Console.WriteLine($"? {pid}");
        // we might not have access to the process
    }
}

It is now time to process allocation events.

Collecting allocation details

The GCSampledObjectAllocationTraceData payload contains the size and count of instances since the last event. We just need to store them for the corresponding process:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


private void OnSampleObjectAllocation(GCSampledObjectAllocationTraceData data)
{
    if (FilterOutEvent(data)) return;
    
    GetProcessAllocations(data.ProcessID)
        .AddAllocation(
            (ulong)data.TotalSizeForTypeSample, 
            (ulong)data.ObjectCountForTypeSample, 
            GetProcessTypeName(data.ProcessID, data.TypeID)
            );
}
private string GetProcessTypeName(int pid, ulong typeID)
{
    if (!_processes.Types.TryGetValue(pid, out var mapping))
    {
        return typeID.ToString();
    }

    var name = mapping[typeID];
    return string.IsNullOrEmpty(name) ? typeID.ToString() : name;
}

The AddAllocation() helper method is simply accumulating these numbers for a given type in the ProcessAllocationInfo associated to the related process.

Displaying the results

When the profiling session ends, it is easy to show the allocated count and size per type:

The code is using a Linq syntax to get top allocations sorted either by count or by size:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


private static void ShowResults(string name, ProcessAllocationInfo allocations, bool sortBySize, int topTypesLimit)
{
    Console.WriteLine($"Memory allocations for {name}");
    Console.WriteLine();
    Console.WriteLine("---------------------------------------------------------");
    Console.WriteLine("    Count        Size   Type");
    Console.WriteLine("---------------------------------------------------------");
    IEnumerable<AllocationInfo> types = (sortBySize)
        ? allocations.GetAllocations().OrderByDescending(a => a.Size)
        : allocations.GetAllocations().OrderByDescending(a => a.Count)
        ;
    if (topTypesLimit != -1)
        types = types.Take(topTypesLimit);

    foreach (var allocation in types)
    {
        Console.WriteLine($"{allocation.Count,9} {allocation.Size,11}   {allocation.TypeName}");
    }
}

Another usage could be a long-running monitoring system that shows the allocation rate: a nice complement to the other GC metrics. However, compared to the other profilers, one important feature is missing: if an unexpected number of instances are created, how to know which part of the code is responsible for the spike?

The next post will explain how to enhance such a sampled memory profiler with call stacks per sampled allocation.

Resources

Source code available on Github.
Spying on .NET Garbage Collector with TraceEvent
Pavel Yosifovich — Writing a .NET Core cross-platform profiler in an hour
Original Microsoft ClrProfiler source code and documentation

Like what you read? Don’t forget to check out part 2 on this topic:

Build your own .NET memory profiler in C# — call stacks (2/2–1) *This post explains how to get the call stack corresponding to the allocations with CLR events.*medium.com

Interested in joining our journey? Check this out:

Product, Research & Development | Criteo Careers *Product, Research & Development at Criteo. At Criteo, come and meet our teams and join our R & D and also enjoy…*careers.criteo.com

Debugging Wednesday at Criteo — Cancel this task!

Fri, 21 Feb 2020 08:18:08 +0000

Introduction

Last Wednesday was a great day for Kevin and myself: We spent a lot of time investigating the reasons why a test was failing. Let’s share with you these frustrating but interesting minutes.

One of our colleagues came to us because an integration test would get stuck in some specific conditions. Here is the simplified code of the service that is supposed to do some background processing until it is stopped:

1
2
3
4
5
6
7
8


public class Service
{
    private CancellationTokenSource _cancellationSource;
    private Task _backgroundProcessing;
    private Task _cleanup;

    ...
}

In the test, the service is created: Two Task instances are created in the constructor (for background processing irrelevant for this discussion) that (1) are started when the service starts and (2) are canceled when the service is stopped. A CancellationTokenSource is used to cancel the tasks if needed.

1
2
3
4
5
6
7
8
9


public Service()
{
    _cancellationSource = new CancellationTokenSource();
    var cancellationToken = _cancellationSource.Token;
    _backgroundProcessing = new Task(DoStuffInTheBackground,
        cancellationToken, TaskCreationOptions.LongRunning);
    _cleanup = new Task(DoStuffInTheBackground, 
        cancellationToken, TaskCreationOptions.LongRunning);
}

However, in the specific conditions of this test, the Start method would never be called, skipping straight to the Stop method.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


public void Start()
{
    _backgroundProcessing.Start();
    _cleanup.Start();
}

public async Task Stop()
{
    _cancellationSource.Cancel();
    await _backgroundProcessing;
    await _cleanup;
}

The task is never started because the test skips the Service.Start() method, but it should transition to “Cancelled” state as soon as the CancellationTokenSource associated with the CancellationToken passed to the Task gets canceled. In that particular situation, we expect the await _backgroundProcessing code to immediately return in the Stop() method.

In our colleague Visual Studio, the debugger never came back from await _backgroundProcessing, indicating that the task never completed…

Reproduce the problem

I wanted to un-validate any side effect related to our test framework or custom Criteo libraries and confirm my understanding of task cancellation, so I wrote a small Console application with the same 4.7.2 version of .NET Framework with the following code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


static async Task ReproduceWorking()
{
    var source = new CancellationTokenSource();
    var token = source.Token;
    var t = new Task(
        () => Console.WriteLine("Done"), token, TaskCreationOptions.LongRunning);

    source.Cancel();

    try
    {
        await t;
    }
    catch (TaskCanceledException x)
    {
        Console.WriteLine(x.ToString());
    }
}

And guess what? I got the expected TaskCancellationException:

System.Threading.Tasks.TaskCanceledException: A task was canceled. at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.GetResult() at TaskCancellation.Program.d__2.MoveNext()

The next step was to double-check the understanding of how tasks are working related to cancellation. So I set a breakpoint after the task is instantiated

And its status is Created.

Doing the same after Cancel() is called on the CancellationTokenSource, gives the expected Canceled status.

So, let’s do the same when debugging the test code:

It gives the same Created status after the task creation but after canceling the source:

…the status does not switch to Canceled like in my Console repro!

Looking for cancellation

The only thing that came to my mind was: maybe the cancellation token is not taken into account. However, a CancellationToken is just a struct that keeps a reference to its CancellationTokenSource. It should be easy in Visual Studio debugger to double-check that our task keeps track of the token somewhere and goes back to the cancellation source. Well… the token is not kept as a field of the task but stored inside the m_contingentProperties field deep during the task construction code path.

Let’s look at the value in the Quick Watch after the cancellation source gets canceled:

It sounds like the cancellation token is not canceled… But if we look at the source Token property of our canceled CancellationTokenSource,

we don’t have the same value for IsCancellationRequested!

It’s like we don’t look at the same cancellation source… To find out, we just have to compare the reference to our CancellationTokenSource with the one we see in the m_contingentProperties token of the task. To achieve that, we could copy the expression from QuickWatch, paste it into the Debug | Windows | Memory… pane and press ENTER to get the address where the object is stored in memory

After having done the same with _cancellationSource, I did not get the same address:

It means that we were dealing with two different instances of CancellationTokenSource.

But this might be too C++ish for you… Kevin prefers leveraging the Make Object ID feature of the C# debugger. You simply right-click the Data Tip of the cancellation source and select Make Object ID:

Once this is done, a numeric identifier is displayed for this instance in Data Tip and any Watch window (#1 in this screenshot):

When we looked at the cancellation source of the token stored by the task,

we didn’t see any ID so it was not the same object.

So we decided to use Make Object ID on this m_source that became marked as #2.

And when we looked at the m_source of the second cleanup Task, we realized that it was the same object but not the one we created!

We started to think that we were becoming crazy. So, let’s restart from the beginning and follow the CancellationTokenSource from its creation because we are sure that we passed a valid cancellation token (linked to this source) to the task constructor. Or… Did we? The QuickWatch gives a different answer just after the task gets created compared to what we’ve seen already: a token with an empty source property now!

I closed the QuickWatch pane and reopened it for Kevin to confirm. And like in a nightmare, the source was not null anymore…

Visual Studio must be responsible for that weird behavior!

Kevin remembered the ”Enable property evaluation” settings in the Options dialog. If it is checked (which is the default), it means that the Debugger would fetch the value of an instance field and then call the Getter of each property in order to display its value.

However, if you uncheck it, only the fields are displayed. So in our case, we then always got a null m_contingentProperties field (and, as expected, all property would not be displayed):

The m_contingentProperties is initialized in EnsureContingentPropertiesInitialized() when called by AssignCancellationToken() from the TaskContructorCore() helper used by the Task constructor but it did not seem to be the case because it was definitively null…

Kevin decided to stop at the CancellationTokenSource constructor with a new breakpoint (more on how to set a breakpoint on a .NET Framework method soon) to see where the one shown in the Debugger was created but the breakpoint was never hit. So the CancellationTokenSource #2 must have been created even before our own was created by our code. In fact, a static CancellationTokenSource is created and is set to m_source when InitializeDefaultSource() gets called by one of the Getter. This explains why we saw the same instance #2 in both tasks token.

To sum up, we were now sure that the passed token was not “received” by the Task.

Eureka!

Maybe there is a magic trick done by the .NET Framework to lazily set the token source after the creation of the task. However, we did not find such a code in the .NET Framework and this is not what we see in our repro.

Back to the basics: are we sure that we are executing the code we think is executed? We looked for mscorlib in the Debug | Windows | Modules pane,

and we opened it with a decompiler: the code of the methods called during the Task** **construction was the same as the one shown in https://referencesource.microsoft.com/#mscorlib/system/threading/Tasks/Task.cs.

Next, in order to better follow the execution and the passing of parameters (including our token), we decided to set breakpoints on Task private method responsible for its initialization.

internal void TaskConstructorCore(object action, object state, CancellationToken cancellationToken, TaskCreationOptions creationOptions, InternalTaskOptions internalOptions, TaskScheduler scheduler)

In the Debug | Windows | Breakpoints pane, click New | Function Breakpoint…

and type the full name of the method. This is working even for a method of a class defined in the .NET Framework assembly for which you do not have the source code:

We checked that the breakpoints were well set (i.e. no typo in the full name) by looking at the filled red circle:

The cancellationToken parameter should contain the token that we passed at Task creation. Unfortunately, the QuickWatch pane displayed a “cannot read memory” error that we never saw in Visual Studio before!

At that time, we thought we were doomed but we looked at the Call Stack pane and we realized that the code was calling the wrong Task constructor:

public Task(Action action, object state, TaskCreationOptions creationOptions)

Its signature is compatible with our code:

_backgroundProcessing = new Task(DoStuffInTheBackground, cancellationToken, TaskCreationOptions.LongRunning);

and this is why the compiler did not complain.

Our CancellationToken was passed as the state parameter is given directly to our DoStuffInTheBackground Action: the created Task had no idea that it was supposed to be its CancellationToken.

Note that if we had noticed the Auto Completion (Ctrl + Shift + Space) hint, we might have figured out the root cause much sooner…

The fix was straightforward; just using the right constructor:

public Task(Action action, object state, CancellationToken cancellationToken, TaskCreationOptions creationOptions)

that accepts both a state for the callback and a CancellationToken for the Task to create:

_backgroundProcessing = new Task(DoStuffInTheBackground, cancellationToken, cancellationToken, TaskCreationOptions.LongRunning);

Under the debugger, we validated that the source was now the expected one:

and the test did not hang anymore.

If, from the beginning, we would have been able to step into .NET Framework compiled code as we do with Jetbrains Resharper integration in Visual Studio, we would have found the issue almost immediately. Thankfully, Microsoft has just announced decompilation of C# code made easy with Visual Studio.

We wish we had it last Wednesday…

Interested in reading more about Christophe’s & Kevin’s work? Check out their latest articles:

Build your own .NET memory profiler in C# *This post explains how to collect allocation details by writing your own memory profiler in C#.*medium.com Switching back to the UI thread in WPF/UWP, in modern C# Leveraging the async machinery to transparently switch to the UI thread when neededmedium.com

If you are looking for a change and would love to work with these two, head over to our careers page and let us know if there is something that sounds like you!

Product, Research & Development | Criteo Careers *Come and meet our teams …*careers.criteo.com

Getting another view on thread stacks with ClrMD

Tue, 31 Dec 2019 10:52:00 +0000

This post of the series details how to look into your threads stack with ClrMD.

Introduction

It’s been a long time (see the resources at the end) since I’ve been discussing what ClrMD could bring to .NET developers/DevOps! My colleague Kevin just wrote an article about how to emulate SOS DumpStackObjects command both on Windows and Linux with ClrMD. This implementation lists the objects on the stack but without their values (like strings content for example) nor the stack frames corresponding to the method calls.

The rest of the post will show you, with ClrMD, how to get an higher view, closer to what the SOS ClrStack command could provide.

Let’s take this simple application as an example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


class Program
{
    static void Main(string[] args)
    {
        Host h = new Host();
        h.Base();
    }
}

class Host
{
    public void Base()
    {
        int iValue = Guid.NewGuid().GetHashCode();
        bool bValue = (iValue % 2) == 0;
        string parameter = Guid.NewGuid().ToString().Substring(0,1) + "_1234567890";

        Console.WriteLine(parameter + " = " + First(iValue, bValue, parameter));
    }

    private int First(int iValue, bool bValue, string parameter)
    {
        Guid guid = Guid.NewGuid();
        return Second(bValue, guid) / 2;
    }

    private int Second(bool bValue, Guid guid)
    {
        return Third(guid).Length;
    }

    private string Third(Guid guid)
    {
        Console.WriteLine($"call   procdump -ma {Process.GetCurrentProcess().Id}");
        Console.ReadLine();
        return $"{guid.GetHashCode()}#{guid.GetHashCode()}#{guid.GetHashCode()}";
    }
}

As you can see, I’ve mixed value and reference types as parameters and local variables up to the call to the Third method that displays the procdump command line to execute in order to generate a memory dump of the process.

Use WinDBG + SOS Luke!

When you open it with WinDBG and load SOS, here is the result of the dso command:

The clrstack command shows the stacked method calls:

And if you use the -a parameter, you will get methods with their parameters and local variables (or -p for parameters only and -l for local variables only):

It is weird that SOS implementation does not give the type of both the parameters and locals. But wait! While researching for this post, I looked at the SOS implementation (now in the strike.cs file moved from the coreclr to the diagnostics repository) to find this nice comment:

So I tried clrstack with -i and I got the types for parameters (and locals unlike what the comments implies):

Even though clrstack supports the -all flag to dump the call stack of all managed threads, you might need to do your own automatic analysis on hundreds of threads and this is where ClrMD shines.

Merging methods and parameters/locals

When I read Kevin’s post, I immediately thought about adding the method call on the stack based on the work I’ve done in March 2019 to implement the pstacks tool. At that time, my goal was to aggregate the call stacks of a large number of threads in order to find out pattern of blocked threads, sharing the “same” call stacks. Visual Studio provides a great “Parallel Stacks” pane but I needed it for both Windows and Linux.

To list all the call stack with ClrMD, you simply enumerate the managed threads and for each one, its StackTrace property contains the list of StackFrame objects corresponding to each method call.

The StackPointer property of each frame contains the address of the frame in the call stack, allowing a mapping of the method call with its parameters and locals:

As always with stacks, lower addresses correspond to the last things added to the stack (i.e. last called method). While checking between what is shown by SOS, the parameters/locals addresses and frame stack pointers, you realize that all objects at an address in the stack equal or below the StackPointer of a frame are either parameters or local variables of the frame method.

Even better, for non static method, you can guess what is the this* *implicit parameter if the address is the same as the frame StackPointer; shown with the green = sign in the previous screenshot and prefixed by > in Kevin’s updated code that merges the method calls to the parameters and locals:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46


for (ulong ptr = stackTop; ptr <= stackLimit; ptr += (ulong)runtime.PointerSize)
{
    // look for the frame corresponding to the current position in the stack
    if (currentFrame.StackPointer <= ptr)
    {
        Console.WriteLine(FormatFrame(currentFrame));

        nFrame++;
        if (nFrame < frames.Count)
        {
            currentFrame = frames[nFrame];
        }
        else
        {
            break;
        }
    }

    ulong obj;
    if (!runtime.ReadPointer(ptr, out obj))
    {
        break;
    }

    if (!IsInHeap(heap, obj))
    {
        continue;
    }

    var type = heap.GetObjectType(obj);
    if (type == null || type.IsFree)
    {
        continue;
    }

    // try to find implicit "this" parameter in case of non-static method
    var myFrame = (nFrame == 0) ? currentFrame : frames[nFrame-1];
    separator = ((myFrame.StackPointer == ptr) &&
                 (myFrame.Method != null) &&
                 (!myFrame.Method.IsStatic))
        ? ">" 
        : " ";
    Console.Write($"{ptr:x16}   {separator} ");
    DumpObject(heap, type, (ulong)obj);

}

The FormatFrame helper method simply prefix static methods with # instead of | for instance methods:

Unfortunately, I did not find any way with ClrMD to make the difference between parameters and locals. Based on what you can see in SOS implementation of this part of the clrstack command, it relies on the EnumerateArguments and EnumetateLocalVariables methods of ICorDebugILFrame which is not exposed by ClrMD. There is another undocumented implementation based on private interfaces I could not leverage neither. For a larger discussion around stack walking in .NET, read this great post by Matt Warren.

Also, without any explicit access to specific parameter or local, I did not find a way to get the value of primitive and value type instances stored on the stack. However, it is still possible to get them for boxed ones and reference type instances such as string for example.

Getting instances from the stack

In the last code excerpt, I did not describe the DumpObject helper method used to display an object on the stack. The implementation provided by Kevin was used to show the address and the type of the object:

1

Console.WriteLine($"{objAddress:x16} {type.Name}");

The next step would be to display value for primitive types such as numbers, boolean, string and even array size:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


private static void DumpObject(ClrHeap heap, ClrType type, ulong objAddress)
{
    // get value for simple types
    string valueOrAddress = 
        (type.Name == "System.Char") ? $"{type.GetValue(objAddress),16}" :
        (type.Name == "System.String") ? $"{FormatString(type.GetValue(objAddress).ToString())}" :
        (type.Name == "System.Bool") ? $"{FormatString(((bool)type.GetValue(objAddress)).ToString())}" :
        (type.Name == "System.Byte") ? $"{FormatString(((byte)type.GetValue(objAddress)).ToString())}" :
        (type.Name == "System.SByte") ? $"{FormatString(((sbyte)type.GetValue(objAddress)).ToString())}" :
        (type.Name == "System.Decimal") ? $"{FormatString(((decimal)type.GetValue(objAddress)).ToString())}" :
        (type.Name == "System.Double") ? $"{FormatString(((double)type.GetValue(objAddress)).ToString())}" :
        (type.Name == "System.Single") ? $"{FormatString(((float)type.GetValue(objAddress)).ToString())}" :
        (type.Name == "System.Int32") ? $"{FormatString(((int)type.GetValue(objAddress)).ToString())}" :
        (type.Name == "System.SInt32") ? $"{FormatString(((uint)type.GetValue(objAddress)).ToString())}" :
        (type.Name == "System.Int64") ? $"{FormatString(((long)type.GetValue(objAddress)).ToString())}" :
        (type.Name == "System.SInt64") ? $"{FormatString(((ulong)type.GetValue(objAddress)).ToString())}" :
        (type.IsArray) ? $"{FormatString(GetArrayAsString(type, objAddress))}" :
        $"{objAddress:x16}";  // work also for IntPtr
    Console.WriteLine($"{valueOrAddress} {type.Name}");
}

Most of this code is based on the GetValue helper from ClrType: it returns the right “thing” for simple types. Look at ClrMD implementation details to get a better understanding of how the value is rebuilt.

The GetArrayAsString simply returns the number of elements in the array:

1
2
3
4
5


private static string GetArrayAsString(ClrType type, ulong objAddress)
{
    var elementCount = type.GetArrayLength(objAddress);
    return $"length = {elementCount}";
}

And the call stack is now complete!

Note that you may even get more locals or parameters than with WinDBG+SOS but don’t ask me why…

For more advanced object formatting cases such as dumping structs or enumerating fields and their value, I would highly recommend to look at the related ClrMD documentation page (just replace GCHeapType by ClrType and you’ll be safe).

Resources

Dumping stack objects with ClrMD by Kevin Gosse
Stack walking in the .NET Runtime by Matt Warren
Part 1: Bootstrap ClrMD to load a dump.
Part 2: Find duplicated strings with ClrMD heap traversing.
Part 3: List timers by following static fields links.
Part 4: Identify timers callback and other properties.
Part 5: Use ClrMD to extend SOS in WinDBG.
Part 6: Manipulate memory structures like real objects.
Part 7: Manipulate nested structs using dynamic.
Part 8: Spelunking inside the .NET Thread Pool
Part 9: Deciphering Tasks and Thread Pool items

WSL + Visual Studio = attaching/launching a Linux .NET Core application on my Window 10

Thu, 21 Nov 2019 15:57:01 +0000

This post shows how to attach to a .NET Core process running on Linux with WSL and also how to start a Linux process with Visual Studio debugger

Coming from the Windows world, I don’t find that easy to develop .NET Core applications for Linux. I’m used to code and debug in Visual Studio. Now, I need to build on Windows (due to our Criteo continuous integration), deploy an artifact to Marathon in order to get an application running inside a Mesos container. At Criteo, we had to build a whole set of services to allow remote debugging or memory dump analysis.

But what if I just wanted to test and debug a small scenario on my beloved Windows machine? Windows Subsystem for Linux (aka WSL) is perfect for running a Linux .NET Core application on Windows. However, how to attach to it or even start a debugging session from Visual Studio? In the rest of this post, I’ll explain how to setup your Windows 10 machine to achieve these miracles.

Linux on Windows: welcome WSL

It is obviously not the place to dig into Windows Subsystem for Linux. You just need to know that once installed, it allows you open up a Linux shell on your Windows machine without any virtual machine kind of technology. It is also possible to share folders between Windows (where you want to build your application) and Linux (where you want to execute the built assemblies).

The first step is to turn on WSL feature in your Windows:

After a reboot, you are able to start a WSL prompt:

You can also install your favorite Linux distro if you want.

The next step is to install .NET Core runtime or SDK so you are able to run your application on Linux.

It is now time to look at your hard drive from a Linux perspective:

Your drives are mapped under /mnt without the “:”. In my case, I created a wsl folder under my d: drive to do my experiments. With Visual Studio, I generated a TestConsole application with the following code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


using System;
 
namespace TestConsole
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Enter x to EXIT...");
            while(true)
            {
                var cmd = Console.ReadLine();
                if (cmd.ToLower() == "x")
                    return;
 
                Console.WriteLine($"> {cmd}");
            }
        }
    }
}

I’m publishing the application to get all needed assemblies:

In a WSL prompt, type dotnet with the name of your application assembly et voila!

A Linux .NET Core application is running on your Windows machine.

How to attach to a running Linux application

Let’s say that I’m detecting a problem and I want to debug the Linux application with Visual Studio. When attaching the Visual Studio debugger to a process, several connection types are available:

The SSH connection type will be used with WSL with the following kind of communications architecture:

The Visual Studio debugger is sending commands to the remote Linux debugger vsdbg via an SSH channel. Here are the steps to follow to install the missing components:

By default, an SSH server is installed with WSL. However, I was not able to make the whole pipeline work with it so I had to uninstall and reinstall it: sudo apt-get remove openssh-server sudo apt-get install openssh-server
The SSH configuration needs also to be changed in order to allow username/password kind of security needed by Visual Studio (if you prefer key-based security, look at the end of the post for available resources). If you don’t know how to use vi efficiently to simply edit a file, install nano (thanks @kookiz for the tips :^) sudo apt-get install nano
In /etc/ssh/sshd_config, change the PasswordAuthentication settings sudo nano /etc/ssh/sshd_config PasswordAuthentication yes
Restart the ssh server sudo service ssh start
You need to install unzip in order to get vsdbg sudo apt-get install unzip curl -sSL https://aka.ms/getvsdbgsh | bash /dev/stdin -v latest -l ~/vsdbg

You are now ready to select SSH as connection type and enter your machine name before clicking the Refresh button. At that time, a new dialog should pop up for you to enter your WSL credentials:

After you click the Refresh button, the list at the bottom should contain the Linux processes running in WSL

Select your .NET Core application and click Attach to select the Managed debugger:

Now, if you set a breakpoint in the code and trigger it with an appropriate action in your WSL prompt

then the debugger will break as expected:

Note that when you stop your debugging session, the Linux application is not stopped; just detached from the debugger and keeps on running.

Showtime for F5!

Attaching to a running Linux application is nice but it would be even better to start a Linux process from Visual Studio debugger. In order to achieve this goal you need to add another piece to the architecture puzzle:

As detailed in this WIKI page, it is possible to tell Visual Studio to execute debugging actions thanks to a launch.json file such as the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


{
  "version": "0.2.0",
  "adapter": "c:\\tools\\plink.exe",
  "adapterArgs": "-ssh -pw  chrisnas@DBGDQPZ1 -batch -T ~/vsdbg/vsdbg --interpreter=vscode",
  "configurations": [
    {
      "name": ".NET Core Launch",
      "type": "coreclr",
      "cwd": "/mnt/d/wsl/TestConsole/TestConsole/bin/Debug/netcoreapp2.1/publish",
      "program": "TestConsole.dll",
      "request": "launch"
    }
  ]
}

The plink tool from putty will be used as an adapter for Visual Studio to communicate with vsdbg running in WSL. The adapterArgs property gives the same SSH/machine/user/password information that you provide via Visual Studio UI in the Attach scenario. The configuration section defines which request (“launch” instead of attach and which folder/assembly to start) will be sent to vsdbg.

Once this file is created (in my case in d:\wsl\vs folder), you just need to type the following command in the Immediate pane of Visual Studio:

DebugAdapterHost.Launch /LaunchJson:d:\wsl\vs\launch.json

and if you had set a breakpoint on the first line of your application, the debugger should break there:

In a WSL prompt, you can see the expected 2 new spawned processes:

But wait!

I have a problem now: I don’t have any prompt in which typing input for my console application… However, as always in Linux, you simply need to write to a file to fix this. The stdin stream of your application is accessible under /proc//fd/0.

So, when I type the following command:

echo "Launching a Linux app is not a problem!" > /proc/18341/fd/0

my breakpoint is hit in Visual Studio:

Also note that everything that is sent to the console appears in Visual Studio Output pane:

Note that, unlike the Attach scenario, if you stop the debugging session, the Linux application (and vsdbg) will be terminated.

Resources

During my investigations I’ve found a few resources you might find useful (especially about configuring SSL with keys instead of passing clear user/password)

Basic VS + WSL
Good description about how to use VS 2017 to attach to .NET Core app running in WSL
Debugging .NET Core from VS 2017 and WSL
VS/C++ with WSL (describe how to install WSL and setup open ssh server)
Setup SSH on WSL
Wiki for VSDBG
Great description about how to setup your linux dev environment with VS Code and WSL

Let’s debug the Core CLR with WinDBG!

Thu, 04 Apr 2019 13:32:51 +0000

This post of the series shows how we debugged the Core CLR to figure out insane contention duration.

Part 1: Replace .NET performance counters by CLR event tracing.

Part 2: Grab ETW Session, Providers and Events.

Part 3: CLR Threading events with TraceEvent.

Part 4: Spying on .NET Garbage Collector with TraceEvent.

Part 5: Building your own Java GC logs in .NET.

Introduction

Long before migrating our .NET applications to Linux, our first step was to build a monitoring pipeline based on LTTng instead of ETW on Windows. To achieve this goal, the open source TraceEvent Nuget package needed to be updated in order to listen to LTTng live session (only a file based implementation was provided by Microsoft; mostly to allow Perfview to be able to open traces taken on Linux machines). This was a huge development task that led sometimes to weird results. Among the metrics we wanted to monitor, the contention duration gave insane value such as thousands of minutes… per minute:

As shown in a previous episode, this duration is computed by comparing the time between the two events ContentStart and ContentionStop. What could be the possible reasons to get such insane values?

A lot of small contentions are happening
A few very long contentions are happening

As a first step, it would be great to be able to debug the Core CLR and figure out what call stacks end up to triggering these contention events. Unfortunately for us, the .NET debugging ecosystem on Linux is far from being as rich as on Windows. So this episode is detailing the steps to compile and debug the Core CLR on Windows with WinDBG.

From the source to debugging the runtime

To better understand the implementation details in the CLR, we needed to find where the two events are emitted. In fact, during the CLR compilation, a lot of helpers are created based on the name of the event. In our case, FireEtwContentionStart_V1 and FireContentionStop are the two helpers in charge. Both are called in the AwareLock::EnterEpilogHelper function.

As a Windows developer, I would like to debug the CLR code and set a breakpoint in the EnterEpilogHelper with Visual Studio to see what are the call stacks that end up to contention. However, I did not find a way to do it with Visual Studio. I turned to WinDBG and things gets “easier”… in a certain way.

Here are the different steps you need to follow before setting a breakpoint on any Core CLR function in WinDBG:

Clone the Core CLR repository from https://github.com/dotnet/coreclr
Build it:
Get the Visual Studio, .NET Core SDK, CMake, Python, Powershell prerequisites from the documentation
Goto the root folder and type .\build -skiptests to build a DEBUG version of the Core CLR
Leave your desk and go to lunch (ok… maybe just take a coffee break)

When you go back, the result of the compilation should be available in the following folder:

…\coreclr\bin\Product\Windows_NT.x64.debug.

the next step is to use your custom Core CLR build in the application:

the application must be self-contained by adding win-x64 (or linux-x64 for Linux) in a PropertyGroup section of the .csproj.
publish the application by running dotnet publish or from within Visual Studio

Click the Configure link and select Debug configuration

after clicking Save and Publish, you should now have the result under the \bin\Debug\netcoreapp2.2\publish folder.
after clicking Save and Publish, you should now have the result under the \bin\Debug\netcoreapp2.2\publish folder.

It is now time to copy the following files from the Core CLR output to your application publication folder:

coreclr.dll (for the native part of the CLR) and System.Private.CoreLib.dll (if the CLR C# code has been modified)
in the PDB subfolder, coreclr.pdb and System.Private.CoreLib.pdb
note that you might also need the sos.dll and mscordaccore.dll files for any investigation in WinDBG.

If you wonder why the CoreFx repo is not rebuilt, the answer is simple: the contention related code is in the CoreCLR. Also, most of the managed “mscorlib” is defined in System.Private.CoreLib.dll that gets built with CoreCLR. The rest of the BCL is covered by CoreFX and not needed in this investigation.

From running to debugging in WinDBG

You should use corerun.exe instead of dotnet.exe to run an application with the debug version of the Core CLR you’ve just built.

Open up a command prompt in the coreclr\bin\Product\Windows_NT.x64.debug folder and type corerun** c:bin\Debug\netcoreapp2.2\publish folder of your application>**

You have to tell corerun where to find the CoreFx assemblies via the CORE_LIBRARY environment variable:

CORE_LIBRARIES=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.0

If you forget about it, don’t be surprised if the application stops with FileNoteFoundException for a missing assembly (usually System.Runtime)…

If, like me, your applications are running with server mode GC, you know that it is set in the application .csproj file to end up into the runtimeconfig.json file. Unfortunately, this is not taken into account by corerun (yet?) and you need to set it (and if you need concurrent version too) explicitly through the following environment variables:

COMPlus_gcServer=1 COMPlus_gcConcurrent=1

From there, (install WinDBG if not already done and) start the debugger: click the File menu and select Launch Executable (advanced) to setup a debugging session:

The Executable text field points to the corerun.exe file generated during the compilation of the Core CLR. The same folder is used as Start Directory and the Arguments text field contains the full path of the application to debug. You could also attach to a running process but sometimes you need to access Core CLR data structures before any C#-compiled managed code of your application starts executing (to see how the garbage collector initializes for example).

As soon as you click the Ok button, the application starts but is almost immediately stopped by WinDBG

Don’t be scared by the last lines of the output: even through you read the word exception, this int 3** **assembly instruction tells you that a breakpoint has been set for you by WinDBG, has been hit when the application reached it and the application is now paused just before calling its entry point.

As you can see from the list of loaded modules, even though CoreRun.exe is there, no managed assembly (especially your application) has been loaded yet; not even the Core CLR itself! This means that you have to tell WinDBG to keep on executing the application until a point you would be interested in. To achieve that task, you will first need a quick tour of WinDBG user interface even though this post is not there to replace the WinDBG online help nor provide a detailed walkthrough.

The debugging section of the Home tab is not too different from what you get in Visual Studio:

The icons are even easier to understand because their action is also displayed. If you want to see the current call stack, select the View tab and click the Stack icon:

Like in Visual Studio, you are able to pin each panel wherever you want into the IDE

The next step would be to set a breakpoint on the line of code you are interested in. But let’s be clear here: I’m talking about a line of code in a function exported by a native dll; not a line of C# code in a managed assembly. Remember that WinDBG is a native debugger and it debugs only native code. If you want to debug managed code with WinDBG, you need to use commands from the sos extension; but this is another story.

So let’s go back to the native world. Even though WinDBG does not have the notion of “solution” like Visual Studio provides, it is still possible to open a C++ file and set a breakpoint in it. Click File |Open Source File menu and go to your Core CLR github repo to select syncblk.cpp under the \src\vm folder. Look for AwareLock::EnterEpilogHelper with CTRL+F (yes: search is working in WinDBG) and go down to the call to the FireEtwContentionStart_V1 helper. Setting a breakpoint on this line is as simple as pressing **F9 **like in Visual Studio. Press the View tab and click the Breakpoint button to see the result:

Since the dll in which the breakpoint is set is not loaded yet, you can’t see the details of the breakpoint.

There is a way to tell WinDBG to continue the execution of the application until a dll get’s loaded. For coreclr.dll, type the following command:

sxeld:coreclr

and type F5 (or type g as a command or click the green triangle in the Home toolbar) to resume the execution of the application. The Breakpoints panel shows more details now:

Press F5 to resume the execution and the breakpoint should be triggered when the first contention happens.

From symbols to call stacks in WinDBG

Before digging into call stacks, I would like to show you one of the differences between native dlls and managed assemblies. As a .NET developer, you are used to Intellisense and strongly typed environment provided by the metadata stored in an assembly itself. For native dll, the story is different. Exported functions are visible with tools such as Dependency Walker or dumpbin /exports from the SDK. If the dll exports symbols built by the C++ compiler, their name gets mangled to describe their signature. To get human readable symbols, you need the associated .pdb file. It will also be required to map a function address to its name in call stacks.

WinDBG allows you to browse these symbols with the dt command. For example, if you want to know all members defined by the AwareLock class, use the following command:

dt CoreClr!AwareLock::*

Like what was shown in the previous Breakpoints screenshot, the prefix of a name is the dll in which the symbol is defined. Next, use ! as separator before the class name. Since Visual Studio is really slow to navigate the source code of the Core CLR or search in the thousands of include and C/C++ files, this is a very convenient way to navigate and learn its different parts. Don’t forget that the compilation could also inline functions (that won’t be visible in the symbols) and expand macros.

If you want to set a breakpoint on a function, use the bp command with the same syntax as dt. For example, the following command:

bp coreClr!AwareLock::EnterEpilogHelper

sets a breakpoint at the beginning of the function in which I already set a breakpoint.

This is the very basics of breakpoints in WinDBG. You are also able to define which actions to start when a breakpoint is hit. This is extremely powerful! For example, in the case of thread contention, you typically don’t want to stop the execution of the application because it will pause all threads and disturb the normal flow of execution that could lead to thread contention. Instead, you could ask WinDBG to dump the call stack leading to the function we are interested in and lets the execution resume with the following syntax:

bp coreClr!AwareLock::EnterEpilogHelper "!clrstack; g"

The commands to execute after the breakpoint is hit are defined between quotes. In this example, I’m using the clrstack command exported by the sos.dll extension (that must be previously loaded via .loadby sos coreclr) and once it is done, g resumes the execution.

What’s next?

Due to automatic suspension of all threads when the clrstack command gets executed (before g resumes), the interactions between threads are not the same as normal execution outside of a debugger. I have even used some code available in DEBUG to dump the callstacks outside of a debugger if the contention last more than a threshold. However, it was not possible to reproduce the problem on Windows.

In parallel on Linux, another colleague investigated another lead: some events may also be skipped by our LTTng implementation. Due to complicated event management, if a ContentionStop and ContentionStart are missed, a possible previous ContentionStart could be used by the next ContentionStop and the duration would be unrelated to the real contention that happened.

So there could be a simpler way to narrow down the issue: instead of relying on two events, why not simply compute the duration of the contention in the AwareLock::EnterEpilogHelper function and emit only one new event with the duration as payload? Well… this will be the topic of the next episode of this series.

References

Series of videos from the Defrag Tools show where Maoni Stephens explains how to debug the Garbage Collector for a better understanding of its arcana

Debugging Friday — Hunting down race condition

Fri, 22 Feb 2019 09:25:50 +0000

Introduction

At Criteo, CLR metrics are collected by a service that listens to ETW events (see the related series). On a few servers, the metrics stopped being collected and we had to fix the problem by manually polling new and dead processes. After deploying the new version, the same scenario started to happen: on some servers, the metrics were no more collected.

In an investigation, the first step is always trying to check the environment. In our case, on a server where the metrics collector is up and running, a dedicated ETW session should be created to listen to the CLR events.

The name given to the session allows us to easily detect if the session is present or not. In the case of a faulted server, the session was not present.

If you look at the code described in a previous post, it is not easy to guess why the session would be stopped by the metrics collector:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


private void ListenToEtw(TraceEventSession etwSession)
{
    try
    {
        // this call is blocking... until ewtSession.Stop is called (done in Dispose)
        etwSession.Source.Process();
    }
    finally
    {
        etwSession.Dispose();
    }
}

A TraceEventSession is created and passed to a dedicated thread to process the events until Stop is called at the end of the application.

The second step of an investigation is trying to reproduce the issue in a controlled environment such as… my developer machine. I setup the Exception Settings of the debugger to stop on any managed exception:

That way, if something bad happens while I’m debugging the application, Visual Studio tells me exactly where the exception was thrown.

After starting and stopping applications monitored by the metrics collector a few times, an exception was thrown in the code responsible for mapping the application id and the component in charge of storing the metrics of this process. When looking at the code, it seems that there was a “timing” conflict between the code in charge of detecting new and dead processes (in a timer) and the code receiving the events from ETW (in the dedicated thread described earlier). A CLR event was received after the corresponding process was detected as being dead. The net effect of the uncaught exception was fast: the TraceEvent session stopped its execution and the Process method returned. Nothing special visible outside of a debugger with the right exception settings. This is a great scenario to understand why swallowing all exceptions is not a good pattern…

Still not working

The next step is to fix the code to handle the dead process case, build it and test it. Unfortunately, the new metrics collector, from time to time, does not seem to receive any CLR event. Even worse, this time the ETW session is still here as shown by logman -ets. Going back inside the Visual Studio debugger, everything is working fine: the TraceEvent session is created, its Process method called and blocked in a dedicated thread and… the events are received! It means that I’m not able to reproduce the problem while running under debugger control. Maybe the problem could come from the code responsible to filter out events sent by unmonitored applications.

So, I’m adding a Breakpoint in the method responsible for checking monitored process to ensure that there is no bug there (events could be received but skipped due to invalid process ID mapping for example):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


private bool IsMonitoredEvent(TraceEvent traceEvent)
{
    var isMonitored = traceEvent.ProcessID == _monitoredPid;
    if (isMonitored)
    {
        NotifyProcessedEvent(traceEvent);
    }

    return isMonitored;
}

The breakpoint is hit and I’m able to validate that there is no problem in the mapping code

I can even check that the events I’m expecting are all received by asking Visual Studio to trace the event name with a tracepoint when isMonitored is true:

And I get all expected events in the Output Window. If some events were missing, it could have explained that metrics based on events series (such as contention duration) were not computed.

I’m now running the application outside of the debugger… and no event. Just to confirm that I’m not crazy, I decide to add traces in the source code:

But how to get the output without an attached debugger? The trick is to start SysInternals Debug View and wait for the event names to appear: nothing. Even by moving the Debug.WriteLine call outside of the if block, no event is ever received, even from unmonitored processes.

Navigating memory by stack frame

Let’s summarize the investigation status:

The metrics collector is working only when under the control of a debugger.
If the debugger is attached after it is started, the events are not received.

I don’t know why but this kind of weird behaviors is always happening at Criteo on a Friday. So let’s start a joined debugging session with Kevin, Jean-Philippe and Gregory!

To better understand what is the state of the metrics collector when the events are not received, the application is launched and the debugger is then attached to it. I open the Parallel Stacks panel and double-click the stack frame with a valuable context:

In our case, it would be interesting to get a view on the state of the ETWTraceEventSource object used by TraceEvent to process the events. Even if you don’t have the source code, it is still possible for the debugger to get a view of the object used as implicit “this” pointer by the ProcessOneFile method. Summon the Quick Watch dialog (Shift-F9 with my old VC 6 keyboard shortcut) and type “this”

Based on own understanding of how the ETWTraceEventSource is working, we know that registered event handlers are associated to entries in its templates field.

Instead of the expected GC, thread pool, exception and contention events, only kernel related events are defined:

But breakpoints have been set and hit on the code that registers our own event handlers! Well… it was the case when we debugged the application from start. What if… the event source we are looking at now is not the one our code has registered its handlers to?

Without the address of the object available like in C++, it is complicated to easily check if two references actually point to the same object in memory. However, it is still possible to associate a numeric ID to an object with Make Object ID:

And its ID {$1} is now visible after the type name:

To compare with the ETWTraceEventSource object manipulated by our own source code, double-click the right frame in Parallel Stack:

In the ListenToEtw method, this refers to our SessionManager in which the ClrTraceEventParser property references the ETWTraceEventSource object we believe we initialized with the right event handlers:

And… this object does not have any ID: it should be {$1}.

To confirm that we are not looking at the source object {$1} that TraceEvent uses to receive events, it is just a question of checking its template field:

And here are the expected CLR events we are interested in!

Race condition… again

So now the question is: how is it possible to have two instances of a TraceEvent internal class?

Here is the sequence of execution in our code:

Thread 1 is creating a TraceEventSession object
Thread 1 starts Thread 2
Thread 1 accesses the clrTraceEventParser via _currentSession?.Source.Clr

and

Thread 2 calls etwSession.Source.Process()

So the Source property getter could be called from two different threads. Unfortunately, in the getter, the source is lazily created in a non thread-safe way.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


public ETWTraceEventSource Source
{
   get
   {
      if (m_source == null)
      {
         ... // long code 
         m_source = new ETWTraceEventSource(SessionName, TraceEventSourceType.Session);
      }
      return m_source;
   }
}

When both threads enter the getter, m_source could be null and in that case, two ETWTraceEventSource objects are created and returned. One is used by TraceEvent to listen to events and the other by our code to register handlers to events that will never be received.

The fix is simply to force the initialization of the Source object in the first thread.

It is now a good time to go back home… to take a well deserved vacation!

In-process CLR event listeners with .NET Core 2.2

Thu, 06 Dec 2018 13:52:09 +0000

As the .NET Core 2.2 blog post introduced, it is now possible for a .NET Core application to listen to the events generated by the CLR that power it up. If you remember the Grab ETW Session, Providers and Events post, the CLR is emitting a lot of valuable events through ETW on Windows and LTTng on Linux. Thanks to TraceEvent nuget package, it is not that difficult to fetch these events at runtime on Windows, either in-process or out of process. However, it is much more complicated to achieve the same goal on Linux… With .NET Core 2.2, it is now super easy to listen to the events emitted by the CLR while your application is running: you simply need to implement a class that derives from System.Diagnostics.Tracing.EventListener and create an instance of it. Nothing more.

This class exists since .NET Framework 4.5 and .NET Core 1.0 but it could only be used to listen events pushed by managed code. Since .NET Core 2.2, it can also be used to listen to native events pushed by the CLR. The usage is simple, even a little bit magical.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48


sealed class GcFinalizersEventListener : EventListener
{
    // from https://docs.microsoft.com/en-us/dotnet/framework/performance/garbage-collection-etw-events
    private const int GC_KEYWORD =                 0x0000001;
    private const int TYPE_KEYWORD =               0x0080000;
    private const int GCHEAPANDTYPENAMES_KEYWORD = 0x1000000;

    protected override void OnEventSourceCreated(EventSource eventSource)
    {
        Console.WriteLine($"{eventSource.Guid} | {eventSource.Name}");

        // look for .NET Garbage Collection events
        if (eventSource.Name.Equals("Microsoft-Windows-DotNETRuntime"))
        {
            EnableEvents(
                eventSource, 
                EventLevel.Verbose, 
                (EventKeywords) (GC_KEYWORD | GCHEAPANDTYPENAMES_KEYWORD | TYPE_KEYWORD)
                );
        }
    }

    // from https://blogs.msdn.microsoft.com/dotnet/2018/12/04/announcing-net-core-2-2/
    // Called whenever an event is written.
    protected override void OnEventWritten(EventWrittenEventArgs eventData)
    {
        ...
    }
}

class Program
{
    static void Main(string[] args)
    {
        GcFinalizersEventListener listener = new GcFinalizersEventListener();

        Console.WriteLine("\nPress ENTER to trigger a few finalizers...");
        Console.ReadLine();
        for (int i = 0; i < 4; i++)
        {
            Thread t = new Thread(()=> {});
        }
        GC.Collect(2, GCCollectionMode.Forced, true, true);

        Console.WriteLine("\nPress ENTER to exit...");
        Console.ReadLine();
    }
}

First, you implement a class that derives from EventListener and override the following two methods:

1
2


   void OnEventSourceCreated(EventSource eventSource)
   void OnEventWritten(EventWrittenEventArgs eventData)

As soon as you new up an instance of your class, the OnEventSourceCreatedoverride is called for each event source defined in the application. An event source, as its name implies, produces events. You can define your own in managed code if you wish. For the sake of this post, I will focus on listening to the Microsoft-Windows-DotNETRuntime event source. The EventSourceinstance passed to the OnEventSourceCreatedmethod provides two interesting properties to let us identify the available sources. The following code :

1
2
3
4


protected override void OnEventSourceCreated(EventSource eventSource)
{
    Console.WriteLine($"{eventSource.Guid} | {eventSource.Name}");
}

generates the following output :

5e5bb766-bbfc-5662-0548-1d44fad9bb56 | Microsoft-Windows-DotNETRuntime
 2e5dba47-a3d2-4d16-8ee0-6671ffdcd7b5 | System.Threading.Tasks.TplEventSource
 8e9f5090-2d75-4d03-8a81-e5afbf85daf1 | System.Diagnostics.Eventing.FrameworkEventSource

You can imagine that the first one is the source we are interested in listening to its events!

By default, there is no connection between the sources and your listeners: you need to enable the source by calling the EnableEventsmethod in your OnEventSourceCreatedoverride. This EventListenermethod takes the following arguments:

EventSource eventSource: the event source you want to listen to
EventLevel level: minimum verbosity level for the received events
EventKeywords matchAnyKeyword: a keyword to filter on specific events

The Microsoft Docs provides the level and keywords for each events documented in the CLR. For the complete list, you take a look at ClrETWAll.man in CoreClr source code or in ClrTraceEventParser class of TraceEvent. In the sample code at the beginning of this post, I selected a group of keywords GC_KEYWORD | GCHEAPANDTYPENAMES_KEYWORD | TYPE_KEYWORD to receive only events related to the garbage collector and type information (read this previous post for more details).

Once the source has been paired to the listener, each time an event is emitted by the source with the right level and for the given keywords, the OnEventWrittenoverride will get called. The EventWrittenEventArgsinstance received as a parameter describes each event.

The Payload contains the value of the different properties stored in a ReadOnlyCollection and the corresponding property names are provided via the ReadOnlyCollection PayLoadNames. The following code shows how to extract all properties values:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


// from https://blogs.msdn.microsoft.com/dotnet/2018/12/04/announcing-net-core-2-2/
// Called whenever an event is written.
protected override void OnEventWritten(EventWrittenEventArgs eventData)
{
    // Write the contents of the event to the console.
    Console.WriteLine($"ThreadID = {eventData.OSThreadId} ID = {eventData.EventId} Name = {eventData.EventName}");
    for (int i = 0; i < eventData.Payload.Count; i++)
    {
        string payloadString = eventData.Payload[i] != null ? eventData.Payload[i].ToString() : string.Empty;
        Console.WriteLine($"    Name = \"{eventData.PayloadNames[i]}\" Value = \"{payloadString}\"");
    }
    Console.WriteLine("\n");
}

Here is the kind of output you get for common garbage collector and finalizer events:

ThreadID = 17456 ID = 200 Name = IncreaseMemoryPressure
Name = "BytesAllocated" Value = "1672"
Name = "ClrInstanceID" Value = "8"

ThreadID = 17456 ID = 9 Name = GCSuspendEEBegin_V1
Name = "Reason" Value = "1"
Name = "Count" Value = "0"
Name = "ClrInstanceID" Value = "8"

ThreadID = 17456 ID = 8 Name = GCSuspendEEEnd_V1
Name = "ClrInstanceID" Value = "8"

ThreadID = 17456 ID = 35 Name = GCTriggered
Name = "Reason" Value = "10"
Name = "ClrInstanceID" Value = "8"

ThreadID = 17456 ID = 1 Name = GCStart_V2
Name = "Count" Value = "1"
Name = "Depth" Value = "2"
Name = "Reason" Value = "10"
Name = "Type" Value = "0"
Name = "ClrInstanceID" Value = "8"
Name = "ClientSequenceNumber" Value = "0"

ThreadID = 18860 ID = 29 Name = FinalizeObject
Name = "TypeID" Value = "1210056592"
Name = "ObjectID" Value = "1371069040"
Name = "ClrInstanceID" Value = "8"

ThreadID = 18860 ID = 15 Name = BulkType
Name = "Count" Value = "1"
Name = "ClrInstanceID" Value = "8"

The fact that there is no strongly typed event argument per event is not as good as what TraceEvent provides. In addition, after a few tests, it seems that the .NET Core 2.2 implementation is not complete:

GC events are not all received when in Server Mode
Properties are missing for BulkType event necessary to figure out finalizer type names

However, with EventListener, Microsoft is giving us a very simple way to get valuable information, in-process, from the CLR while the application is running. A forthcoming blog post will show how to leverage this infrastructure to provide insights on how the garbage collection impacts an application.

Before leaving you building your own event listeners, you should know a couple of last details. Under the hood, the framework is creating a dedicated thread for you that will execute the two OnXXXmethods of your EventListener-derived class. It means that your code should not block or spend to much time processing the events if you want to keep on receiving events at a regular pace.

This thread will last as long as one of your listeners still exists. When I say “exist”, I mean until you decide to dispose them. This is the way for you to tell the sources that you are no more interested in receiving events. When all your listeners are disposed, then the processing thread will exit.