Cursor on Welcome to Christophe Nasarre's Blog

How to support .NET Framework PDB format and source line with ISymUnmanagedReader

Wed, 11 Feb 2026 09:16:11 +0000

In my previous posts, I explained how to use DIA and DbgHelp to map a method to its line in source code. I forgot to mention that it was correct for .NET Core but not for the “old” .NET Framework Windows PDB format. Instead of encoding the method token in the name, the symbol file contains the name of the methods. So, how to do the mapping for .NET Framework assemblies? You will find the answer (plus some tricks) in this article.

When I started to work on the support of the old Windows PDB format, I looked at what existed to parse the raw format and… I decided to try DbgHelp instead. With this first implementation, I realized that some token where missing and source code information was not retrieved for most of the methods.

So, I looked for another API to use and I found ISymUnmanagedReader. The usage philosophy is totally different from DIA or DbgHelp.

A little bit of magic

This interface is implemented in diasymreader.dll that comes with every .NET Framework installation. But you need to do COM magic to get it. After having called CoInitialize to setup COM, you ask for an instance of ISymUnmanagedBinder from CLSID_CorSymBinder_SxS:

1
2
3
4
5
6
7
8


CComPtr<ISymUnmanagedBinder> pBinder;
hr = CoCreateInstance(
    CLSID_CorSymBinder_SxS,
    NULL,
    CLSCTX_INPROC_SERVER,
    IID_ISymUnmanagedBinder,
    (void**)&pBinder
);

From the binder, you can get the ISymUnmanagedReader interface corresponding to the assembly you are interested in with GetReaderForFile. However, there are two tiny details to consider.

First, one parameter expects the path to the assembly, not to the .pdb file. That symbol file has to be stored in the same folder but note that the documentation states that you could have more flexible search with ISymUnmanagedBinder2::GetReaderForFile2 but I did not test it.

The second detail is the first parameter: an instance of IMetaDataImport for the same assembly. The steps to get it are… complicated.

Hosting the CLR

The idea is to host the .NET Framework and get the corresponding ICLRMetaHost interface:

1
2


CComPtr<ICLRMetaHost> pMetaHost;
HRESULT hr = CLRCreateInstance(CLSID_CLRMetaHost, IID_ICLRMetaHost, (void**)&pMetaHost);

Calling the CLRCreateInstance API allows you to get an instance of ICLRMetaHost from which you could enumerate installed version of .NET Framework. In my case, I know which version I want:

1
2
3


// Get the installed .NET Framework runtime (v4.0+)
CComPtr<ICLRRuntimeInfo> pRuntimeInfo;
hr = pMetaHost->GetRuntime(L"v4.0.30319", IID_ICLRRuntimeInfo, (void**)&pRuntimeInfo);

The ICLRRuntimeInfo interface allows you to get access to runtime services via GetInterface:

1
2


CComPtr<IMetaDataDispenser> pDispenser;
hr = pRuntimeInfo->GetInterface(CLSID_CorMetaDataDispenser, IID_IMetaDataDispenser, (void**)&pDispenser);

The service I’m interested in is the IMetadataDispenser interface that allows you to “open a scope” on the assembly you are interested in:

1
2
3
4
5
6


hr = pDispenser->OpenScope(
    wModulePath.c_str(),
    ofRead,
    IID_IMetaDataImport,
    (IUnknown**)&_pMetaDataImport
);

Note that the first parameter is the path to the assembly not to the .pdb file. The scope is abstracted by an IMetadataImport interface I have already described and that is needed to call GetReaderForFile: and get the ISymUnmanagedReader:

hr = pBinder->GetReaderForFile(_pMetaDataImport, wModulePath.c_str(), nullptr, &_pReader);

The road to get symbol details for a method

The ISymUnmanagedReader interface implements GetMethod to get details about a given method token via an ISymUnmanagedMethod interface. So, the next question is how to get these tokens. If you remember the previous article, these tokens are from the 06 MethodDef table in the assembly metadata; starting from 06000001 to the last one.

This means that you could write a simple loop starting from 1 up to a hardcoded maximum value, call TokenFromRid(index, mdtMethodDef) to get the corresponding token. However, since you are a professional developer, you would search for the exact number of tokens from IMetadataTables retrieved from IMetadataImport:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


ULONG cRows = 0;

// Get IMetaDataTables interface to query the MethodDef table
CComPtr<IMetaDataTables> pTables;
hr = _pMetaDataImport->QueryInterface(IID_IMetaDataTables, (void**)&pTables);
if (FAILED(hr) || pTables == nullptr)
{
    cRows = LAST_METHODDEF_TOKEN;
}
else
{
    // Get the number of rows in the MethodDef table (table index 0x06 = Method)
    hr = pTables->GetTableInfo(
        0x06,           // MethodDef table
        NULL,           // cbRow (not needed)
        &cRows,         // pcRows (number of methods)
        NULL,           // pcCols (not needed)
        NULL,           // piKey (not needed)
        NULL            // ppName (not needed)
    );

    if (FAILED(hr))
    {
        cRows = LAST_METHODDEF_TOKEN;
    }
}

Now that you have the number of rows (i.e. number of methods defined in the metadata), it is easy and safe to get method information from symbols:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


for (uint32_t i = 1; i <= cRows; i++)
{
    mdMethodDef token = TokenFromRid(i, mdtMethodDef);

    CComPtr<ISymUnmanagedMethod> pMethod;
    hr = _pReader->GetMethod(token, &pMethod);
    if (SUCCEEDED(hr) && pMethod != nullptr)
    {
        MethodInfo info;
        if (GetMethodInfoFromSymbol(pMethod, info))
        {
            _methods.push_back(info);
        }
    }
}

Note that GetMethod might fail (returning E_FAIL) for P/Invoked functions, abstract methods, or methods decorated with DebuggerHidden attribute.

Give me line and source code!

For the other methods with symbol information, you can get its token via the GetToken method. The ISymUnmanagedMethod interface allows low level access to line/column mapping that is beyond the scope of this article. At a high level, positions in source file are named sequence points. Call GetSequencePointCount to get… the number of sequence points for a given method.

The next step is to call GetSequencePoints with the number of points you want and the corresponding arrays of offsets, lines, columns, end lines, end columns and ISymUnmanagedDocument. In my case, I’m only interested in where the method starts so the first sequence point is good enough:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


// Get sequence points (source line information)
ULONG32 cPoints = 0;
hr = pMethod->GetSequencePointCount(&cPoints);
if (SUCCEEDED(hr) && (cPoints > 0))
{
    cPoints = 1; // We only need the first sequence point for start line
    std::vector<ULONG32> offsets(cPoints);
    std::vector<ULONG32> lines(cPoints);
    std::vector<ULONG32> columns(cPoints);
    std::vector<ULONG32> endLines(cPoints);
    std::vector<ULONG32> endColumns(cPoints);
    std::vector<ISymUnmanagedDocument*> documents(cPoints);

    ULONG32 actualCount = 0;
    hr = pMethod->GetSequencePoints(
        cPoints,
        &actualCount,
        &offsets[0],
        &documents[0],
        &lines[0],
        &columns[0],
        &endLines[0],
        &endColumns[0]
    );

    if (SUCCEEDED(hr) && (actualCount > 0))
    {

The source file is described by ISymUnmanagedDocument that provides its name when GetURL is called:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


// Get the first sequence point's document and line
        ISymUnmanagedDocument* pDoc = documents[0];
        if (pDoc != nullptr)
        {
            // Get document URL (file path)
            ULONG32 urlLen = 0;
            hr = pDoc->GetURL(0, &urlLen, NULL);
            if (SUCCEEDED(hr) && (urlLen > 0))
            {
                std::vector<WCHAR> url(urlLen);
                hr = pDoc->GetURL(urlLen, &urlLen, &url[0]);
                if (SUCCEEDED(hr))
                {
                    // Convert wide string to narrow string
                    int len = WideCharToMultiByte(CP_UTF8, 0, &url[0], urlLen, NULL, 0, NULL, NULL);
                    std::string narrowUrl(len, '\0');
                    WideCharToMultiByte(CP_UTF8, 0, &url[0], urlLen, &narrowUrl[0], len, NULL, NULL);
                    info.sourceFile = narrowUrl;
                }
            }

            // NOTE: 0xFEEFEE is a special value indicating hidden lines
            info.lineNumber = lines[0];
            pDoc->Release();
        }
    }
}

The final interesting trick is that the line number might have the special 0xFEEFEE value. It means that the line is hidden. I have seen it for methods generated by the C# compiler such as MoveNext for async state machines or anonymous methods:

The source code is available from my Github repository.

Happy coding!

References

But where is my method code? DbgHelp comes to the rescue

Fri, 16 Jan 2026 08:21:30 +0000

Introduction

In our Datatog continuous .NET profiler implementation, we collect the call stack of a thread when something interesting happens such as an exception is thrown for example. In addition to the method name we would like to figure out at what line in which source code file this method is implemented.

This information is usually stored in the program database (.pdb) file that is generated by the compiler when the assembly is generated from the source code. The type and the name of the method are stored in the metadata of the assembly itself but I already told this story before. The .NET compilers support two formats of .pdb: the Portable format for .NET Core and the Windows format for .NET Framework.

I’ve explained how to use the DIA API and it is now time to show how to leverage the DbgHelp API that is available on all Windows machines (even though it is recommended to always install the latest release via the Debugging Tools for Windows.

This time, my goal is to extract from a Windows .pdb file the source code and line information for a given managed method. You can find the corresponding source code of this DumpLine tool in my Github repository.

Starting with DbgHelp

There are two major ways to get access to symbols with DbgHelp: either from a running process (i.e. to map the currently loaded .dll to their associated .pdb files) or from a tool that would explicitly load a .dll or a .pdb file.

Before anything, you tell DbgHelp which options you want by calling SymSetOptions:

1
2
3
4
5
6
7
8


DWORD options = SymGetOptions();
    options |= SYMOPT_DEBUG;
    options |= SYMOPT_LOAD_LINES;           // Load line number information
    options |= SYMOPT_UNDNAME;              // Undecorate symbol names
    //options |= SYMOPT_DEFERRED_LOADS;       // Defer symbol loading
    options |= SYMOPT_EXACT_SYMBOLS;        // Require exact symbol match
    options |= SYMOPT_FAIL_CRITICAL_ERRORS; // Don't show error dialogs
    SymSetOptions(options);

This is where I ask that line number information should be collected.

The next step is to call SymInitialize to setup DbgHelp environment. The first parameter expects a process handle (returned by GetCurrentProcess in my case). You could pass a path where to find the .pdb files for your dlls as a second parameter. In my case, since I will provide a .pdb file path, I don’t need it, and NULL will be passed. It means that, if needed, DbgHelp will use the current folder and the path set in _NT_SYMBOL_PATH and _NT_ALTERNATE_SYMBOL_PATH environment variables.

The last boolean parameter tells DbgHelp if you want that SymLoadModule64 to be called for each and every loaded .dll in the given process. Definitively not what I want so I’m passing FALSE.

1
2
3
4
5


_hProcess = GetCurrentProcess();
    if (!SymInitialize(_hProcess, NULL, FALSE))
    {
        _hProcess = NULL;
    }

At that point, I’m ready to load a .pdb file.

Loading a .pdb file

The API is straightforward: just call SymLoadModuleEx :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


_baseAddress = SymLoadModuleEx(
        _hProcess,
        NULL,
        pdbFilePath.c_str(),
        NULL,
        0x10000000, // arbitrary base address
        0,
        NULL,
        0
    );

    if (_baseAddress == 0)
    {
        return false;
    }

The important parameters are the process handle (same as for SymInitialize) and the path of the .pdb file. I’ve lost some time trying to understand why my code was not working due to a weird behavior of this function. You know that it succeeds when the returned address is not 0. Well… This is not 100% correct. If the path you provide does not exist, you won’t get 0 but the base address that you also provide. Even worth, when you call the functions I’ll detail later on, no error will happen but nothing will work as expected. So, I simply check that the file exists:

1
2
3
4
5


// BUG? : dbghelp does not fail if the .pdb file does not exist...
    if (GetFileAttributesA(pdbFilePath.c_str()) == INVALID_FILE_ATTRIBUTES)
    {
        return false;
    }

Note that it is possible to unload the symbols of a given loaded module by calling SymUnloadModule with the same process handle and its base address: this will reduce the memory consumption if you don’t need the symbols anymore.

In case of deferred load symbols, it is needed to call SymGetModuleInfo64 before trying to access the symbols:

1
2
3
4
5
6


IMAGEHLP_MODULE64 moduleInfo = { 0 };
    moduleInfo.SizeOfStruct = sizeof(IMAGEHLP_MODULE64);
    if (!SymGetModuleInfo64(_hProcess, _baseAddress, &moduleInfo))
    {
        return false;
    }

In addition, this will fill up an IMAGEHLP_MODULE64 structure with possibly interesting details:

The .pdb signature and age could be useful to build urls to communicate with symbol servers; but this is another story:

1
2
3
4
5
6
7
8
9


_age = moduleInfo.PdbAge;
    GUID guid = moduleInfo.PdbSig70;
    char strGUID[80];
    sprintf_s(strGUID, 80, "%08x%04x%04x%02x%02x%02x%02x%02x%02x%02x%02x",
        guid.Data1, guid.Data2, guid.Data3,
        guid.Data4[0], guid.Data4[1], guid.Data4[2], guid.Data4[3],
        guid.Data4[4], guid.Data4[5], guid.Data4[6], guid.Data4[7]
        );
    _guid = strGUID;

You also know if line numbers are available or not thanks to the LineNumbers field.

Note that if you asked for deferred symbols option, you won’t get any interesting details:

Only the module name and path are provided but nothing else.

Enumerating the methods

It is now time to iterate on the symbols in the loaded .pdb thanks to SymEnumSymbols:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


if (!SymEnumSymbols(
            _hProcess,
            _baseAddress,
            "*!*",  // Mask (all symbols)
            EnumMethodSymbolsCallback,
            this    // User context to store the methods in _methods instance field
    ))
    {
        return false;
    }

In addition to the obvious parameters, this function expects a callback function that will be called for each symbol in the module specified by the process handle and the base address. Note that you can pass any context as the last parameter. In my case, the instance of my DbgHelpParser class is passed to be able to store the methods in a dedicated _methods field:

1
2
3
4
5
6
7
8
9


struct MethodInfo
{
    std::string name;
    uint64_t address;
    uint32_t size;
    std::string sourceFile;
    uint32_t lineNumber;
};
std::vector<MethodInfo> _methods;

The “*!*” mask tells DbgHelp to look for symbols in all modules. This might sound counter intuitive, but the syntax is similar to what you find in WinDBG or Visual Studio: !. This could be useful if you load more than one .pdb.

The job of the callback function is to detect the symbols you are interested in from the SYMBOL_INFO structure passed for each matching symbol:

1
2
3
4
5
6
7
8
9


BOOL CALLBACK DbgHelpParser::EnumMethodSymbolsCallback(PSYMBOL_INFO pSymInfo, ULONG SymbolSize, PVOID UserContext)
{
    DbgHelpParser* parser = reinterpret_cast<DbgHelpParser*>(UserContext);

    if (
        (pSymInfo->Tag == SymTagFunction) &&
        ((pSymInfo->Flags & (SYMFLAG_CLR_TOKEN | SYMFLAG_METADATA)) == (SYMFLAG_CLR_TOKEN | SYMFLAG_METADATA))
        )
    {

The Tag field contains a value from SymTagEnum but, for a managed .pdb file, you will only get SymTagFunction. Also, the Flags field should contain SYMFLAG_CLR_TOKEN and SYMFLAG_METADATA because we are only interested in managed methods.

Next, you get the name, address and size from other fields before looking for the source file and line details by calling SymGetLineFromAddr64:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


MethodInfo info;
        info.name = pSymInfo->Name;
        info.address = pSymInfo->Address;
        info.size = pSymInfo->Size;

        // Try to get source file and line information
        IMAGEHLP_LINE64 line = { 0 };
        line.SizeOfStruct = sizeof(IMAGEHLP_LINE64);
        DWORD displacement = 0;

        if (SymGetLineFromAddr64(parser->_hProcess, pSymInfo->Address, &displacement, &line))
        {
            info.sourceFile = line.FileName ? line.FileName : "";
            info.lineNumber = line.LineNumber;
        }
        else {
            info.sourceFile = "";
            info.lineNumber = 0;
        }

        parser->_methods.push_back(info);
    }

    return TRUE; // Continue enumeration
}

The callback returns TRUE to continue the enumeration. You could return FALSE if you would look for specific symbols and wanted to speed up the processing.

The managed side of the story

When the tool is run on a managed assembly, you get the following kind of output:

The name of the method does not match at all with the names in my test assembly! Why do all methods have this generic Method.# format?

Well… the number corresponds to the RID of the corresponding method in the metadata of the assembly. Let’s have a look at what the MethodDef metadata table of this assembly looks like in ILSpy:

The RID column corresponds to the number in the name in the tool output. So, the Method.#3 is the get_Records property getter in PDBFormatReaderTest.cs:28. And this is exactly what I can see in the test source code:

You can also check that the lines look correct compared to what is listed by the tool:

FilePath getter at line 19
FilePath setter at line 20
Records getter at line 28
Records setter at line 29

Feel free to look at the source code from my Github repository.

DbgHelp provides many more services to look into symbols but that’s all for today!

How to dump function symbols from a .pdb file

Mon, 08 Dec 2025 10:12:20 +0000

During this R&D week at Datadog, I wanted to implement a tool accepting a .pdb file and generate a .sym file listing functions symbols with their address, size, name with signature and if they are public or private. This post dig into the implementation details of using Microsoft Debug Interface Access (DIA) COM API to achieve these objectives. If you want to see what my vibe coding experience in Cursor was, read this other post instead.

One self-contained tool please!

I would like the tool to be self-contained but since DIA is based on a COM server, it would require registering msdia40.dll on the machine. Not a good idea. In case the dll is in the same folder as the tool, one could “emulate” the magic done by CoCreateInstance to get an instance of IDiaDataSource (more on this interface soon) by:

Call LoadLibrary to load the dll in memory
Call GetProcAddress to get the DllGetClassObject implementation
Call this function to get the IClassFactory implementation
Call its CreateInstance method to get an object implementing IDiaDataSource

Here is the corresponding code (without error checking for readability)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


// Create DIA data source without registration using DLL loading
HRESULT PdbSymbolExtractor::NoRegCoCreate(const std::wstring& dllPath, REFCLSID rclsid, REFIID riid, void** ppv) {
    HMODULE hDll = LoadLibraryW(dllPath.c_str());

    typedef HRESULT(__stdcall* DllGetClassObjectFunc)(REFCLSID, REFIID, LPVOID*);
    DllGetClassObjectFunc pDllGetClassObject = (DllGetClassObjectFunc)GetProcAddress(hDll, "DllGetClassObject");

    CComPtr<IClassFactory> pClassFactory;
    HRESULT hr = pDllGetClassObject(rclsid, IID_IClassFactory, (void**)&pClassFactory);

    hr = pClassFactory->CreateInstance(NULL, riid, ppv);

    // Note: We intentionally don't call FreeLibrary here because the DLL needs to stay loaded
    // The COM object references will keep it alive
    return S_OK;

This function is called with the path of msdia40.dll and the UUID of the expected IDiaDataSource:

1
2


// Create DIA data source without registration
hr = NoRegCoCreate(dllPath, CLSID_DiaSource, __uuidof(IDiaDataSource), (void**)&_pDiaDataSource);

But still, I don’t want to have two binaries!

The trick is to embed the msdia40.dll inside the tool as a Windows resource. In the .rc file, add an RCDATA entry that points to the dll:

1

IDR_MSDIA_DLL      RCDATA      "x64\\Release\\msdia140.dll"

You should see it in the Resource View in Visual Studio:

Here is the code that extracts it as a file on disk is straightforward (error checking has been removed for readability):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


// Extract embedded msdia140.dll from resources
bool PdbSymbolExtractor::ExtractEmbeddedDll(const std::wstring& outputPath) {
    // Find the resource
    HMODULE hModule = GetModuleHandle(NULL);
    HRSRC hResource = FindResource(hModule, MAKEINTRESOURCE(IDR_MSDIA_DLL), RT_RCDATA);

    // Load the resource
    HGLOBAL hLoadedResource = LoadResource(hModule, hResource);

    // Lock the resource to get a pointer to the data
    LPVOID pResourceData = LockResource(hLoadedResource);

    // Get the size of the resource
    DWORD resourceSize = SizeofResource(hModule, hResource);

    // Write the DLL to disk
    std::ofstream outFile(outputPath, std::ios::binary);
    outFile.write(static_cast<const char*>(pResourceData), resourceSize);
    outFile.close();

    return true;
}

Where are my function symbols?

After calling NoRegCoCreate(), _pDiaDataSource stores a reference to the entry point into the DIA APIs. Here are the steps to follow before being able to list the symbols:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


HRESULT PdbSymbolExtractor::ExtractSymbolsFromPdb(const std::wstring& pdbPath, std::vector<FunctionSymbol>& symbols) 
{
    // Load the PDB file
    HRESULT hr = _pDiaDataSource->loadDataFromPdb(pdbPath.c_str());

    // Open a session
    CComPtr<IDiaSession> pSession;
    hr = _pDiaDataSource->openSession(&pSession);

    // Get the global scope
    CComPtr<IDiaSymbol> pGlobal;
    hr = pSession->get_globalScope(&pGlobal);

Now, you have the global scope of the symbols, you can ask for an enumerator for the type of symbols you are interested in; SymTagFunction in my case:

1
2
3
4
5
6
7


// Enumerate all function symbols
    CComPtr<IDiaEnumSymbols> pEnumSymbols;
    hr = pGlobal->findChildren(SymTagFunction, NULL, nsNone, &pEnumSymbols);

    LONG count = 0;
    pEnumSymbols->get_Count(&count);
    std::wcout << L"Found " << count << L" function symbols" << std::endl;

The pEnumSymbols iterator allows you to loop on each SymTagFunction symbol and get its name:

1
2
3
4
5
6
7
8
9


while (SUCCEEDED(pEnumSymbols->Next(1, &pSymbol, &celt)) && celt == 1) {
        FunctionSymbol func;

        // Get function name
        BSTR bstrName;
        if (pSymbol->get_name(&bstrName) == S_OK) {
            func.name = bstrName;
            SysFreeString(bstrName);
        }

Note that each symbol details are stored in a FunctionSymbol instance:

1
2
3
4
5
6
7
8


struct FunctionSymbol {
    std::wstring name;
    ...
    std::wstring signature; // Function signature (parameters only, no return type)
    DWORD rva;              // Relative Virtual Address
    ULONGLONG length;
    bool isPublic;
};

with the rest of the code in the while() loop:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


// Get function signature (parameters only, no return type)
    func.signature = ExtractFunctionSignature(pSymbol);

    // Get relative virtual address
    DWORD rva;
    if (pSymbol->get_relativeVirtualAddress(&rva) == S_OK) {
        func.rva = rva;
    }

    // Get function length
    ULONGLONG length;
    if (pSymbol->get_length(&length) == S_OK) {
        func.length = length;
    }

    // Determine if function is public or private
    // Check access level - default to private
    func.isPublic = false;
    DWORD access;
    if (pSymbol->get_access(&access) == S_OK) {
        func.isPublic = (access == CV_public);
    }

    pSymbol.Release();

I did not have the time to do more trial for private/public state, but I should have tried by enumerating SymTagPublicSymbol or SymTagExport that could be considered as public.

Better with a signature

The final step is to figure out the signature of each function. This is where the genericity of DIA could be confusing because so many things are represented by IDiaSymbol: a symbol, a function, the type of a function, or the type of a parameter…

So, the type of the function is retrieved as an IDiaSymbol by calling getType() on the function symbol. From that IDiaSymbol, findChildren() lets you iterate on the parameters:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


// Extract function signature (parameters only, no return type)
std::wstring PdbSymbolExtractor::ExtractFunctionSignature(IDiaSymbol* pSymbol) {
    if (!pSymbol) {
        return L"()";
    }

    // Get function type
    CComPtr<IDiaSymbol> pFunctionType;
    if (pSymbol->get_type(&pFunctionType) != S_OK || !pFunctionType) {
        return L"()";
    }

    // Enumerate function arguments
    CComPtr<IDiaEnumSymbols> pEnumArgs;
    if (FAILED(pFunctionType->findChildren(SymTagFunctionArgType, NULL, nsNone, &pEnumArgs))) {
        return L"()";
    }

    LONG argCount = 0;
    pEnumArgs->get_Count(&argCount);

    if (argCount == 0) {
        return L"()";
    }

Now, the same Next() method is called on the enumerator to iterate on each parameter:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


// Build signature string
    std::wstring signature = L"(";
    CComPtr<IDiaSymbol> pArg;
    ULONG argCelt = 0;
    bool first = true;

    while (SUCCEEDED(pEnumArgs->Next(1, &pArg, &argCelt)) && argCelt == 1) {
        if (!first) {
            signature += L", ";
        }
        first = false;

        // Get the argument type
        CComPtr<IDiaSymbol> pArgType;
        if (pArg->get_type(&pArgType) == S_OK && pArgType) {
            signature += GetTypeName(pArgType);
        } else {
            signature += L"?";
        }

        pArg.Release();
    }

    signature += L")";
    return signature;
}

The final step is to get the name of the type from the IDiaSymbol returned by get_type(). If it is a custom type, call get_name() like any other symbol. Otherwise, for basic types, call get_baseType() and get_length() as shown by the code below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55


std::wstring PdbSymbolExtractor::GetTypeName(IDiaSymbol* pType) {
    if (!pType) {
        return L"?";
    }

    // Try to get type name directly
    BSTR bstrTypeName;
    if (pType->get_name(&bstrTypeName) == S_OK && bstrTypeName && wcslen(bstrTypeName) > 0) {
        std::wstring typeName = bstrTypeName;
        SysFreeString(bstrTypeName);
        return typeName;
    }

    // For basic types or unnamed types, try getting basic type info
    DWORD baseType = 0;
    ULONGLONG length = 0;

    if (pType->get_baseType(&baseType) == S_OK) {
        pType->get_length(&length);

        // Map basic types to names
        switch (baseType) {
            case btVoid: return L"void";
            case btChar: return L"char";
            case btWChar: return L"wchar_t";
            case btBool: return L"bool";

            case btInt:
            case btLong:
                if (length == 1) return L"char";
                else if (length == 2) return L"short";
                else if (length == 4) return L"int";
                else if (length == 8) return L"__int64";
                else return L"int" + std::to_wstring(length * 8);

            case btUInt:
            case btULong:
                if (length == 1) return L"unsigned char";
                else if (length == 2) return L"unsigned short";
                else if (length == 4) return L"unsigned int";
                else if (length == 8) return L"unsigned __int64";
                else return L"uint" + std::to_wstring(length * 8);

            case btFloat:
                if (length == 4) return L"float";
                else if (length == 8) return L"double";
                else return L"float" + std::to_wstring(length * 8);

            default:
                return L"?";
        }
    }

    return L"?";
}

This is a “simple” implementation that does not take pointers, addresses, arrays, and more into account. For a more complete solution, I would recommend looking at the PrintType() implementation in the DIA2Dump code sample that is installed with Visual Studio.

I hope this will get your foot in the door of symbol parsing and make you want to dig further into DIA.

References

Corresponding source code is available in my github repository.
Archived Microsoft documentation/implementation of .pdb format including a symbol dumper code.
DIA2Dump Visual Studio code sample.

Vibe coding a .pdb dumper or how I became a Product Manager

Mon, 08 Dec 2025 10:12:16 +0000

One self-contained tool please!

Call LoadLibrary to load the dll in memory
Call GetProcAddress to get the DllGetClassObject implementation
Call this function to get the IClassFactory implementation
Call its CreateInstance method to get an object implementing IDiaDataSource

Here is the corresponding code (without error checking for readability)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


// Create DIA data source without registration using DLL loading
HRESULT PdbSymbolExtractor::NoRegCoCreate(const std::wstring& dllPath, REFCLSID rclsid, REFIID riid, void** ppv) {
    HMODULE hDll = LoadLibraryW(dllPath.c_str());

    typedef HRESULT(__stdcall* DllGetClassObjectFunc)(REFCLSID, REFIID, LPVOID*);
    DllGetClassObjectFunc pDllGetClassObject = (DllGetClassObjectFunc)GetProcAddress(hDll, "DllGetClassObject");

    CComPtr<IClassFactory> pClassFactory;
    HRESULT hr = pDllGetClassObject(rclsid, IID_IClassFactory, (void**)&pClassFactory);

    hr = pClassFactory->CreateInstance(NULL, riid, ppv);

    // Note: We intentionally don't call FreeLibrary here because the DLL needs to stay loaded
    // The COM object references will keep it alive
    return S_OK;

This function is called with the path of msdia40.dll and the UUID of the expected IDiaDataSource:

1
2


// Create DIA data source without registration
hr = NoRegCoCreate(dllPath, CLSID_DiaSource, __uuidof(IDiaDataSource), (void**)&_pDiaDataSource);

But still, I don’t want to have two binaries!

The trick is to embed the msdia40.dll inside the tool as a Windows resource. In the .rc file, add an RCDATA entry that points to the dll:

1

IDR_MSDIA_DLL      RCDATA      "x64\\Release\\msdia140.dll"

You should see it in the Resource View in Visual Studio:

Here is the code that extracts it as a file on disk is straightforward (error checking has been removed for readability):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


// Extract embedded msdia140.dll from resources
bool PdbSymbolExtractor::ExtractEmbeddedDll(const std::wstring& outputPath) {
    // Find the resource
    HMODULE hModule = GetModuleHandle(NULL);
    HRSRC hResource = FindResource(hModule, MAKEINTRESOURCE(IDR_MSDIA_DLL), RT_RCDATA);

    // Load the resource
    HGLOBAL hLoadedResource = LoadResource(hModule, hResource);

    // Lock the resource to get a pointer to the data
    LPVOID pResourceData = LockResource(hLoadedResource);

    // Get the size of the resource
    DWORD resourceSize = SizeofResource(hModule, hResource);

    // Write the DLL to disk
    std::ofstream outFile(outputPath, std::ios::binary);
    outFile.write(static_cast<const char*>(pResourceData), resourceSize);
    outFile.close();

    return true;
}

Where are my function symbols?

After calling NoRegCoCreate(), _pDiaDataSource stores a reference to the entry point into the DIA APIs. Here are the steps to follow before being able to list the symbols:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


HRESULT PdbSymbolExtractor::ExtractSymbolsFromPdb(const std::wstring& pdbPath, std::vector<FunctionSymbol>& symbols) 
{
    // Load the PDB file
    HRESULT hr = _pDiaDataSource->loadDataFromPdb(pdbPath.c_str());

    // Open a session
    CComPtr<IDiaSession> pSession;
    hr = _pDiaDataSource->openSession(&pSession);

    // Get the global scope
    CComPtr<IDiaSymbol> pGlobal;
    hr = pSession->get_globalScope(&pGlobal);

Now, you have the global scope of the symbols, you can ask for an enumerator for the type of symbols you are interested in; SymTagFunction in my case:

1
2
3
4
5
6
7


// Enumerate all function symbols
    CComPtr<IDiaEnumSymbols> pEnumSymbols;
    hr = pGlobal->findChildren(SymTagFunction, NULL, nsNone, &pEnumSymbols);

    LONG count = 0;
    pEnumSymbols->get_Count(&count);
    std::wcout << L"Found " << count << L" function symbols" << std::endl;

The pEnumSymbols iterator allows you to loop on each SymTagFunction symbol and get its name:

1
2
3
4
5
6
7
8
9


while (SUCCEEDED(pEnumSymbols->Next(1, &pSymbol, &celt)) && celt == 1) {
        FunctionSymbol func;

        // Get function name
        BSTR bstrName;
        if (pSymbol->get_name(&bstrName) == S_OK) {
            func.name = bstrName;
            SysFreeString(bstrName);
        }

Note that each symbol details are stored in a FunctionSymbol instance:

1
2
3
4
5
6
7
8


struct FunctionSymbol {
    std::wstring name;
    ...
    std::wstring signature; // Function signature (parameters only, no return type)
    DWORD rva;              // Relative Virtual Address
    ULONGLONG length;
    bool isPublic;
};

with the rest of the code in the while() loop:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


// Get function signature (parameters only, no return type)
    func.signature = ExtractFunctionSignature(pSymbol);

    // Get relative virtual address
    DWORD rva;
    if (pSymbol->get_relativeVirtualAddress(&rva) == S_OK) {
        func.rva = rva;
    }

    // Get function length
    ULONGLONG length;
    if (pSymbol->get_length(&length) == S_OK) {
        func.length = length;
    }

    // Determine if function is public or private
    // Check access level - default to private
    func.isPublic = false;
    DWORD access;
    if (pSymbol->get_access(&access) == S_OK) {
        func.isPublic = (access == CV_public);
    }

    pSymbol.Release();

I did not have the time to do more trial for private/public state, but I should have tried by enumerating SymTagPublicSymbol or SymTagExport that could be considered as public.

Better with a signature

So, the type of the function is retrieved as an IDiaSymbol by calling getType() on the function symbol. From that IDiaSymbol, findChildren() lets you iterate on the parameters:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


// Extract function signature (parameters only, no return type)
std::wstring PdbSymbolExtractor::ExtractFunctionSignature(IDiaSymbol* pSymbol) {
    if (!pSymbol) {
        return L"()";
    }

    // Get function type
    CComPtr<IDiaSymbol> pFunctionType;
    if (pSymbol->get_type(&pFunctionType) != S_OK || !pFunctionType) {
        return L"()";
    }

    // Enumerate function arguments
    CComPtr<IDiaEnumSymbols> pEnumArgs;
    if (FAILED(pFunctionType->findChildren(SymTagFunctionArgType, NULL, nsNone, &pEnumArgs))) {
        return L"()";
    }

    LONG argCount = 0;
    pEnumArgs->get_Count(&argCount);

    if (argCount == 0) {
        return L"()";
    }

Now, the same Next() method is called on the enumerator to iterate on each parameter:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


// Build signature string
    std::wstring signature = L"(";
    CComPtr<IDiaSymbol> pArg;
    ULONG argCelt = 0;
    bool first = true;

    while (SUCCEEDED(pEnumArgs->Next(1, &pArg, &argCelt)) && argCelt == 1) {
        if (!first) {
            signature += L", ";
        }
        first = false;

        // Get the argument type
        CComPtr<IDiaSymbol> pArgType;
        if (pArg->get_type(&pArgType) == S_OK && pArgType) {
            signature += GetTypeName(pArgType);
        } else {
            signature += L"?";
        }

        pArg.Release();
    }

    signature += L")";
    return signature;
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55


std::wstring PdbSymbolExtractor::GetTypeName(IDiaSymbol* pType) {
    if (!pType) {
        return L"?";
    }

    // Try to get type name directly
    BSTR bstrTypeName;
    if (pType->get_name(&bstrTypeName) == S_OK && bstrTypeName && wcslen(bstrTypeName) > 0) {
        std::wstring typeName = bstrTypeName;
        SysFreeString(bstrTypeName);
        return typeName;
    }

    // For basic types or unnamed types, try getting basic type info
    DWORD baseType = 0;
    ULONGLONG length = 0;

    if (pType->get_baseType(&baseType) == S_OK) {
        pType->get_length(&length);

        // Map basic types to names
        switch (baseType) {
            case btVoid: return L"void";
            case btChar: return L"char";
            case btWChar: return L"wchar_t";
            case btBool: return L"bool";

            case btInt:
            case btLong:
                if (length == 1) return L"char";
                else if (length == 2) return L"short";
                else if (length == 4) return L"int";
                else if (length == 8) return L"__int64";
                else return L"int" + std::to_wstring(length * 8);

            case btUInt:
            case btULong:
                if (length == 1) return L"unsigned char";
                else if (length == 2) return L"unsigned short";
                else if (length == 4) return L"unsigned int";
                else if (length == 8) return L"unsigned __int64";
                else return L"uint" + std::to_wstring(length * 8);

            case btFloat:
                if (length == 4) return L"float";
                else if (length == 8) return L"double";
                else return L"float" + std::to_wstring(length * 8);

            default:
                return L"?";
        }
    }

    return L"?";
}

I hope this will get your foot in the door of symbol parsing and make you want to dig further into DIA.

References

Corresponding source code is available in my github repository.
Archived Microsoft documentation/implementation of .pdb format including a symbol dumper code.
DIA2Dump Visual Studio code sample.

How to monitor .NET applications startup

Thu, 13 Mar 2025 10:50:55 +0000

In the previous article, I presented what is needed (i.e. listen to WaitHandleWait events) to compute lock/wait durations and call stacks for Mutex, Semaphore, SemaphoreSlim, Manual/AutoResetEvent, ManualResetEventSlim, ReaderWriterLockSlim .NET synchronization constructs for a running process.

However, since the application is already running, some JIT-related events are missing, and some frames of the call stacks cannot be symbolized. Also, it would be great to monitor an application’s startup to see if it could be faster.

This post will detail how to monitor a .NET application since the very beginning of its life and the issues you might face.

Preparing a new .NET process to be monitored

From .NET 5, the dotnet-trace CLI tool allows you to pass a command line to execute and trace it from startup. In a very interesting article, Olivier Coanet presented the gory details about how to tell the .NET runtime to start an application in a pseudo-suspended mode as shown in the following diagram:

The first step is to create a ReverseDiagnosticsServer instance with a specific port (i.e. dotnet-wait_1234 in the diagram). Next, the process to monitor is spawned with the DOTNET_DiagnosticPorts environment variable set to the same port (i.e. dotnet-wait_1234). Look at the Diagnostics documentation of the Diagnostic Ports with DOTNET_DiagnosticPorts environment variable for more details. The .NET runtime is the new process will listen to this port and… wait.

When the tool is ready, it sends a resume command via a DiagnosticsClient: from that point in time, the CLR executes the normal flow of actions to run the application and… you will receive all events without missing one!

Get my command line please

Following the dotnet-trace example, my dotnet-wait tool accepts the command line of the child process in its final arguments that follow the** — **trigger. For example, dotnet-wait — dotnet foo.dll will start the program in foo.dll by using dotnet.exe. I’m reusing the code in ReversedServerHelper.cs to deal with arguments containing spaces:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


else if (current == "--")  // this is supposed to be the last one
{
    i++;
    if (i < args.Length)
    {
        parameters.pathName = args[i];

        // use the remaining arguments as the arguments for the child app to spawn
        i++;
        if (i < args.Length)
        {
            parameters.arguments = "";
            for (int j = i; j < args.Length; j++)
            {
                if (args[j].Contains(' '))
                {
                    parameters.arguments += $"\"{args[j].Replace("\"", "\\\"")}\"";
                }
                else
                {
                    parameters.arguments += args[j];
                }

                if (j != args.Length)
                {
                    parameters.arguments += " ";
                }
            }
        }

        // no need to look for more arguments
        break;
    }
    else
    {
        throw new InvalidOperationException($"Missing path name value...");
    }
}

The code to spawn the child process is simple:

1
2
3
4
5
6
7
8
9


// start the monitored app
var psi = new ProcessStartInfo(pathName);
if (!string.IsNullOrEmpty(arguments))
{
    psi.Arguments = arguments;
}
psi.EnvironmentVariables["DOTNET_DiagnosticPorts"] = port;
psi.UseShellExecute = false;
var process = System.Diagnostics.Process.Start(psi);

Here is an example with the following prompt:

-- dotnet "C:\CommandLineTest.dll" one two "t h r e e" four 'five six'

that generates the output (the test application is just listing its arguments):

dotnet-wait v1.0.0.0 - List wait duration
by Christophe Nasarre

Press ENTER to exit...
6 arguments
   1 | one
   2 | two
   3 | t h r e e
   4 | four
   5 | 'five
   6 | six'

This test reminded me to never use simple quotes in prompts :^)

It’s my console!

Once I implemented these steps, I immediately faced a very simple problem: my dotnet-wait tool and the test application are console applications. It means that they will share the same console for both input and output. For example, both are waiting for the RETURN key to (1) stop for the tool and (2) start for the test application: too bad for me because the tool will stop as soon the application starts…

Going back in time in my Windows memories, I remembered that the Win32 CreateProcess API accepts CREATE_NEW_CONSOLE as creation flag to automagically start the child process into its own new console. Unfortunately, it is not possible to pass this flag in .NET; maybe a limitation due to Linux support.

One simple solution could be to redirect the output of the tool or the application to a file: that would avoid mixing them in the console. Note that, by default, dotnet-trace discards output from the child process (by setting RedirectStandardOutput, RedirectStandardError and RedirectStandardInput to false and by ignoring the error and output streams) except if you pass — show-child-io on the command line. In this case, no output for dotnet-trace.

I decided to do the opposite for dotnet-wait: by default, you also get the child output but you can redirect the output of the tool to a file with -o . Still, this does not solve the input problem in case of common expected keys.

If you remember the interactions between the tool and the monitored application, the latter is suspended until DiagnosticsClient::ResumeRuntime is called. So, why not starting the tool that spawns the application in one console and another instance of the tool in a new console that will resume the application? This is exactly what my friend Kevin Gosse imagined and how dotnet-wait works.

After the timeout that you give to diagnosticsServer.AcceptAsync(cancellation.Token) has elapsed, the runtime in the child process will display the following message:

The runtime has been configured to pause during startup and is awaiting a Diagnostics IPC ResumeStartup command from a Diagnostic Port.
DOTNET_DiagnosticPorts="dotnet-wait_34296"
DOTNET_DefaultDiagnosticPortSuspend=0

And this is exactly what the -r 34296 parameter will do!

You can now install dotnet wait and monitor the lock and wait contentions of your .NET9+ applications.

Measuring the impact of locks and waits on latency in your .NET apps

Mon, 13 Jan 2025 15:31:03 +0000

Introduction

In an old post, I detailed how to use ContentionStart and ContentionStop events to measure the lock contentions duration for a .NET application. In a .NET 9 pull request, a former Criteo’s colleague Grégoire Verdier has added new events to be notified when wait time similar to lock contention is happening for Mutex, Semaphore, Manual/AutoResetEvent. Read his post for more details about what he was trying to investigate.

With asynchronous and multi-threaded algorithms, it is essential to detect unexpected wait/locks in our applications. This post shows you how to leverage these events to measure the duration of these waits and get the call stack when the wait started:

New WaitHandleWait events

These new events are emitted by the Microsoft-Windows-DotNETRuntime CLR provider when you enable the WaitHandle (= 0x40000000000) keyword with Verbose verbosity. Each time WaitOne is called on a waitable object and this object is already owned, a WaitHandleWaitStart event is emitted. When the object is released, a WaitHandleWaitStop event is emitted.

For example, the following code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32


static Mutex mutex = new Mutex();

static void Main()
{
    var owningThread = new Thread(OwningThread);
    owningThread.Start();
    var mutexThread = new Thread(MutexThread);
    mutexThread.Start();

    owningThread.Join();
    mutexThread.Join();
}

static void OwningThread()
{
    Console.WriteLine($"    [{GetCurrentThreadId(), 8}] Start to hold resources");
    Console.WriteLine("___________________________________________");
    mutex.WaitOne();

    Thread.Sleep(3000);  // the wait should last ~3 seconds

    Console.WriteLine("    Release resources");
    mutex.ReleaseMutex();
}

static void MutexThread()
{
    Console.WriteLine($"    [{GetCurrentThreadId(), 8}] waiting for Mutex...");
    mutex.WaitOne();  // events are emitted in the implementation when a contention happens
    mutex.ReleaseMutex();
    Console.WriteLine("    <-- Mutex");
}

generates a Start and Stop events pair:

1
2


125980 | 00000000-0000-0000-0000-000000000000 > event 301 __ [ 1| Start] WaitHandleWait/Start
125980 | 00000000-0000-0000-0000-000000000000 > event 302 __ [ 2|  Stop] WaitHandleWait/Stop

There is no associated activity ID so you rely on the fact that the same waiter thread (125980 in the previous example) is emitting for both events.

Listening to the new Wait events

As usual, you should rely on the TraceEvent nuget to start an EventPipe session with an already running .NET application. The last version already contains the definition of the keyword:

1

keywords |= ClrTraceEventParser.Keywords.WaitHandle; // .NET 9 WaitHandle kind of contention

and the C# events for Start and Stop:

1
2


source.Clr.WaitHandleWaitStart += OnWaitHandleWaitStart;
source.Clr.WaitHandleWaitStop += OnWaitHandleWaitStop;

The handler’s implementation is straightforward. The start of the wait is recorded for the current thread:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


private void OnWaitHandleWaitStart(WaitHandleWaitStartTraceData data)
{
    // get the contention info for the current thread
    ContentionInfo info = _contentionStore.GetContentionInfo(data.ProcessID, data.ThreadID);
    if (info == null)
        return;

    // keep track of the wait start
    info.ContentionStartRelativeMSec = data.TimeStampRelativeMSec;
}

When the wait ends, the duration is computed based on the recorded wait start because it is not provided in the payload like for ContentionStop:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


private void OnWaitHandleWaitStop(WaitHandleWaitStopTraceData data)
{
    ContentionInfo info = _contentionStore.GetContentionInfo(data.ProcessID, data.ThreadID);
    if (info == null)
        return;
    // unlucky case when we start to listen just after the WaitHandleStart event
    if (info.ContentionStartRelativeMSec == 0)
    {
        return;
    }

    // Too bad the duration is not provided in the payload like in ContentionStop...
    var contentionDurationMSec = data.TimeStampRelativeMSec - info.ContentionStartRelativeMSec;
    info.ContentionStartRelativeMSec = 0;
    var duration = TimeSpan.FromMilliseconds(contentionDurationMSec);
    Console.WriteLine($"{e.ThreadId,7} | {e.Duration.TotalMilliseconds} ms");
}

This is nice but it would be more useful if we could get the call stack of long waits.

Call stacks with EventPipe

In a previous post, I explained that it is possible to get the call stack when an event is emitted thanks to the ClrStackWalk event that follows the event you are interested in. Unfortunately, this is not more the case for .NET 5+ that is using EventPipe instead of ETW.

As Olivier Coanet presents in his post, you can get the call stack as an array of addresses from the hidden event record that is mapped by the TraceEvent parameter passed to each event handlers. This EVENT_RECORD structure contains a ExtendedData field that is an array of EVENT_HEADER_EXTENDED_DATA_ITEM:

1
2
3
4
5
6
7
8


public struct EVENT_HEADER_EXTENDED_DATA_ITEM
{
    public ushort Reserved1;
    public ushort ExtType;
    public ushort Reserved2;
    public ushort DataSize;
    public ulong DataPtr;
}

If the ExtType value is EVENT_HEADER_EXT_TYPE_STACK_TRACE64 (=6) then DataPtr points to a EVENT_EXTENDED_ITEM_STACK_TRACE64 structure:

1
2
3
4
5


public struct EVENT_EXTENDED_ITEM_STACK_TRACE64
{
    public ulong MatchId;
    public unsafe fixed ulong Address[1];
}

that contains an array of 64-bit addresses. The size of this array is given by DataSize — sizeof(ulong).

For 32-bit applications, you will get EVENT_HEADER_EXT_TYPE_STACK_TRACE32 (=5) as ExtType value and DataPtr will point to EVENT_EXTENDED_ITEM_STACK_TRACE32:

1
2
3
4
5


public struct EVENT_EXTENDED_ITEM_STACK_TRACE32
{
    public ulong MatchId;
    public unsafe fixed uint Address[1];
}

that stores an array of 32-bit addresses.

Knowing that makes writing the code to get the call stacks as an array of 64-bit addresses (same with 32-bit applications for simplicity sake) pretty straightforward:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50


public static EventPipeUnresolvedStack ReadStackUsingUnsafeAccessor(TraceEvent traceEvent)
{
    return GetFromEventRecord(traceEvent.eventRecord);
}

private static EventPipeUnresolvedStack GetFromEventRecord(TraceEventNativeMethods.EVENT_RECORD* eventRecord)
{
    if (eventRecord == null)
        return null;

    var extendedDataCount = eventRecord->ExtendedDataCount;

    for (var dataIndex = 0; dataIndex < extendedDataCount; dataIndex++)
    {
        var extendedData = eventRecord->ExtendedData[dataIndex];
        if (extendedData.ExtType == TraceEventNativeMethods.EVENT_HEADER_EXT_TYPE_STACK_TRACE64)
        {
            var stackRecord = (TraceEventNativeMethods.EVENT_EXTENDED_ITEM_STACK_TRACE64*)extendedData.DataPtr;
            var addresses = &stackRecord->Address[0];
            var addressCount = (extendedData.DataSize - sizeof(UInt64)) / sizeof(UInt64);
            if (addressCount == 0)
                return null;

            var callStackAddresses = new ulong[addressCount];
            for (var index = 0; index < addressCount; index++)
            {
                callStackAddresses[index] = addresses[index];
            }
            return new EventPipeUnresolvedStack(callStackAddresses);
        }
        else if (extendedData.ExtType == TraceEventNativeMethods.EVENT_HEADER_EXT_TYPE_STACK_TRACE32)
        {
            var stackRecord = (TraceEventNativeMethods.EVENT_EXTENDED_ITEM_STACK_TRACE32*)extendedData.DataPtr;
            var addresses = &stackRecord->Address[0];
            var addressCount = (extendedData.DataSize - sizeof(UInt32)) / sizeof(UInt32);
            if (addressCount == 0)
                return null;

            var callStackAddresses = new ulong[addressCount];  // store the 32 addresses as 64 bit addresses
            for (var index = 0; index < addressCount; index++)
            {
                callStackAddresses[index] = addresses[index];
            }

            return new EventPipeUnresolvedStack(callStackAddresses);
        }
    }

    return null;
}

Note that the last version of TraceEvent nuget provides a public access to the eventRecord field so it is no more needed to use the UnsafeAccessor attribute used by Olivier.

Symbolize the call stack addresses

Address is good but the corresponding method name is better. I won’t repeat what I’ve already detailed in an older post that shows how to get the name of a native and managed name from an instruction pointer address. Instead, I want to pinpoint a big limitation of this solution to listen to CLR provider MethodLoadVerbose/MethodDCStartVerboseV2 events. If the methods you are interested in are jitted BEFORE your tool attaches to the application, you will never get these events.

You could get the same mapping address span/method name via the other “Microsoft-Windows-DotNETRuntimeRundown” provider and its MethodDCEndVerbose event that contains the expected MethodStartAddress, MethodSize and MethodName in its payload. But I need this information before the end of the application…

Looking at the documentation, it seems that the rundown provider accepts the StartRundownKeyword value to emit the DCStart events when the provider is enabled! Since .NET 9, it is possible to pass the keywords you want (before, the default value did not contain StartRundownKeyword) when creating the EventPipe session

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


//                V-- this is the default rundown keyword
rundownKeywords = 0x80020139 | (long)ClrTraceEventParser.Keywords.StartEnumeration;
var config = new EventPipeSessionConfiguration(GetProviders(), 256, rundownKeywords, true);
using (var session = client.StartEventPipeSession(config))
{
    var source = new EventPipeEventSource(session.EventStream);
    RegisterListeners(source);

    // this is a blocking call
    source.Process();
}

Note that you should not add the rundown provider to the list passed as parameter.

Unfortunately, there is currently an issue in the runtime since September 2020 that pinpoints this exact problem. I even tried to create and close a session to get the DCStop events before recreating a new one, but I failed.

The next episode will talk about how it is possible to start a .NET application and get the events since its startup… with the problems that are happening.

Monitor HTTP redirects to reduce unexpected latency

Fri, 13 Dec 2024 14:21:41 +0000

In the previous post, I detailed how I used the undocumented events from the BCL to create the dotnet-http CLI tool to monitor your outgoing HTTP requests. After testing with older versions of .NET, I realized that the code needed to be updated and I’m sharing my findings in this post.

The main point is that url redirections could have a major impact on requests latency:

Always test supported versions…

When I wrote the initial version of dotnet-http, I only tested it with .NET 8 and .NET 9 with limited formats of urls. Unfortunately, things went bad when I tried to monitor applications running on .NET 5 and .NET 6: no events are emitted by these versions of the BCL.

So, the next step was to test .NET 7 and the result was simple: crash! After investigating, I realized that some events I looked at in .NET 8 source code did not have the same payload in .NET 7; even no payload at all:

Even though there is no version field in the events payload, it is easy to check its size such as the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


private void OnConnectionEstablished(
     DateTime timestamp,
     int threadId,
     Guid activityId,
     Guid relatedActivityId,
     byte[] eventData
     )
 {
     ...
     EventSourcePayload payload = new EventSourcePayload(eventData);
     var versionMajor = payload.GetByte();
     var versionMinor = payload.GetByte();

     // in .NET 7, nothing else is available
     Int64 connectionId = 0;
     var scheme = "";
     var host = "";
     UInt32 port = 0;
     var path = "";

     if (eventData.Length > 2)
     {
         connectionId = payload.GetInt64();
         scheme = payload.GetString();
         host = payload.GetString();
         port = payload.GetUInt32();
         path = payload.GetString();
     }
     ...

Another difference is that one even is not even emitted in .NET 7:

Explain what a redirection is please!

Because I based the code on this Redirect event, I needed to find another way to support .NET 7 even though I would not have a redirected url to display. But first, let’s see what I’m talking about in terms of HTTP communication.

When you try to get the content / status code behind a url, the code is following different phases with the related events:

**Start **RequestStart
**DNS resolution **ResolutionStart ResolutionStop/Fail
**Socket connection **ConnectStart ConnectStop
**Security hand check (HTTPS only) **HandshakeStart HandshakeStop/Failed
**Request/response **RequestHeadersStart RequestHeadersStop ResponseHeadersStart ResponseHeadersStop Redirect (.NET 8+) ResponseContentStart ResponseContentStop
**Request stop **RequestStop/Failed

Based on the received url, a server can decide to answer that another url should be used instead. For example, if you call github with http:// instead of https://, such a redirection will happen. Without invasive tools such as Wireshark, these redirections are impossible to detect and could cause unnecessary delay.

From the client perspective, this can be detected in the ResponseHeadersStop event payload that provides a status code. If its value is 301, then it is a redirection. The other effect of a redirection is that the BCL code will change the flow of events because it needs to start over with the new url from step 2. to step 6. As you can see, instead of paying the cost of just one request, two are actually emitted and processed.

Impact on the implementation

In addition to the payload size checks addition, my initial implementation was not properly handling the redirection because the values (timestamps and durations) where overridden by the events related to the second redirected url.

The new implementation is splitting the request details into two classes. A base class that contains common fields to both parts of the request in case of redirection:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


private class HttpRequestInfoBase
   {
       public HttpRequestInfoBase(DateTime timestamp, string scheme, string host, uint port, string path)
       {
           StartTime = timestamp;
           if (scheme == string.Empty)
           {
               Url = string.Empty;
           }
           else
           {
               if (port != 0)
               {
                   Url = $"{scheme}://{host}:{port}{path}";
               }
               else
               {
                   Url = $"{scheme}://{host}:{path}";
               }
           }
       }

       public string Url { get; set; }
       public DateTime StartTime { get; set; }
       public DateTime ReqRespStartTime { get; set; }
       public double ReqRespDuration { get; set; }

       // DNS
       public double DnsWait { get; set; }
       public DateTime DnsStartTime { get; set; }
       public double DnsDuration { get; set; }

       // HTTPS
       public double HandshakeWait { get; set; }
       public DateTime HandshakeStartTime { get; set; }
       public double HandshakeDuration { get; set; }

       // socket connection
       public DateTime SocketConnectionStartTime { get; set; }
       public double SocketWait { get; set; }
       public double SocketDuration { get; set; }

       public DateTime QueueuingEndTime { get; set; }
       public double QueueingDuration { get; set; }
   }

The second one inherits from the base class and contains addition details; including the details of the redirected url if any:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


private class HttpRequestInfo : HttpRequestInfoBase
   {
       public HttpRequestInfo(DateTime timestamp, string scheme, string host, uint port, string path)
           :
           base(timestamp, scheme, host, port, path)
       {
       }

       public HttpRequestInfoBase Redirect { get; set; }

       public UInt32 StatusCode { get; set; }

       // HTTPS
       public string HandshakeErrorMessage { get; set; }

       public string Error { get; set; }
   }

A new instance of HttpRequestInfoBase is created when a 301 status code is received in HttpResponseHeaderStop handler:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


private void OnHttpResponseHeaderStop(object sender, HttpRequestStatusEventArgs e)
    {
        // used to detect redirection in .NET 8+
        if (e.StatusCode != 301)
        {
            return;
        }

        // create a new request info for the redirected request
        // because .NET 7 does not emit a Redirect event, we need to create a new request info here
        // --> it means that the redirect url will be empty in .NET 7
        var root = GetRoot(e.ActivityId);
        if (_requests.TryGetValue(root, out HttpRequestInfo info))
        {
            info.Redirect = new HttpRequestInfoBase(e.Timestamp, "", "", 0, "");

            // if you really want to have the duration of both original request + redirected request,
            // then do the following:
            //    info.ReqRespDuration = (e.Timestamp - info.ReqRespStartTime).TotalMilliseconds;
            // However, I prefer to show the duration of the redirected request only to more easily
            // compute the cost of the initial redirected request = total duration - other durations
        }
    }

For .NET 8+, the redirected url is provided to the Redirect handler and stored in the Url field of the instance created in the previous handler:

1
2
3
4
5
6
7
8
9


private void OnHttpRedirect(object sender, HttpRedirectEventArgs e)
    {
        // since this is an Info event, the activityID is the root
        var root = ActivityHelpers.ActivityPathString(e.ActivityId);
        if (_requests.TryGetValue(root, out HttpRequestInfo info))
        {
            info.Redirect.Url = e.RedirectUrl;
        }
    }

In each handler of events that could be received for both initial and redirected requests,

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


private void OnHttpResponseContentStop(object sender, EventPipeBaseArgs e)
   {
       var root = GetRoot(e.ActivityId);
       if (!_requests.TryGetValue(root, out HttpRequestInfo info))
       {
           return;
       }

       if (info.Redirect == null)
       {
           info.ReqRespDuration = (e.Timestamp - info.ReqRespStartTime).TotalMilliseconds;
       }
       else
       {
           info.Redirect.ReqRespDuration = (e.Timestamp - info.Redirect.ReqRespStartTime).TotalMilliseconds;
       }
   }

The wait time and durations are now computed as the events are received and aggregated at the end of the request by adding the value of both parts (initial and redirected if any:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


double dnsDuration = info.DnsDuration + ((info.Redirect != null) ? info.Redirect.DnsDuration : 0);
    if (dnsDuration > 0)
    {
        double dnsWait = info.DnsWait + ((info.Redirect != null) ? info.Redirect.DnsWait : 0);
        Console.Write($"{dnsWait,9:F3} | {dnsDuration,9:F3} | ");
    }
    else
    {
        Console.Write($"          |           | ");
    }

As a conclusion, you should try to monitor these redirections by using my dotnet-http CLI tool. Feel free to download it or install it with the following command line: dotnet tool install -g dotnet-http or update to the latest version with: dotnet tool update -g dotnet-http

You could also integrate some event listening code into your framework that simply handles the ResponseHeadersStop/Redirect events.

Implementing dotnet-http to monitor your HTTP requests

Wed, 13 Nov 2024 13:18:37 +0000

The previous episode detailed how to find the events dealing with network requests that are emitted by the BCL classes with their undocumented payload. It is now time to see how to listen to them and extract valuable insights such as what is happening when an HTTP request is sent to a server as an example. This is how my new dotnet-http CLI tool is implemented.

You are now able to see the cost of DNS, socket connection, security and redirection as shown in the following screenshot.

As you can see, the cost of an unexpected redirection could be high and hides security handshakes. In this example, using https://github.com/Maoni0 instead of http://github.com/Maoni0 divides by 2 the request duration!

Once the DNS checks are done, the socket connections are established, and the security handshakes are validated, the corresponding phases are no more needed for the next requests.

Listen to custom EventSource events

As explained in previous posts, it is easy to listen to events emitted by .NET application thanks to the Microsoft TraceEvent nuget. You also need to use EventPipeClient from the Microsoft.Diagnostics.NETCore.Client nuget to connect to the running application. In my case, I’m relying on an older version that supports both ETW and EventPipe:

1
2
3
4
5
6
7


var configuration = new SessionConfiguration(
    circularBufferSizeMB: 2000,
    format: EventPipeSerializationFormat.NetTrace,
    providers: GetProviders()
);
var binaryReader = EventPipeClient.CollectTracing(_processId, configuration, out var sessionId);
EventPipeEventSource source = new EventPipeEventSource(binaryReader);

The configuration contains a list of providers that are emitting the events you are interested in with the right keyword and verbosity. Here is what is needed for the HTTP requests monitoring :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


private static IReadOnlyCollection<Provider> GetProviders()
{
    var providers = new Provider[]
    {
        new Provider(
            name: "System.Net.Http",
            keywords: (ulong)(1),
            eventLevel: EventLevel.Verbose),

        new Provider(
            name: "System.Net.Sockets",
            keywords: (ulong)(0xFFFFFFFF),
            eventLevel: EventLevel.Verbose),

        new Provider(
            name: "System.Net.NameResolution",
            keywords: (ulong)(0xFFFFFFFF),
            eventLevel: EventLevel.Verbose),

        new Provider(
            name: "System.Net.Security",
            keywords: (ulong)(0xFFFFFFFF),
            eventLevel: EventLevel.Verbose),

        new Provider(
            name: "System.Threading.Tasks.TplEventSource",
            keywords: (ulong)(0x80),
            eventLevel: EventLevel.Verbose),

    };

    return providers;
}

You might be surprised by the presence of the TplEventSource provider but it is required to get a correct ActivityID. This major subject is detailed later.

There are so many events to listen to that it is easier to listen to the source AllEvents C# event:

1

source.AllEvents += OnEvents;

This global handler passes the events to ParseEvent that forwards the important data to handlers for each provider:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41


private static Guid NetSecurityEventSourceProviderGuid = Guid.Parse("7beee6b1-e3fa-5ddb-34be-1404ad0e2520");
private static Guid DnsEventSourceProviderGuid = Guid.Parse("4b326142-bfb5-5ed3-8585-7714181d14b0");
private static Guid SocketEventSourceProviderGuid = Guid.Parse("d5b2e7d4-b6ec-50ae-7cde-af89427ad21f");
private static Guid HttpEventSourceProviderGuid = Guid.Parse("d30b5633-7ef1-5485-b4e0-94979b102068");

private void ParseEvent(
    DateTime timestamp,
    int threadId,
    Guid activityId,
    Guid relatedActivityId,
    Guid providerGuid,
    string taskName,
    Int64 keywords,
    UInt16 id,
    byte[] eventData
    )
{
    if (providerGuid == NetSecurityEventSourceProviderGuid)
    {
        HandleNetSecurityEvent(timestamp, threadId, activityId, relatedActivityId, id, taskName, eventData);
    }
    else
    if (providerGuid == DnsEventSourceProviderGuid)
    {
        HandleDnsEvent(timestamp, threadId, activityId, relatedActivityId, id, taskName, eventData);
    }
    else
    if (providerGuid == SocketEventSourceProviderGuid)
    {
        HandleSocketEvent(timestamp, threadId, activityId, relatedActivityId, id, taskName, eventData);
    }
    else
    if (providerGuid == HttpEventSourceProviderGuid)
    {
        HandleHttpEvent(timestamp, threadId, activityId, relatedActivityId, id, taskName, eventData);
    }
    else
    {
        WriteLogLine();
    }
}

The next step is to handle each event based on its id:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


private void HandleNetSecurityEvent(
    DateTime timestamp,
    int threadId,
    Guid activityId,
    Guid relatedActivityId,
    ushort id,
    string taskName,
    byte[] eventData
    )
{
    switch (id)
    {
        case 1: // HandshakeStart
            OnHandshakeStart(timestamp, threadId, activityId, relatedActivityId, eventData);
            break;
        case 2: // HandshakeStop
            OnHandshakeStop(timestamp, threadId, activityId, relatedActivityId, eventData);
            break;
        case 3: // HandshakeFailed
            OnHandshakeFailed(timestamp, threadId, activityId, relatedActivityId, eventData);
            break;
        default:
            WriteLogLine();
            break;
    }
}

Extract information from an event payload

The payload of each event has been detailed in the previous post. For example, I need to extract the following fields from the RequestStart event payload:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


private void OnRequestStart(
    DateTime timestamp,
    int threadId,
    Guid activityId,
    Guid relatedActivityId,
    byte[] eventData
    )
{
    ...
    // string scheme
    // string host
    // int port
    // string path
    // byte versionMajor
    // byte versionMinor
    // enum HttpVersionPolicy

I’ve implemented the EventSourcePayload class that provides strongly typed helpers to get the different fields one after the other:

1
2
3
4
5
6
7
8
9


public class EventSourcePayload
{
    private byte[] _payload;
    private int _pos = 0;

    public EventSourcePayload(byte[] payload)
    {
        _payload = payload;
    }

It accepts the payload as the array of bytes received by each handler.

A string information is serialized as a list of UTF-16 Unicode characters; each one stored in 2 bytes:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


public string GetString()
    {
        StringBuilder builder = new StringBuilder();
        while (_pos < _payload.Length)
        {
            var characters = UnicodeEncoding.Unicode.GetString(_payload, _pos, 2);
            _pos += 2;

            if (characters == "\0")
            {
                break;
            }
            builder.Append(characters);
        }

        return builder.ToString();
    }

The current position in the array is incremented character by character up to the final ‘\0’.

The other helpers implementation is straightforward thanks to the BitConverter class:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40


public byte GetByte()
    {
        return _payload[_pos++];
    }

    public UInt16 GetUnit16()
    {
        UInt16 value = BitConverter.ToUInt16(_payload, _pos);
        _pos += sizeof(UInt16);
        return value;
    }

    public UInt32 GetUInt32()
    {
        UInt32 value = BitConverter.ToUInt32(_payload, _pos);
        _pos += sizeof(UInt32);
        return value;
    }

    public UInt64 GetUInt64()
    {
        UInt64 value = BitConverter.ToUInt64(_payload, _pos);
        _pos += sizeof(UInt64);
        return value;
    }

    public Int64 GetInt64()
    {
        Int64 value = BitConverter.ToInt64(_payload, _pos);
        _pos += sizeof(UInt64);
        return value;
    }

    public double GetDouble()
    {
        double value = BitConverter.ToDouble(_payload, _pos);
        _pos += sizeof(double);
        return value;
    }
}

Again, the position in the array is incremented to reflect the size of the field being read.

If you look at the rest of the OnRequestStart handler, you will see how each field is extracted:

1
2
3
4
5
6
7


EventSourcePayload payload = new EventSourcePayload(eventData);
    var scheme = payload.GetString();
    var host = payload.GetString();
    var port = payload.GetUInt32();
    var path = payload.GetString();
    var versionMajor = payload.GetByte();
    var versionMinor = payload.GetByte();

My activity or not my activity: that is the question

In the previous episode, I forgot (on purpose) to mention that the EventSource is keeping track of “activities” when emitting events:

1
2
3
4


protected unsafe void WriteEventCore(int eventId, int eventDataCount, EventData* data)
{
    WriteEventWithRelatedActivityIdCore(eventId, null, eventDataCount, data);
}

The WriteEventWithRelatedActivityIdCore method looks at the event metadata and if its opcode is Start then a new activity is created; if it is Stop then the current activity ends:

1
2
3
4
5
6
7
8


if (opcode == EventOpcode.Start)
{
    m_activityTracker.OnStart(m_name, metadata.Name, metadata.Descriptor.Task, ref activityId, ref relActivityId, metadata.ActivityOptions);
}
else if (opcode == EventOpcode.Stop)
{
    m_activityTracker.OnStop(m_name, metadata.Name, metadata.Descriptor.Task, ref activityId);
}

These OnStart and OnStop methods are doing nothing if the TplEventSource is not enabled with Keywords.TasksFlowActivityIds (= 0x80) set. This explains the code in GetProviders listed earlier where this non-HTTP provider is enabled.

When a request is created, the current global count managed by the ActivityTracker is incremented and it becomes the id of the current activity. Note that there is a “root” identifier before any activity gets created corresponding to the current AppDomain ID; starting at 1. If you think of an HTTP request, after a RequestStart event, each phase starts a new activity with, for example, ResolutionStart or ConnectStart. Informative events are emitted with the current request activity such as Redirect or ConnectionEstablished.

Here is a simplified view of the events emitted for an HTTP request (without DNS nor security events):

Thread +-- Path >------- ID ---- Opcode -- Event ----------------------------------
 78568 |    1/1 >  event  1 __ [ 1| Start] RequestStart
       |
 32388 |  1/1/1 >  event  1 __ [ 1| Start] ResolutionStart
       |
 32388 |  1/1/1 >  event  2 __ [ 2|  Stop] ResolutionStop
       |
 32388 |  1/1/2 >  event  1 __ [ 1| Start] ConnectStart
       |
 32388 |  1/1/2 >  event  2 __ [ 2|  Stop] ConnectStop
       |
 32388 |    1/1 >  event  4 __ [ 0|  Info] ConnectionEstablished
       |
 53324 |    1/1 >  event  6 __ [ 0|  Info] RequestLeftQueue
       |
 53324 |  1/1/3 >  event  7 __ [ 1| Start] RequestHeadersStart
       |
 53324 |  1/1/3 >  event  8 __ [ 2|  Stop] RequestHeadersStop
       |
 68024 |  1/1/4 >  event 11 __ [ 1| Start] ResponseHeadersStart
       |
 68024 |  1/1/4 >  event 12 __ [ 2|  Stop] ResponseHeadersStop
       |
 68024 |  1/1/5 >  event 13 __ [ 1| Start] ResponseContentStart
       |
 68024 |  1/1/5 >  event 14 __ [ 2|  Stop] ResponseContentStop
       |
 68024 |    1/1 >  event  2 __ [ 2|  Stop] RequestStop
  200 <|

As you can see, different threads are emitting events associated to the same request. At the ActivityTracker level, an ActivityInfo is stored in an async local so that each thread has its own storage that will be propagated by the methods of the Task Parallel Library (a.k.a. TPL) from task to task. This is why the very asynchronous code of the HTTP client implementation can go back to the current activity from different threads.

The activities are encoded into the 16 bytes of a GUID. In fact, only the first 12 bytes are used, and the final 4 bytes contain a checksum that includes the current process ID. Since the activity identifiers in the path do, most of the time, have a small value, the encoding is using 4 bits, also known as a nibble, to encode each of them. There was a bug in ActivityTracker.AddIdToGuid that will be fixed in .NET 10. Unfortunately, the decoding code in Perfview needs to take it into account so the last activity is not lost.

Computing the different durations

With this infrastructure in place, it is now possible to extract the “root” activity path corresponding to an HTTP request during each phase and update the corresponding state that is used to output the details when a request ends:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


private class HttpRequestInfo
{
    public HttpRequestInfo(DateTime timestamp, Guid activityId, string scheme, string host, uint port, string path)
    {
        Root = ActivityHelpers.ActivityPathString(activityId);
        if (port != 0)
        {
            Url = $"{scheme}://{host}:{port}{path}";
        }
        else
        {
            Url = $"{scheme}://{host}:{path}";
        }
        StartTime = timestamp;
    }

    public string Root { get; set; }
    public string Url { get; set; }
    public string RedirectUrl { get; set; }
    public DateTime StartTime { get; set; }
    public DateTime ReqRespStartTime { get; set; }
    public double ReqRespDuration { get; set; }

    public UInt32 StatusCode { get; set; }

    // DNS
    public DateTime DnsStartTime { get; set; }
    public double DnsDuration { get; set; }

    // HTTPS
    public DateTime HandshakeStartTime { get; set; }
    public double HandshakeDuration { get; set; }
    public string HandshakeErrorMessage { get; set; }

    // socket connection
    public DateTime SocketConnectionStartTime { get; set; }
    public double SocketDuration { get; set; }
}

As you have seen earlier, the different phases of a request may be processed by different threads. It means that an available thread needs to be found in order to execute them. If the thread pool is busy, you can expect some wait time. This is shown in the different wait sections between the phases. They are easily computed using the timestamp of each event. Having long wait durations might be reduced by increasing the number of threads in the thread pool.

Feel free to use the new dotnet-http CLI tool available from nuget.org or via dotnet tool install -g dotnet-http.

The corresponding sources are available from my github repository in case you would like to integrate this kind of analysis directly in your code or your monitoring pipeline.

Happy monitoring!

Digging into the undocumented .NET events from the BCL

Sun, 13 Oct 2024 12:12:37 +0000

I’ve presented in depth the events emitted by the CLR in many posts to get insightful details about how the .NET runtime is working (lock contention, GC, allocations, …). Some .NET features are not implemented at the runtime level but at the Base Class Library (a.k.a. BCL) level. For example, if you are using HttpClient, you might want to measure how long it takes to get the response to your HTTP requests.

In this new series, I will describe how the BCL is using EventSource to emit the events, how you can listen to them with TraceEvent; focusing on HTTP requests.

Where do these events come from?

When you look for these BCL events in the documentation, you end up to this well-known event provides in .NET page and in the Framework libraries section. It lists the providers with their name and the emitted events with their keyword and verbosity. However, there is no detail about their payload! For example, for the “System.Net.Http” provider, you know that the RequestStart informal event is emitted each time “an HTTP request has started”. But you don’t know the corresponding url… Let me tell you how to get these tiny details.

Within the BCL, the classes responsible for emitting events derive from EventSource with the “Telemetry” suffix as a naming convention. They are decorated with an EventSource attribute to define their name (used as provider name by a listener) and their Guid that will be provided with each event. For example, here is the declaration of the class responsible for events related to sending HTTP requests:

1
2
3


[EventSource(Name = "System.Net.Http")]
internal sealed partial class HttpTelemetry : EventSource
{

If a Guid is not provided, one is automatically computed (see the corresponding code in Perfview).

Next, public helper methods decorated with a NonEvent attribute are provided to be used in the BCL code when it is needed to emit events. These methods are calling private methods decorated with an Event attribute to define their unique ID and their verbosity level:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


[NonEvent]
public void RequestStart(HttpRequestMessage request)
{
    ...
    RequestStart(
        request.RequestUri.Scheme,
        request.RequestUri.IdnHost,
        request.RequestUri.Port,
        request.RequestUri.PathAndQuery,
        (byte)request.Version.Major,
        (byte)request.Version.Minor,
        request.VersionPolicy);
} 

[Event(1, Level = EventLevel.Informational)]
private void RequestStart(string scheme, string host, int port, string pathAndQuery, byte versionMajor, byte versionMinor, HttpVersionPolicy versionPolicy)
{
    ...
    WriteEvent(eventId: 1, scheme, host, port, pathAndQuery, versionMajor, versionMinor, versionPolicy);
}

The different WriteEvent overloads are responsible for filling an array of EventData elements (each one contains a pointer to the data and its size) that is passed to EventSource.WriteEventCore. This helper methods dispatches the event to EventPipe/ETW pipelines.

Document the undocumented

Unlike CLR events, their payload is not explicitly described in a text file but, instead, you have to look at the implementation of the different WriteEvent overloads. The rest of this section provides the details of the sources I’ve looked at and the corresponding events payload.

HTTP

Sources: https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.Http/src/System/Net/Http/HttpTelemetry.cs Name: System.Net.Http Guid: d30b5633–7ef1–5485-b4e0–94979b102068

Sockets

Sources: https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketsTelemetry.cs

Name: System.Net.Sockets Guid: d5b2e7d4-b6ec-50ae-7cde-af89427ad21f

DNS

Sources: https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.NameResolution/src/System/Net/NameResolutionTelemetry.cs

Name: System.Net.NameResolution Guid: 4b326142-bfb5–5ed3–8585–7714181d14b0

Network Security

Sources: https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.Security/src/System/Net/Security/NetSecurityTelemetry.cs

Name: System.Net.Security Guid: 7beee6b1-e3fa-5ddb-34be-1404ad0e2520

The next episode will describe how to listen to these events and extract their payload in C#.

Unexpected usage of EventSource or how to test statistical results in CLR pull request

Fri, 13 Sep 2024 14:25:30 +0000

Testing the statistical results

In parallel of the performance impact, it is important to validate the expected statistical distribution of the sampled allocations. Basically, I need to execute the same run of allocations multiple times in a row. Each run allocates the same number of instances of different types. For example, it is interesting to know if sampling instances of types with sizes proportional to a base value gives good results. Same question for totally different sized types or with Finalizers.

I would like to pass the number of runs to execute and a given scenario to a C# runner program and listen to the emitted events in another C# listener.

I’m facing 3 issues here:

How many instances are allocated to validate the upscaling algorithm (sampled vs real count)
What are the types I want to focus on because I don’t want to hard code them in the listener application.
When does each run start?

It would be great if I could send the answer to these questions via events so the listener would know at runtime. Well… This is exactly what a class inherited from EventSource allows you to do!

In the runner application, I’ve defined the AllocationsRunEventSource that is decorated with the EventSource attribute to set its name that will be used as a provider name like Microsoft-Windows-DotNETRuntime for the .NET runtime provider.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


[EventSource(Name = "Allocations-Run")]
public class AllocationsRunEventSource : EventSource
{
    public static readonly AllocationsRunEventSource Log = new AllocationsRunEventSource();

The four implemented methods are defining which event ID for which verbosity will be emitted with which payload:
    [Event(600, Level = EventLevel.Informational)]
    public void StartRun(int iterationsCount, int allocationCount, string listOfTypes)
    {
        WriteEvent(eventId: 600, iterationsCount, allocationCount, listOfTypes);
    }

    [Event(601, Level = EventLevel.Informational)]
    public void StopRun()
    {
        WriteEvent(eventId: 601);
    }

    [Event(602, Level = EventLevel.Informational)]
    public void StartIteration(int iteration)
    {
        WriteEvent(eventId: 602, iteration);
    }

    [Event(603, Level = EventLevel.Informational)]
    public void StopIteration(int iteration)
    {
        WriteEvent(eventId: 603, iteration);
    }
}

To make the payload serialization and parsing easy, the list of types that will be allocated is passed as a string with the following format allocatedTypes = “Object24;Object48;Object72;Object32;Object64;Object96”.

The code of the runner calls these methods as expected at different moment of the execution:

1
2
3
4
5
6
7
8


AllocationsRunEventSource.Log.StartRun(iterations, allocationsCount, allocatedTypes);
 for (int i = 0; i < iterations; i++)
 {
     AllocationsRunEventSource.Log.StartIteration(i);
     allocationsRun.Allocate(allocationsCount);
     AllocationsRunEventSource.Log.StopIteration(i);
 }
 AllocationsRunEventSource.Log.StopRun();

Instead of recording the events with dotnet-trace, this time I’m using TraceEvent and Microsoft.Diagnostics.NETCore.Client to code a listener application. The code is very similar to what was presented for my dotnet-fullgc CLI tool except that I’m enabling the AllocationsRun provider corresponding to the event source of the runner in addition to the .NET runtime one:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


public static void PrintEventsLive(int processId)
{
    var providers = new List<EventPipeProvider>()
    {
        new EventPipeProvider(
                "Microsoft-Windows-DotNETRuntime",
                EventLevel.Verbose, // verbose is required for AllocationTick
                (long)0x80000000001 // new AllocationSamplingKeyword + GCKeyword
                ),
        new EventPipeProvider(
                "Allocations-Run",
                EventLevel.Informational
                ),
    };

The custom events from that provider are received via the source.Dynamic.All C# event:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


var client = new DiagnosticsClient(processId);
    using (var session = client.StartEventPipeSession(providers, false))
    {
        Console.WriteLine();

        Task streamTask = Task.Run(() =>
        {
            var source = new EventPipeEventSource(session.EventStream);
            _source = source;
            ClrTraceEventParser clrParser = new ClrTraceEventParser(source);
            clrParser.GCAllocationTick += OnAllocationTick;
            source.Dynamic.All += OnEvents;
            ...

Because TraceEvent is not already aware of the new AllocationSampled event emitted by the PR code, it will also be received via the same OnEvent handler:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


private static void OnEvents(TraceEvent eventData)
{
    if (eventData.ID == (TraceEventID)303)
    {
        // AllocationSampled parsing 
        ...
        return;
    }

    if (eventData.ID == (TraceEventID)600)  // Start run
    {
        // keep track of the expected types and the number of allocated instances 
        ...
        return;
    }

    if (eventData.ID == (TraceEventID)601)  // Stop run
    {
        // show the results of the run
        ...
        return;
    }

    if (eventData.ID == (TraceEventID)602)  // Start an iteration in a run
    {
        // reset for a new iteration
        ...
        return;
    }

    if (eventData.ID == (TraceEventID)603)  // Stop an iteration in a run
    {
        // Show iteration results
        ...
        return;
    }
}

The parsing of the payload of the run related events is done the same way as for AllocationSampled by a dedicated xxxData class:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


class AllocationsRunData
{
    const int EndOfStringCharLength = 2;
    private TraceEvent _payload;

    public AllocationsRunData(TraceEvent payload)
    {
        _payload = payload;

        ComputeFields();
    }

    public int Iterations;
    public int Count;
    public string AllocatedTypes;

    private void ComputeFields()
    {
        int offsetBeforeString = 4 + 4;

        Span<byte> data = _payload.EventData().AsSpan();
        Iterations = BitConverter.ToInt32(data.Slice(0, 4));
        Count = BitConverter.ToInt32(data.Slice(4, 4));
        AllocatedTypes = Encoding.Unicode.GetString(data.Slice(offsetBeforeString, _payload.EventDataLength - offsetBeforeString - EndOfStringCharLength));
    }
}

By keeping track of this data, it is possible to show each iteration results:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


> starts 100 iterations allocating 1000000 instances
0|
Tag  SCount  TCount          SSize          TSize   UnitSize     UpscaledSize  UpscaledCount  Name
--------------------------------------------------------------------------------------------------
 ST     247     384           5928           9216         24         24702711        1029279  Object24
 ST     322     106          10304           3392         32         32205122        1006410  Object32
 ST     435     509          20880          24432         48         43510266         906463  Object48
 ST     587     776          37568          49664         64         58718825         917481  Object64
 ST     747     481          53784          34632         72         74726662        1037870  Object72
 ST     958     916          91968          87936         96         95845392         998389  Object96

that integrates AllocationTick numbers too.

At the end of the run, error distribution per type is also computed over the iterations:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


Object72
-------------------------
   1  -10.5 %
   2   -8.9 %
   3   -8.4 %
   4   -7.8 %
   5   -6.4 %
        ...
  49    0.2 %
  50    0.2 %
  51    0.2 %
        ...
  96    6.8 %
  97    6.8 %
  98    7.7 %
  99    8.6 %
 100   10.0 %

You could use the same mechanisms (a custom EventSource to emit additional information and an EventPipe listener to aggregate the data) for your own usage. This is a different way to use EventSource rather than emitting events for monitoring like what is done by the BCL.

Testing the standalone GC

In addition to the usual .NET GC, it is needed to validate that the changes are also working for the standalone GC. Long story short, it is possible to replace the existing .NET garbage collector by your own implementation. For the test, I needed to check that the standalone GC clrgcexp.dll generated by the .NET compilation generates the expected AllocationSampled events when the corresponding keyword with informational verbosity is enabled.

Debugging NativeAOT scenarios

The final step was to implement the feature for the NativeAOT scenario. When you build your C# application for NativeAOT, a lot happens behind the scenes, based on compilers known by Visual Studio corresponding to the official released version of the .NET runtime. In my case, I needed to use the brand-new code of my local branch and debug some simple C# applications. The steps to reach that goal are not that simple.

First, you follow the steps given by the documentation to build AOT CLR in debug and libs in release:

1
2


build clr.aot+lib -rc debug -lc release
build -c release

Then, open src\coreclr\tools\aot\ilc.sln

the repro project contains a program.cs file where you write the C# code you want to test and debug.

When you build a NativeAOT application, you need to select if you want the runtime that emits events or not. This is done by setting the EventSourceSupport to true in the .csproj:

Add the following in the .csproj to get events:

1

true

However, with the repro project, you need to change ILCompiler.csproj in a different way:

1

 Include="--feature:System.Diagnostics.Tracing.EventSource.IsSupported=true" />

Also, change the reproNative.vcxproj file to bind to eventpipe-enabled.lib instead of eventpipe-disabled.lib for the platform/configuration you want to debug:

1
2
3
4
5
6


 Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
    ...
    
      ...
      ...$(ArtifactsRoot)bin\coreclr\windows.x64.Debug\aotsdk\eventpipe-enabled.lib;...
    

Then, build repro in Debug x64

Next, change the target for the ILCompiler project:

Build and run it to generate the .obj file corresponding to the repro project

Finally, open src\coreclr\tools\aot\ILCompiler\reproNative\reproNative.vcxproj that will allow you to debug the program.cs you’ve just built!

Tips and tricks from validating a Pull Request in .NET CLR

Tue, 13 Aug 2024 09:23:28 +0000

Introduction

During the implementation of our .NET allocation profiler, we realized that the current sampling mechanism based on a fixed threshold did not provide a good enough statistical distribution. With the help of Noah Falk from the CLR Diagnostics team, I started to implement a randomized sampling based on a Bernoulli distribution model for .NET.

With this kind of changes, you need to ensure that you don’t break any existing code, the impact on performance is limited and the mathematical results map the expected mathematical distribution.

The rest of this blog series details the different tests I wrote and the corresponding tips and tricks that could be reused when you write C# code.

Testing the basics

From a high-level view, the code change does something simple: each time an allocation context is needed to fulfill an allocation, the code checks if it should be sampled. In that case, a new AllocationSampled event is emitted with the same information as the existing AllocationTick event plus an additional field. So, the first level of testing is to validate that the events are emitted when the keyword and verbosity are enabled for the .NET runtime provider.

The runtime has already some tests in place to validate that some events are emitted under the** \src\tests\tracing\eventpipe** folder. Here is the code of my XUnit test that mimics the existing ones such as simpleruntimeeventvalidation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


[Fact]
public static int TestEntryPoint()
{
    // check that AllocationSampled events are generated and size + type name are correct
    var ret = IpcTraceTest.RunAndValidateEventCounts(
        new Dictionary<string, ExpectedEventCount>() { { "Microsoft-Windows-DotNETRuntime", -1 } },
        _eventGeneratingActionForAllocations,
        // AllocationSamplingKeyword (0x80000000000): 0b1000_0000_0000_0000_0000_0000_0000_0000_0000_0000_0000
        new List<EventPipeProvider>() { new EventPipeProvider("Microsoft-Windows-DotNETRuntime", EventLevel.Informational, 0x80000000000) },
        1024, _DoesTraceContainEnoughAllocationSampledEvents, enableRundownProvider: false);
    if (ret != 100)
        return ret;

    return 100;
}

The IpcTraceTest.RunAndValidateEventCounts helper method accepts:

The list of providers to enable with which keyword and verbosity level.
How many events are expected (using -1 in my case because I can’t predict how many random events will be generated
A callback with the code that will generate events (allocating a lot of instances of a custom type in my case)
A callback that looks at emitted events

The last callback code relies on TraceEvent to listen to emitted events:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


private static Func<EventPipeEventSource, Func<int>> _DoesTraceContainEnoughAllocationSampledEvents = (source) =>
{
    int AllocationSampledEvents = 0;
    int Object128Count = 0;
    source.Dynamic.All += (eventData) =>
    {
        if (eventData.ID == (TraceEventID)303)  // AllocationSampled is not defined in TraceEvent yet
        {
            AllocationSampledEvents++;

            AllocationSampledData payload = new AllocationSampledData(eventData, source.PointerSize);
            // uncomment to see the allocation events payload
            // Logger.logger.Log($"{payload.HeapIndex} - {payload.AllocationKind} | ({payload.ObjectSize}) {payload.TypeName}  = 0x{payload.Address}");
            if (payload.TypeName == "Tracing.Tests.SimpleRuntimeEventValidation.Object128")
            {
                Object128Count++;
            }
        }
    };
    return () => {
        Logger.logger.Log("AllocationSampled counts validation");
        Logger.logger.Log("Nb events: " + AllocationSampledEvents);
        Logger.logger.Log("Nb object128: " + Object128Count);
        return (AllocationSampledEvents >= MinExpectedEvents) && (Object128Count != 0) ? 100 : -1;
    };
};

In my case, I’m adding a new event that is emitted when a new keyword is enabled. It means that TraceEvent does not know yet its ID (hence the 303 hardcoded value) or how to unpack the new event payload. This is why I created the AllocationSampleData type to expose the payload as public fields:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40


class AllocationSampledData
{
    const int EndOfStringCharLength = 2;
    private TraceEvent _payload;
    private int _pointerSize;
    public AllocationSampledData(TraceEvent payload, int pointerSize)
    {
        _payload = payload;
        _pointerSize = pointerSize;
        TypeName = "?";

        ComputeFields();
    }

    public GCAllocationKind AllocationKind;
    public int ClrInstanceID;
    public UInt64 TypeID;
    public string TypeName;
    public int HeapIndex;
    public UInt64 Address;
    public long ObjectSize;
    public long SampledByteOffset;

    ...
}
And the extraction of each field from the payload is done in the ComputeFields method:
// The payload of AllocationSampled is not defined in TraceEvent yet
//
//  
//  
//  
//  
//  
//  
//  
//  
//
    private void ComputeFields()
    {
        int offsetBeforeString = 4 + 2 + _pointerSize;

This offsetBeforeString value is computed based on the size of UInt32 (=4 bytes), UInt16 (=2 bytes) and a Pointer (depends on 32 bit=4 or 64 bit=8) fields before the string. As Span wraps the binary payload provided by TraceEvent:

1

Span<byte> data = _payload.EventData().AsSpan();

Since I know the size of each field from the payload definition in ClrEtwAll.man, the numeric fields are extracted thanks to the BitConverter methods:

1
2


AllocationKind = (GCAllocationKind)BitConverter.ToInt32(data.Slice(0, 4));
        ClrInstanceID = BitConverter.ToInt16(data.Slice(4, 2));

Things start to be more complicated when you need to get the value of an address. Its size is 4 bytes in 32 bit and 8 bytes in 64 bit:

1
2
3
4
5
6
7
8


if (_pointerSize == 4)
        {
            TypeID = BitConverter.ToUInt32(data.Slice(6, _pointerSize));
        }
        else
        {
            TypeID = BitConverter.ToUInt64(data.Slice(6, _pointerSize));
        }

The bitness of the monitored application is given by the EventPipeSource’s PointerSize property that is passed to the AllocationSampledData constructor.

For the string case, you need to know that it is stored as UTF16 (so each character requires 2 bytes) with the trailing \0 and its length is the total size of the payload minus the size of the other fields. That way, you can slice the Span to properly read the characters:

1
2


//   \0 should not be included for GetString to work
        TypeName = Encoding.Unicode.GetString(data.Slice(offsetBeforeString, _payload.EventDataLength - offsetBeforeString - EndOfStringCharLength - 4 - _pointerSize - 8));

The rest of the fields are extracted with BitConverter helpers taking into account the size of the string:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


HeapIndex = BitConverter.ToInt32(data.Slice(offsetBeforeString + TypeName.Length * 2 + EndOfStringCharLength, 4));
        if (_pointerSize == 4)
        {
            Address = BitConverter.ToUInt32(data.Slice(offsetBeforeString + TypeName.Length * 2 + EndOfStringCharLength + 4, _pointerSize));
        }
        else
        {
            Address = BitConverter.ToUInt64(data.Slice(offsetBeforeString + TypeName.Length * 2 + EndOfStringCharLength + 4, _pointerSize));
        }        ObjectSize = BitConverter.ToInt64(data.Slice(offsetBeforeString + TypeName.Length * 2 + EndOfStringCharLength + 4 + 8, 8));
    }

The name of the sampled allocated type from the parsed payload is used to ensure that the expected allocations are indeed emitted when the keyword/verbosity are enabled for the .NET provider.

Testing the performance impact

The next step was to validate the impact of the changes on the GC performance. The baseline was the .NET 9 branch before the changes and in Release. The GCPerfSim library from the performance repository was used to allocate 500 GB of mixed size objects on 4 threads with a 50MB live object size. From the output, the seconds_taken line provides the duration to allocate these objects.

To ensure that you run with the rebuilt branch, you need to use the following commands:

1
2


build.cmd clr+libs -c release
src\tests\build.cmd generatelayoutonly Release

The next step is to use \artifacts\tests\coreclr\windows.x64.Release\Tests\Core_Root\corerun.exe instead of the usual dotnet.exe like the following:

1

\artifacts\tests\coreclr\windows.x64.Release\Tests\Core_Root\corerun \artifacts\bin\GCPerfSim\Release\net7.0\GCPerfSim.dll -tc 4 -tagb 500 -tlgb 0.05 -lohar 0 -sohsi 0 -lohsi 0 -pohsi 0 -sohpi 0 -lohpi 0 -sohfi 0 -lohfi 0 -pohfi 0 -allocType reference -testKind time

I run this scenario 10 times to compute the median and the average. I’m doing the same for the PR branch. So far so good. Now, how to do the same but to measure the impact of the random sampling? Remember that the code only triggers if the .NET provider is enabled with a certain keyword and verbosity. It means that you have to use a tool such as dotnet-trace to start an event pipe session but you would need the process id. I could have changed the code of GCPerfSim to show the process id but I would still need to wait for the session to have been created before starting the seconds_taken computation. Not really easy to script a 10x runs that way…

Don’t worry! dotnet-trace supports the — show-child-io true arguments that makes it start the session as the process starts and — providers allows you to enable a provider the way you want. Here is an example of the command line used for the performance runs:

1

dotnet-trace collect --show-child-io true --providers Microsoft-Windows-DotNETRuntime:0x80000000000:4 -- corerun \artifacts\bin\GCPerfSim\Release\net7.0\GCPerfSim.dll …

These dotnet-trace features are very handy for any scripting scenario unrelated to testing the CLR. For example, you could use Perfview to later on analyze how an application behaves thanks to the emitted events stored in the generated .nettrace file!

The next episode will describe unexpected usage of EventSource and debugging NativeAOT scenario.

Trigger your GCs with dotnet-fullgc!

Wed, 22 May 2024 18:03:17 +0000

Introduction

If you have read Microsoft documentation, you probably know that it is not recommended to trigger a garbage collection in your application code. However, in some troubleshooting cases, you might want to trigger a GC. For example, you don’t want to wait for a full gen2 compacting GC to figure out if your application is really leaking memory. For web applications, you can imagine having a hidden HTTP end point that simply call GC.Collect. What if you could simply call a command line tool to trigger a GC in any .NET application? This is exactly what my new dotnet-fullgc CLI tool is doing!

This Is The Way

If you have read a few of my past posts about the .NET diagnostics mechanisms, you know that you can send commands to another .NET process via EventPipes. Well… there is no explicit command to trigger a GC.

I have also explained how you could listen to events emitted by the CLR by enabling a provider with a set of keywords with a verbosity corresponding to the events you are interested in. This is how dotnet-trace and Perfview are collecting these events. If you want to trigger a GC, you simply need to enable the Microsoft-Windows-DotNETRuntime provider with the GCHeapCollect (= 0x800000L) keyword and an informal verbosity. Yes: it is as simple as that, and it is also working for .NET Framework!

So, you could trigger a GC with dotnet-trace via the following command line:

1

dotnet trace collect -p  - providers Microsoft-Windows-DotNETRuntime:0x800000:4 - duration 00:00:01

However, a .nettrace file would be generated and it is not possible to pass parameters (more on this later).

The rest of this post shows the C# code to obtain the same result. With Microsoft.Diagnostics.NETCore.Client and TraceEvent, you create a DiagnosticsClient with the ID of the process you are interested in:

1

var client = new DiagnosticsClient(processId);

The next step is to start an EventPipe session with the right provider, keyword and verbosity:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


var providers = new List<EventPipeProvider>()
{
    new EventPipeProvider(
        "Microsoft-Windows-DotNETRuntime",
        EventLevel.Informational,
        (long)ClrTraceEventParser.Keywords.GCHeapCollect,
        Arguments  // more on this later
        ),
};
using (var session = client.StartEventPipeSession(providers, false))
{

The source must be processed in another thread to avoid blocking the main thread:

1
2
3
4
5
6
7


Task streamTask = Task.Run(() =>
    {
        // without source to process, session.Stop() will not return
        var source = new EventPipeEventSource(session.EventStream);

        source.Process();
    });

The question to answer is how to stop the session: just create another task that waits for a second before stopping the session to exit from the Process call:

1
2
3
4
5
6
7
8


Task inputTask = Task.Run(() =>
    {
        Thread.Sleep(1000);
        session.Stop();
    });

    Task.WaitAny(streamTask, inputTask);
}

That’s all!

Pitfalls

Unfortunately, I faced a couple of issue during the implementation of the tool.

What is your number?

If you look at the TraceEvent implementation, you could imagine that is it possible to pass a GC ID as a parameter to the .NET provider:

1
2
3
4
5


//
// Summary:
//     Triggers a GC. Can pass a 64 bit value that will be logged with the GC Start
//     event so you know which GC you actually triggered.
GCHeapCollect = 0x800000L,

Unfortunately, this is not supported and the reasons are explained below.

If you check the .NET source code and look at EtwCallbackCommon(), you can indeed see that a numeric ID can be passed to ETW::GCLog::ForceGC(l64ClientSequenceNumber) and passed as ID in GCStart event instead of the one incrementally increased collection after collection.

At the diagnostics client level, you have the opportunity to pass a dictionary of key/value string pairs. This dictionary is used when defining the EventPipeProvider to be enabled:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


Dictionary<string, string> arguments = new Dictionary<string, string>();
arguments.Add("Id", "42");
var providers = new List<EventPipeProvider>()
{
    new EventPipeProvider(
        "Microsoft-Windows-DotNETRuntime",
        EventLevel.Informational,
        (long)ClrTraceEventParser.Keywords.GCHeapCollect,
        arguments
        ),
};

The diagnostics client transforms the dictionary into a string with key=value pairs separated by ‘;’ such as “Id=42;AnotherId=AnotherValue;…” and serializes it as is in the payload when sending a CollectTraces command.

The question is what identifier is expected by the CLR? To answer this question, you need to look at the code of provider_invoke_callback. The “key=value;…” string is stored into a buffer and each = and ; characters are transformed into \0. So, “Id=42” is transformed into Id\042\0.

The next step is done by ep_event_filter_desc_init() that uses that buffer to fill up the 3 fields of an EventFilterDescriptor:

1
2
3


uint64_t ptr = address of the buffer
uint32_t size = size of the buffer (=6 for id\042\0)
uint32_t type = 0

And finally, EtwCallbackCommon receives the filterdata and tries the following to get the collection id:

1
2
3
4
5
6
7


PEVENT_FILTER_DESCRIPTOR FilterData = (PEVENT_FILTER_DESCRIPTOR)pFilterData;
if ((FilterData != NULL) &&
   (FilterData->Type == 1) &&
   (FilterData->Size == sizeof(l64ClientSequenceNumber)))
{
    l64ClientSequenceNumber = *(LONGLONG *) (FilterData->Ptr);
}

As you can see, it will fail because:

the received type value is 0
the received size is not the size of a 64 bit number
the ptr field does not point to the value as 64 bit number (but as a string)

An issue has been filed with a possible fix.

Only one collection please!

When I test my dotnet-fullgc with dotnet-gcstats, I always see 2 collections!

We investigated with my colleague Kevin Gosse and he created an issue for that. The EtwCallback() function is called whenever a session is enabled or disabled. Unfortunately, the call to ForceGC is made in both cases: so, when the session is stopped, a second garbage collection is triggered.

Next steps

Feel free to install dotnet-fullgc on your machine with dotnet tool install -g dotnet-fullgc.

Next, use dotnet fullgc to trigger two gen2 full garbage collections in your running .NET processes.

Be Aligned! Or how to investigate a stack corruption

Mon, 11 Dec 2023 11:01:45 +0000

Introduction

During the Datadog R&D week, my goal is to mimic the generation of a .gcdump from our .NET profiler. I’ve already written most of the code for a previous post and after changing the required plumbing, it is time to test the workflow.

Unfortunately, I’m facing the dreaded stack corruption dialog:

The rest of the post explains the different steps I’m following to investigate this issue.

Trying to understand the problem

This stack check is done by the debug version of the C Runtime library by basically adding some special bytes on the stack before calling a function and checking these bytes are not tampered when returning from the call.

So, the next step is to debug the application to get more details and at least where in the code the problem happened:

The failed check occurs at the end of a function that looks like the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


bool GcDumpProvider::Get(IGcDumpProvider::gcdump_t& gcDump)
{
    // trigger the GC and get the dump
    GcDump gcd(::GetCurrentProcessId());
    gcd.TriggerDump();

    auto const& dump = gcd.GetGcDumpState();
    auto& types = dump._types;
    for (auto& type : types)
    {
        auto& typeInfo = type.second;

        uint64_t instancesCount = typeInfo._instances.size();
        uint64_t instancesSize = 0;
        for (size_t i = 0; i < instancesCount; i++)
        {
            instancesSize += typeInfo._instances[i]._size;
        }

        gcDump.push_back({typeInfo._name, instancesCount, instancesSize});
    }
    return true;
}

There is much more code behind this; especially in the TriggerDump() function. I have already tested this code many times when I dug into the .gcdump generation process without facing this stack corruption. I’m spending a few hours digging back into the code because:

I’m not running inside the profiled process and not outside like in the blog post
I’m introducing a “slight” change because I need to exit the communication with the CLR when the GC ends.
I need to mention that Visual Studio is refusing to debug (Step Over or Step Into) and only Run to Cursor was possible due to mixed mode (managed and native) debugging. So, I created a simple native console application with my updated code for easier and faster debugging.

After a couple of hours, it is time to go back to the Get() implementation because I do not find anything obviously wrong.

Make it simpler and simpler and simpler again

In that type of situation, I recommend the “remove code and debug” strategy (from “divide and conquer” attributed to Julius Cesar). From the simplified console application, the code now looks like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


bool GetGcDump(int pid, IGcDumpProvider::gcdump_t& gcDump)
{
    GcDump gcd(pid);

    // no more gcd.TriggerDump()

    // no more for (auto& type : types)

    return true;
}

No more complicated call nor iteration on the vector of results. Guess what? Same stack corruption.

It is time to go one level deeper: what does this GcDump class look like?

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


class GcDump
{
public:
    GcDump(int pid);
    ~GcDump();

...

private:
    int _pid;
    DiagnosticsClient* _pClient;
    EventPipeSession* _pSession;
    HANDLE _hListenerThread;
    GcDumpState _gcDumpState;
};

The constructor is setting the fields value to zero/nullptr and the destructor is cleaning up these fields if necessary. Since TriggerDump() is no more called, these fields never change.

I’m commenting out all fields until only _gcDumpState remains and it continues to crash. When is it commented out, it is not more crashing.

Use your debugger Luke!

Let’s turn to the GcDumpState class that is even simpler:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


class GcDumpState
{
public:
    GcDumpState();
    ~GcDumpState();

public:
    // fields removed for brevity

private:
    bool _isStarted;
    bool _hasEnded;
    uint32_t _collectionIndex;
};

The code of the destructor only sends a trace to the console (removing it completely does not fix the issue) and here is the constructor code:

1
2
3
4
5
6


GcDumpState::GcDumpState()
{
    _isStarted = false;
    _hasEnded = false;
    _collectionIndex = 0;
}

Again, same strategy: remove one field after the other. This time, the code stops crashing if the two Boolean fields are removed or if the 32 bits index is removed. If the index field is not set, no more corruption!

How could this assignation corrupt the stack? It is time to use the Visual Studio debugger to better understand what is going on.

First, set a breakpoint on the assignment line and click Debug | Disassembly to see the corresponding assembly code:

The two lines of assembly code are easy to understand:

1
2


mov rax,qword ptr [this]  
mov dword ptr [rax+4],0

The this pointer is stored in the rax register
The 32 bits (mov dword) memory starting 4 bytes after the beginning of the object pointed to by this, is set to 0

I enter “this” in a Memory panel (Debug | Windows | Memory xx)

And Visual Studio gives me the corresponding address and the content of the memory there:

Pressing F10 twice to Step Over the two assembly instructions and this is confirmed:

Instead of storing the 32 bits 0 value just after the two bytes corresponding to the bool fields, it is stored 2 bytes away. It looks like a padding is added on my behalf.

I change the build settings for the GcDumpState.cpp file to enable all warnings:

The compilation confirms what has been seen in the memory:

1

GcDumpState.h(41,14): warning C4820: 'GcDumpState': '2' bytes padding added after data member 'GcDumpState::_hasEnded'

What’s next?

My understanding is that the compiler is:

adding a 2 bytes padding to align the 32 bits index field
generating the constructor code based on that padding
but the stack corruption checking code does not take it into account

The solution is to either add a uint16_t field after the 2 bool fields as an explicit padding or use #pragma pack(1) to decorate the class definition.

However, this looks really weird to me. We should have faced this issue a long time ago because we were never cautious about alignment in all the classes and structures that we allocate in our code. To validate the assumption, I’m writing a small reproduction code outside of all the .gcdump complexity. And guess what? I’m not able to reproduce the stack corruption. Another mystery of the C++ compilation optimizations probably…

This is the end of my debugging Friday at Datadog :^)

How to dig into the CLR

Sun, 12 Nov 2023 09:12:28 +0000

Introduction

When I started to work on the second edition of Pro .NET Memory Management : For Better Code, Performance, and Scalability by Konrad Kokosa, I already spent some time in the CLR code for a couple of pull requests related to the garbage collector. However, updating the book to cover 5 new versions of .NET requires looking at new APIs but also digging deep inside the CLR (and especially the GC) hundreds of thousand lines of code!

The first step is to install Visual Studio 2022 Preview that allows you to compile and run projects targeting .NET 8. Then, goto https://github.com/dotnet/runtime and git clone the tag of the .NET 8 preview version you have installed.

That way, you will be able to directly run the same version that you will debug.

And now, what are the next steps?

The goal of this post is to share with you the tips and tricks I used to navigate into the CLR implementation so you could better understand how things are working.

From C# to C++

As a .NET developer, I’m used to the APIs provided by the Base Class Library built on top of the CLR. Let’s take as an example the following code that is using the GC.AllocateArray method that allows you to allocate a pinned in memory array and available since .NET 5.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


using System;

internal class Program
{
    static void Main(string[] args)
    {
        byte[] pinned = GC.AllocateArray<byte>(90000, true);
        Console.WriteLine($"generation = {GC.GetGeneration(pinned)}");
    }
}

When you Ctrl+click the method name (or use F12), thanks to Source Link integration, you go to its implementation where you can even set breakpoint:

If you don’t use Visual Studio, you could open the generated assembly into a decompiler such as ILSpy or DnSpy. The latter even allows you to set breakpoints and debug the disassembly IL without any source.

In both cases, only the managed implementation will be available: you soon end up to an “internal call” corresponding to a native function implemented by the CLR. The managed methods are decorated with the MethodImplOptions.InternalCall attribute.

For the garbage collector code, you can look into the GC.CoreCLR.cs file where these methods are defined. You can note some methods decorated with the DllImport attribute to bind to native functions exported by a “QCall” library. There is an optimized path in P/Invoke done by the JIT to transform these calls not like a usual LoadLibrary/GetProcAddress as you could expect. Instead, they will be routed to the exported methods by coredll.dll and defined in the s_QCall array in qcallentrypoints.cpp. But where to look further for the native implementation?

Instead of searching among the thousands of files, focus on comutilnative.h that defines the signature of most exported functions. The implementation of the exported native functions is found in comutilnative.cpp. This is where you should start your journey in the native implementation of the CLR. For the list of all functions called by the libraries in the runtime, look at the ecalllist.h file (around gGCInterfaceFuncs and gGCSettingsFuncs specifically for the GC).

Note that you might also find some implementations under the classlibnative folder like in the system.cpp file for GCSettings.IsServerGC.

CLR Source code debugging

It is nice to know that the implementation of most CLR exported native functions used by the BCL is in comutilnative.cpp. For the GC, the functions are either statics from the GCInterface class or static functions prefixed by GCInterface_; I don’t know why all are not part of GCInterface…

When you look at the GC-related methods implementation, a lot are calling methods from the instance returned by GCHeapUtilities::GetGCHeap() that corresponds to the static g_pGCHeap global variable. It is interesting to follow the threads of calls like that, but I have to admit that, after a few hops, I’m starting to get lost. So, I’m drawing boxes for types on a piece of paper and arrays from their fields to other types as boxes.

However, with a code base that big, I definitively prefer to set breakpoints and write a small C# application to call the methods I’m interested in and see what data structures are used in the different layers of implementation. Don’t be scared: WinDBG is not required to achieve this goal. As this page explains, you need to type the following commands in a shell at the root of the repo:

.\build.cmd -s clr -c Debug .\build.cmd clr.nativeprereqs -a x64 -c debug .\build.cmd -msbuild

The last command generates a CoreCLR.sln solution file in artifacts\obj\coreclr\windows.x64.Debug\ide) that you can open in Visual Studio 2022 Preview.

In VS, right-click the INSTALL project, select Properties and setup the Debugging properties

Here are the details of each property:

It could be interesting to set some environment variables such as DOTNET_gcServer to 1 for a GC Server configuration instead of workstation. In that case, click the choice in the combo-box:

And update the textbox at the top:

The final step is to set this project as the startup project:

You are now able to set the breakpoint you want in the native code of the CLR and type F5/Debug in Visual Studio to step into the code!

And what about the assembly code?

Some specific data structures, such as the NonGC Heap, are used by the JIT compiler when generating the assembly code from the IL compiled from your C# code. It means that you need to look at that JITted code to fully understand what is going on.

A first way to get it is to use https://sharplab.io/, type your C# code and select x64 for Core of x86/x64 for Framework:

But as you can see from this screenshot, it is using the .NET 7 compiler. What if you would like to see the .NET 8 compilation result just in case something changed?

The solution I’m using is to generate a memory dump with procdump -ma of a test application. Before opening the dump in WinDBG, there is something you should be aware of: with the tiered compilation, you will need to call a method several times before the final optimized assembly code gets JITed. Or… decorate the method you are interested in with the [MethodImpl(MethodImplOptions.AggressiveOptimization)] attribute to instruct the JIT to directly generate the most optimized tier.

Once the dump loaded in WinDBG, the first step is to get the MethodTable pointer corresponding to the method you are interested in. For that, use the name2ee SOS command:

Click the link corresponding to MethodDesc to run the dumpmd SOS command:

The last step is to click the link corresponding to CodeAddr to run the U command and see the JITted assembly code:

If you compare this code to get the “Hello, World!” string, with the one shown by sharplab,

1
2
3
4


Program.Hello()
    L0000: mov rcx, 0x257f7cbc368
    L000a: mov rcx, [rcx]
    L000d: jmp qword ptr [0x7ff9c9bd7f48]

you might notice a tiny difference: there is one less indirection in .NET 8! But this is another story that will be told in the second edition of the “Pro .NET Memory Management: For Better Code, Performance, and Scalability” book ;^)

Crap: the application is randomly crashing!

Mon, 02 Oct 2023 09:02:26 +0000

Introduction

When you have a call with a customer who explains to you that his application is crashing when your profiler is enabled, it is never a great experience. This post is listing which steps were followed to investigate such an issue I faced last week; from the basics up to the final in analysing memory dumps in WinDbg.

Get as many setup details as possible

The situation was the following:

A web application was running fine with our Datadog .NET profiler on some non-production servers with less traffic.
The same application was crashing on production servers with more traffic.

We were lucky to be able to remote access both machines. A lot of time was spent to check the setup that is based on environment variables. Basically, for our profiler to be loaded by a .NET application, a few Microsoft related environment variables need to be set. Then, you enable the Datadog profiler by setting DD_PROFILING_ENABLED to 1 in order to get the profiling details available in our UI. Since the web application is running in IIS, things get more complicated because some environments variables must be set in… the Registry.

So, we checked the environment variables set at the machine level with the set command in a prompt and those for IIS with the Registry Editor. However, we got some inconsistencies, and we needed a way to validate what were the environment variables really seen by the web application! The Process Explorer tool from Sysinternals was downloaded and launched. After finding the process ID of the running w3wp.exe corresponding to the web application, a simple right-click to get the Properties and selecting the Environment Tab gave us the truth:

(This screenshot shows the results for one of our test applications on my development machine).

Getting a memory dump

Once the setup was checked on both machines without any too weird issues, the next step was to figure out why the application was randomly crashing. Even if the machines received different traffic loads, since applications running without our profiler enabled were not crashing, the chances were high that our C++ code was at the source of the problem. But the crashes were random… And you can’t install Visual Studio on a production server and attach to the process hoping that it will crash and start a debugging session there!

Windows Error Reporting is generating mini dumps when applications are crashing but they are usually not enough to start an investigation. Again, the other Sysinternals tools procdump was installed as a global crash handler with procdump -i c:\dumps -ma. The next time the application crashed, a memory dump was be generated in the c:\dumps folder. Don’t forget to create it manually if it does not exist.

From addresses to source code

To play with a memory dump, WinDbg is my preferred toy. I opened the memory dump and, in the case of a crash, the stack panel automatically displayed the call stack of the faulted thread:

The last frame triggering the issue (i.e., before KiUserExceptionDispatch) is Datadog_Profiler_Native!DllCanUnloadNow+0x2954b. Knowing that WinDbg transforms . in file names into _ leads to Datadog.Profiler.Native.dll which is the file where our profiler is implemented. However, WinDbg was not able to find the name of the function and only looked at the exported public symbols. With the lm command, you can see how WinDbg gets the symbols for this dll:

With DllCanUnloadNow, you could tell that we are dealing with some COM stuff but it did not really help me for the investigation: I needed to know which function was running which part of its code. Hopefully, for each release of the .NET profiler in Github, in addition to the .msi installer, the symbols and the source code are also provided.

Both files were unzipped in the folder where the dumps were copied. Then, I changed the Debugging Settings in WinDbg to point to these folders:

Let’s start with the symbols to let WinDbg match an instruction pointer to a function name. I asked WinDbg to provide details about the symbol resolution with !sym noisy. Then I forced the symbols for my module to gets reloaded with .reload /f “Datadog.Profiler.Native.dll”. In the flow of errors, I find out where the .pdb file should be stored so that WinDbg would find it:

So the problem is triggered somewhere in our Windows64BitStackFramesCollector::CollectStackSampleImplementation function. By simply double-clicking this frame, WinDbg automagically found the corresponding source file and pinpointed the culprit line:

A bit of WinDbg magic

To follow me a bit further, you need to understand what this code is doing: it is walking the stack of a thread to find the instruction pointers of each called function. This line 260 is dereferencing the address contained in context.Rsp. I looked at Locals panel to get its value:

The !address command gave me in which module this code was executed from:

It looked like a valid page with executable code…

I wanted to see why our stack walking code would break here. What if I asked WinDbg to show me this stack? To do that, I first needed to know which thread our code was trying to stack walk. I knew that Windows64BitStackFramesCollector was keeping track of the currently walked thread in a ManagedThreadInfo instance pointed to by its _pCurrentCollectionThreadInfo field:

This instance stores the thread ID in its _osThreadId field: now let’s ask WinDbg to switch to this thread.

The ~ command lists all threads:

A quick CTRL+F with “27600” stopped on the thread #72. Threads have a lot of identifiers in WinDbg and the first one allowed me to switch with ~72s.

The Stack panel was almost empty:

To be sure, I used the kp command… that told me that WinDbg was not really happy neither:

I was kind of stuck but my colleague Kevin Gosse mentioned that I could use r rip to see what would be the next instruction to be executed by this thread:

Then, the ln command (close to the !address command I used just before) allowed me to click the Browse Module link and see that, again, some code from Sentinel One was ready to execute.

This agent is part of an anti-virus (and more) solution that seems to highjack the stack of threads and our code was not dealing properly with this kind of situation. The fix was to protect our dereferencing code against access violation and stop walking the stack in that case.

Another debugging day at Datadog :^)

.NET .gcdump Internals

Fri, 11 Aug 2023 16:31:43 +0000

The .NET runtime (both .NET Framework and .NET Core) allows you to generate a lightweight dump containing the allocated type instances count and references including roots. They are usually generated into .gcdump files by tools such as Perfview or dotnet-gcdump and can also be viewed in Visual Studio. In addition to a view of the allocated types in the managed heap, these files are often used during memory leak investigations because they are much smaller than full memory dump and they contain explicit dependency information between types up to their roots.

The goal of this document is to dig into their generation and see how to leverage the same mechanisms from the .NET CLR for live heap profiling and memory leak detection.

Simply listening to CLR events

Both .NET Framework and .NET Core work the same way to allow a tool to generate a .gcdump file:

It seems that the type table needs to be flushed for .NET Core by first creating and immediately closing an event session with the Microsoft-DotNETCore-SampleProfiler provider.
You create an event session (either through ETW for Framework or EventPipe for Core) where the Microsoft-Windows-DotNETRuntime provider is enabled with a verbose level for a long list of keywords corresponding to the 0x1980001 value:

This will trigger an induced non concurrent gen2 garbage collection during which the GC will walk the remaining live objects with their size and emit tons of events, most of them not documented:

GCStart: wait for the first induced gen2 foreground GC (Depth = 2, Type = GCType.NonConcurrentGC and Reason = GCReason.Induced)
GCStop: detect when the heap walk is over
BulkType: type blocks are enqueued
GCBulkNode: node blocks are enqueued
GCBulkEdge: edge blocks are enqueued
GCBulkRootEdge: enqueue non-weak reference roots (GCRootFlag & GCRootFlags.WeakRef != 0) based on their GCRootKind
GCBulkRootStaticVar: static variable blocks
GCBulkRCW and GCBulkRootCCW for Runtime Callable Wrappers and COM Callable Wrappers COM-based roots
GCBulkRootConditionalWeakTableElementEdge: ??
GCGenerationRange: one for each managed heap segment with generations boundary (not really needed to build the dependency graph but interesting to figure out objects in generation 2)

The payload of most of these events contains arrays of instances with their dependencies. For Perfview and dotnet-gcdump, the whole graph is built in the ConvertHeapDataToGraph method after the garbage collection ends.

Deciphering CLR events payload

When you look at the dotnet-gcdump implementation, you realize that most of the complex code to compute the .gcdump files is physically copied from the Perfview repository.

The GCBulkXXX events payload contains an array of elements; each element being different. The common Count field contains the number of elements. If the element contains a string such as for BulkType, it means that each one has a different size and the string must be read entirely from the payload before accessing the next element.

Type definition

To avoid sending expensive type names all the time, each type will have an identifier that will be used in the GCBulkXXX-nodes related events.

The BulkType events contain an array of types definition elements with the following layout:

TypeID: id of the type (i.e. pointer to the Method Table)
ModuleID: id of the module where the type is defined
TypeNameID: if Name is empty, use this address as a name
Flags: if this bitset contains 0x8, it is an array so append “[]” to the name in that case
CorElementType:?
Name: Unicode string corresponding to the name of the type where `xxx need to be removed in case of generics
TypeParameterCount: for generics but not used
Array of type parameter: for generics but not used

Once the type mappings are known, it becomes possible to build the graph of live type instances instead of just nodes with IDs.

Listing Live Objects and References

The live objects are sent in the GCBulkNode events payload:

Index: incrementing index of the bulk starting from 0
Count: number of objects in the array

followed by an array of Values

Address: address in memory where the object is stored
Size: size of the object (including for arrays)
TypeID: identifier of the object class usable
EdgeCount: number of objects pointed to by this object (i.e. non null reference type fields)

Each event contains an array of live objects identified by their address. The Size and TypeID fields are easy to understand but what does the EdgeCount field represent? This is the number of objects that are referenced by this object. At the code level, this is the count of non-null reference type fields. For example, if a class A defines one integer field and a second one as a reference to an instance of type B, the EdgeCount would be 1 (because an integer is not a reference type).

So the next question is from where do you get which instances are referenced by the objects received in GCBulkNode payload? Since these objects are in memory, they are part of the GCBulkNode events payload but where is the relationship between the EdgeCount value and the corresponding objects? You will have to rebuild this relationship because the referenced objects are received in the GCBulkEdge events payload:

Index: incrementing index of the bulk starting from 0
Count: number of objects in the array

followed by an array of Values

Value: address in memory where the object is stored
ReferencingFieldID: this is not used and is always 0

Again, an array of elements is received as payload and the only interesting information is the address of the object.

But the magic is that both nodes and edges events payload are “in sync”: when an object is read from a GCBulkNode array with let’s say 2 as EdgeCount value, the current 2 elements in the array of the GCBulkEdge payload will contain the addresses of these 2 objects. If the next object in the GCBulkNode array has 1 as EdgeCount value, the next element in the array of the GCBulkEdge payload will be address of this object as shown in the following figure:

It means that both payloads must be iterated in sync.

Just with these two events, it is possible to get a detailed view of the objects still used in memory with their size and their type like what you get with dotnet-gcdump report or !sos.dumpheap -stat.

With the nodes (live objects) and the edges (objects referenced by each object), it is now possible to build the reference graph of live objects.

Listing Roots

In addition to the objects related events, the GC is also emitting events to list the roots that are referencing objects in the managed heap from the stack, statics, handles or other weird places.

The most interesting roots are available thanks to the following events:

GCBulkRootEdge

Index: incrementing index of the bulk starting from 0
Count: number of roots in the array

followed by an array of Values

RootedNodeAddress: address in memory of the root object
GCRootKind: is Stack for local variables
GCRootFlag: if not a local variable, could be RefCounted, Finalizer, strong/pinning handles, or other handles
GCRootID: address of the handle that points to the root object

The static ones are given by the following events:

GCBulkRootStaticVar

Count: number of static roots in the array
AppDomainID: app domain in which the static variable is stored

followed by an array of Values

GCRootID:address of the handle that points to the root object
ObjectID: address of the root object
TypeID: type identifier of the root object
Flags: could be ThreadLocal or not
FieldName: Unicode string corresponding to the name of the field in the type corresponding to the root

Other rarely used roots are available from the GCBulkRootConditionalWeakTableElementEdge and COM-related ones from GCBulkRootCCW/GCBulkRCW with the ref count for example.

So, each of these events provides arrays of root objects addresses. These can be used in conjunction with the reference graph built from the previous node/edge events to identify the reason why objects stay in memory. Like for the ICorProfilerCallback::ObjectReferences usage previously described, it is needed to rebuild the inverse reference chain from the reference graph:

and deal with cycles:

These roots could be an expected cache or a memory leak. For the memory leak scenario, filtering on objects in the gen2 could definitely help. This is where the GCGenerationRange events could help because their payload contains the ranges of memory addresses in each segment with the corresponding generation:

GCGenerationRange

Generation: generation of the segment
RangeStart: address of the start of the segment
RangeUsedLength: size of the committed part of the segment
RangeReservedLength: size of the reserved part of the segment

When an address fits inside RangeStart and RangeStart + RangeUsedLength, it is part of this segment. The generation of the segment could be 0, 1 or 2 for the ephemeral segments, 3 for the Large Object Heap, and 4 for the Pinned Object Heap.

Integration with a .NET Profiler

As a .NET Profiler, it is possible to listen to CLR events via ICorProfilerCallback::EventPipeEventDelivered. If the same keywords as 0x1980001 have been enabled thanks to ICorProfilerInfo12::EventPipeStartSession, the corresponding messages will be received and you have to keep track of the fact that a gcdump is in progress. This should not be a big deal because only the GC keyword (0x1) might be already used and events describing the collections will be processed anyway. There won’t be duplication of events in that case.

However, since it is needed to start a session with the right keyword to trigger the special garbage collection, the ICorProfilerCallback mechanism cannot be used to continuously process the corresponding specific messages. This one time EventPipe session should be started independently by manually connecting to the EventPipe of the currently running CLR as described in details in this blog series.

You are now ready to integrate this feature of the CLR without the need to install a tool!

From Metadata to Event block in nettrace format

Fri, 10 Mar 2023 16:15:40 +0000

The previous episodes started the parsing of the “nettrace” format used when contacting the .NET Diagnostics IPC server, initiate the protocol to receive CLR events and start to parse stacks. This last episode covers the Metadata and Event blocks.

In terms of format, both Metadata and Event blocks share the same memory layout:

The common EventBlockHeader starts the block:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


#pragma pack(1)
struct EventBlockHeader
{
    uint16_t HeaderSize;
    uint16_t Flags;
    uint64_t MinTimestamp;
    uint64_t MaxTimestamp;

    // some optional reserved space might be following
};

The timestamp fields give the time of the first and last event in the block. The HeaderSize fields is important because additional information can be stored in the header. Since I have no idea what could be stored there, I simply skip it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


bool EventParserBase::OnParse()
{
    // read event block header
    EventBlockHeader ebHeader = {};
    if (!Read(&ebHeader, sizeof(ebHeader)))
    {
        return false;
    }

    // skip any optional content if any
    if (ebHeader.HeaderSize > sizeof(EventBlockHeader))
    {
        uint8_t optionalSize = ebHeader.HeaderSize - sizeof(EventBlockHeader);
        if (!SkipBytes(optionalSize))
        {
            return false;
        }
    }

The important piece of information to figure out how to unpack the rest of the block is kept in the Flags field. If the lowest bit is set, it means that the blobs header will be compressed:

1
2
3
4
5
6


// the rest of the block is a list of Event blobs
    //
    DWORD blobSize = 0;
    DWORD totalBlobSize = 0;
    DWORD remainingBlockSize = _blockSize - ebHeader.HeaderSize;
    bool isCompressed = ((ebHeader.Flags & 1) == 1);

The rest of the code iterates on each blob:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


// Note: in order to gain space, some fields of the header could be "inherited"
    // from the header of the previous blob --> need to pass it from blob to blob
    EventBlobHeader header = {};
    while (OnParseBlob(header, isCompressed, blobSize))
    {
        totalBlobSize += blobSize;
        blobSize = 0;

        if (totalBlobSize >= remainingBlockSize - 1) // try to detect last blob
        {
            // don't forget to check the end of block tag
            uint8_t tag;
            if (!ReadByte(tag) || (tag != NettraceTag::EndObject))
            {
                std::cout << "Missing end of block tag\n";
                return false;
            }

            return true;
        }
    }

    return true;
}

Here is the tricky part: to gain space, each blob starts with a header that could be “compressed”. The compression mechanism is simple: the first byte is a bitfield value that indicates which fields are present (i.e. their value should be read from the memory block) or skipped (i.e. their value is the same as the previous blob header). Therefore, an EventBlobHeader is passed by reference to the OnParseBlob function. My MetadataParser and EventParser implementations of OnParseBlob both starts with the same code to read the header:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


bool XXXParser::OnParseBlob(EventBlobHeader& header, bool isCompressed, DWORD& blobSize)
{
    if (isCompressed)
    {
        if (!ReadCompressedHeader(header, blobSize))
        {
            return false;
        }
    }
    else
    {
        if (!ReadUncompressedHeader(header, blobSize))
        {
            return false;
        }
    }

The implementation to read compressed and uncompressed version of the header is a direct translation of the TraceEvent C# code into C++.

The EventBlobHeader contains details of events:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


#pragma pack(1)
struct EventBlobHeader_V4
{
    uint32_t EventSize;
    uint32_t MetadataId;
    uint32_t SequenceNumber;
    uint64_t ThreadId;
    uint64_t CaptureThreadId;
    uint32_t ProcessorNumber;
    uint32_t StackId;
    uint64_t Timestamp;
    GUID ActivityId;
    GUID RelatedActivityId;
    uint32_t PayloadSize;
};

The “identity” of an event is given by the MetadataId field that refers to information defined in Metadata “object“ (for which MetadataId is 0).
The SequenceNumber field is incremented on a per thread basis each time an event is emitted. This could be used to detect if some events have been dropped (for a given CaptureThreadId, two consecutive events have a SequenceNumber incremented by more than 1 — more on dropped events in the forthcoming SequencePoint “object” description). Its value is 0 for a metadata “object”
The ThreadId and CaptureThreadId field have always the same value for Event “object”; CaptureThreadId is 0 for Metadata “object”.
In case of Event “object”, the StackId field refers to one of the stacks extracted from a Stack “object”. Its value is 0 for Metadata “object”.

The Metadata “object”

As the documentation states, each MetadataBlock holds a set of metadata records. Each metadata record has an ID and it describes one type of event. Each event has a metadataId field which will indicate the ID of the metadata record which describes that event.

The resulting mapping is stored in EventPipeSession class:

1
2


// per metadataID event metadata description
    std::unordered_map<uint32_t, EventCacheMetadata> _metadata;

However, the rest of the documentation is partially right in the case of nettrace stream received through EventPipe: Metadata includes an event name, provider name, and the layout of fields that are encoded in the event’s payload section.

First, the fields layout is simply not there. In addition, for some providers (dotnet runtime, private and rundown), the event names are empty strings. So, the data structure filled from the MetadataBlock will most of the time have an empty EventName field. Note that the “Microsoft-DotNETCore-EventPipe” provider (i.e. command events for that specific provider) and EventSource-derived classes written in C# provide the events name:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


class EventCacheMetadata
{
public:
    uint32_t     MetadataId;
    std::wstring ProviderName;
    uint32_t     EventId;
    std::wstring EventName; // empty most of the time
    uint64_t     Keywords;
    uint32_t     Version;
    uint32_t     Level;
};

In addition to the provider’s name serialized as a UTF16 string (including last ‘\0’ wide character), the EventId field is the key used to identify an event.

After these details, you will find a 4 bytes value corresponding to the number of fields in the event payload. As already mentioned, this value is always 0 so my code is skipping the rest of the metadata block payload.

The Event “object”

And at last, here comes the time to parse Event “object” payload! The MetadataId field of the EventBlobHeader is used to find the provider’s name and event id:

1
2
3
4
5


bool EventParser::OnParseBlob(EventBlobHeader& header, bool isCompressed, DWORD& blobSize)
{
    ...

    auto& metadataDef = _metadata[header.MetadataId];

So, the rest of the function reads the payload based on the expected event id:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


switch (metadataDef.EventId)
    {
        case EventIDs::AllocationTick:
            if (!OnAllocationTick(header.PayloadSize, metadataDef))
            {
                return false;
            }
            break;

        case ...
            break;

        case EventIDs::ExceptionThrown:
            if (!OnExceptionThrown(header.PayloadSize, metadataDef))
            {
                return false;
            }
            break;

        default:  // skip events we are not interested in
        {
            SkipBytes(header.PayloadSize);
        }
    }

    blobSize += header.PayloadSize;

    return true;
}

The format of each event payload is usually given by the Microsoft documentation. If not, you should look into the ClrEtwall.man file where the payload of ALL events are defined. For example, the AllocationTick event payload provides the name of the last allocated type to reach the 100 KB threshold (read this blog post for more details about how to use this event):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


//  AllocationAmount    UInt32          The allocation size, in bytes.
//                                      This value is accurate for allocations that are less than the length of a ULONG(4,294,967,295 bytes).
//                                      If the allocation is greater, this field contains a truncated value.
//                                      Use AllocationAmount64 for very large allocations.
//  AllocationKind      UInt32          0x0 - Small object allocation(allocation is in small object heap).
//                                      0x1 - Large object allocation(allocation is in large object heap).
//  ClrInstanceID       UInt16          Unique ID for the instance of CLR or CoreCLR.
//  AllocationAmount64  UInt64          The allocation size, in bytes.This value is accurate for very large allocations.
//  TypeId              Pointer         The address of the MethodTable.When there are several types of objects that were allocated during this event,
//                                      this is the address of the MethodTable that corresponds to the last object allocated (the object that caused the 100 KB threshold to be exceeded).
//  TypeName            UnicodeString   The name of the type that was allocated.When there are several types of objects that were allocated during this event,
//                                      this is the type of the last object allocated (the object that caused the 100 KB threshold to be exceeded).
//  HeapIndex           UInt32          The heap where the object was allocated.This value is 0 (zero)when running with workstation garbage collection.
//  Address             Pointer         The address of the last allocated object.
//

Based on this fields definition, the EventParser::OnAllocationTick function is reading each field after the other thanks to the ReadWord, ReadDWord, ReadLong and ReadWString :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


bool EventParser::OnAllocationTick(DWORD payloadSize, EventCacheMetadata& metadataDef)
{
    DWORD readBytesCount = 0;
    DWORD size = 0;
    std::cout << "\nAllocation Tick:\n";

    // get common fields
    uint32_t dword = 0;
    if (!ReadDWord(dword))
    {
        return false;
    }
    readBytesCount += sizeof(dword);
    std::cout << "   Amount        = " << dword << " bytes\n";

    if (!ReadDWord(dword))
    {
        return false;
    }
    readBytesCount += sizeof(dword);
    std::cout << "   Kind          = " << ((dword == 1) ? "LOH" : "small") << " bytes\n";

    uint16_t word = 0;
    if (!ReadWord(word))
    {
        return false;
    }
    readBytesCount += sizeof(word);
    std::cout << "   CLR ID        = " << word << "\n";

    uint64_t ulong = 0;
    if (!ReadLong(ulong))
    {
        return false;
    }
    readBytesCount += sizeof(ulong);
    std::cout << "   Amount64      = " << ulong << " bytes\n";

The bitness of the monitored application is important when “pointers” need to be read from the payload: use ReadDWord for 32-bit and ReadLong for 64-bit:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


// skip useless MT address
    // Note: handle 32/64 bit difference
    if (_is64Bit)
    {
        if (!ReadLong(ulong))
        {
            return false;
        }
        readBytesCount += sizeof(ulong);
    }
    else
    {
        if (!ReadDWord(dword))
        {
            return false;
        }
        readBytesCount += sizeof(dword);
    }

And if you don’t need the rest of the payload, SkipBytes is your friend:

1
2
3


// skip the rest of the payload
    return SkipBytes(payloadSize - readBytesCount);
}

I had some issues when dealing with the ExceptionThrown event payload:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


// Type             wstring     Exception type
// Message          wstring     Exception message
// EIPCodeThrow     win:Pointer Instruction pointer where exception occurred.
// ExceptionHR      win:UInt32  Exception HRESULT.
// ExceptionFlags   win:UInt16
//      0x01: HasInnerException (see CLR ETW Events in the Visual Basic documentation).
//      0x02: IsNestedException.
//      0x04: IsRethrownException.
//      0x08: IsCorruptedStateException (indicates that the process state is corrupt).
//      0x10: IsCLSCompliant (an exception that derives from Exception is CLS-compliant).
// ClrInstanceID win:UInt16 Unique ID for the instance of CLR or CoreCLR.

In case of an empty message, the field itself was not even there! Not even 0 for a ‘\0’ wide character… In fact, there is a bug in the serialization code that skips the field in that case. This has been fixed in .NET 6 by storing “NULL” as the serialized string: I would have preferred ‘\0’ but it seems to be compatible with the ETW implementation.

To support .NET Core 3+ and .NET 5, my code is comparing the size of the remaining of the payload after reading the exception type with the expected size of the 4 remaining fields after the exception message. If it is greater then it means that there is a string for the message. If not, I know that the message is empty:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


bool EventParser::OnExceptionThrown(DWORD payloadSize, EventCacheMetadata& metadataDef)
{
    DWORD readBytesCount = 0;
    DWORD size = 0;

    // read exception type
    ...

    // Size of the ExceptionThrown payload AFTER the Message field
    uint16_t exceptionRemainingPayloadSize = (_is64Bit ? 8 : 4) + 4 + 2 + 2;

    // In case of "empty" message, it might not be even visible as "\0" before .NET Core 6 (and after, will be "NULL")
    // so it is needed to check if the remaining payload contains such a string
    if ((payloadSize - readBytesCount) == _exceptionRemainingPayloadSize)
    {
        std::wcout << L"   message = ''\n";
    }
    else
    {
        if (!ReadWString(strBuffer, size))
        {
            return false;
        }
        readBytesCount += size;

        // handle empty string case (check for "NULL" in case of .NET 6+)
        if (strBuffer.empty() || (wcscmp(strBuffer.c_str(), L"NULL") == 0))
            std::wcout << L"   message = ''\n";
        else
        {
            std::wcout << L"   message = " << strBuffer.c_str() << L"\n";
        }
    }

    // skip the rest of the payload
    return SkipBytes(payloadSize - readBytesCount);
}

The SequencePointBlock “object”

The last “object” type is the sequence point block that contains the following fields:

In addition to these fields, it also implicitly tells you that new stack “object” will be received (with stack id restarting from 1) to match next Event “objects”. For example, the following trace shows how a sequence point block resets the stacks by restarting at 1:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44


Event block (140 bytes)

blob header:
   StackId           = 3
Contention

blob header:
   StackId           = 4
Event = 81

blob header:
   StackId           = 3
Contention
...
------------------------------------------------ 

________________________________________________
SequencePoint block (217 bytes)
...
------------------------------------------------

________________________________________________
Stack block (105 bytes)

Stack block header:
   FirstID: 1
   Count  : 2
------------------------------------------------

________________________________________________
Event block (92 bytes)

blob header:
   StackId           = 1
Contention

blob header:
   StackId           = 2
Event = 81

blob header:
   StackId           = 1
Contention:
------------------------------------------------

So, the stacks you might have cached based on the already received stack “objects” should now be invalidated like what I’m doing in SequencePointParser::OnParse:

1
2
3
4
5
6


bool SequencePointParser::OnParse()
{
    // reset stack caches
    _stacks32.clear();
    _stacks64.clear();
    ...

You now have all the elements you need to listen to CLR events on Windows and Linux for .NET Core 3+ and .NET 5+. If you are still running applications with .NET Framework, you will need to use ETW but this is another story.

Resources

Episode 1 — Digging into the CLR Diagnostics IPC Protocol in C#
Episode 2 — .NET Diagnostic IPC protocol: the C++ way
[Episode 3 ](/posts/2022-10-23_clr-events-go-for/ CLR events: go for the nettrace file format!
[Episode 4 ](/posts/2022-11-27_parsing-the-nettrace-stream/ Parsing the “nettrace” steam
[Episode 5 ](/posts/2023-01-15_reading-object-in-memory/ Reading “object” in memory — starting with stacks
Source code for the C++ implementation of CLR events listener
Diagnostics IPC protocol documentation

Reading “object” in memory — starting with stacks

Sun, 15 Jan 2023 16:38:45 +0000

The previous episodes started the parsing of the “nettrace” format used when contacting the .NET Diagnostics IPC server and initiate the protocol to receive CLR events. It is now time to see how to get the payload of each “object” type, especially how stacks are stored.

We have seen that the stream starts with a TraceObject that describes the rest of the stream followed by a sequence of “object”:

The remaining of each “object” is a 32 bit block size followed by the payload.

Well… not only. One thing I missed when I started to work on the nettrace format is the fact that all “object” payloads must be 4-bytes aligned on the beginning of the stream!

This is why I’m keeping track of the current position in the EventPipeSession class:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


private:
    IIpcEndpoint* _pEndpoint;
    bool _stopRequested;

    // parsers
    MetadataParser _metadataParser;
    EventParser _eventParser;
    StackParser _stackParser;
    SequencePointParser _sequencePointParser;

    // Keep track of the position since the beginning of the "file"
    // i.e. starting at 0 from the first character of the NettraceHeader
    //      Nettrace
    uint64_t _position;
    ...
};

So each ParseXXXBlock function checks the minimum reader version in the header before reading the “object” payload as a memory block. The idea is being able to support backward compatibility:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


bool EventPipeSession::ParseMetadataBlock(ObjectHeader& header)
{
    if (header.MinReaderVersion != 2) return false;

    uint32_t blockSize = 0;

    // read the block and send it to the corresponding parser
    uint64_t blockOriginInFile = 0;
    if (!ExtractBlock("Metadata", blockSize, blockOriginInFile))
        return false;

    return _metadataParser.Parse(_pBlock, blockSize, blockOriginInFile);
}

The ExtractBlock function reads the size of the payload (and skips the padding if any) with ReadBlockSize:

1
2
3
4
5
6
7
8
9


bool EventPipeSession::ExtractBlock(const char* blockName, uint32_t& blockSize, uint64_t& blockOriginInFile)
{
    // get the block size
    if (!ReadBlockSize(blockName, blockSize))
        return false;

    // skip the block + final EndOfObject tag
    blockSize++;
    ...

The block name is only used for error messages if needed.

The next step is to read the payload in a memory block using these two EventPipeSession fields:

1
2
3
4
5


...
    // buffer used to read each block that will be then parsed
    uint8_t* _pBlock;
    uint32_t _blockSize;
    ...

In the session constructor, _blockSize is set to 4 KB and _pBlock points to an allocated memory buffer of that size.

The rest of ExtractBlock deals with payload size: if the current payload to parse is larger than _blockSize, then these fields are updated up to a maximum of 100 KB (i.e. max block size sent by the CLR).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


// check if it is needed to resize the block buffer
    if (_blockSize < blockSize)
    {
        // don't expect blocks larger than 100KB
        if (blockSize > MAX_BLOCK_SIZE)
            return false;

        delete [] _pBlock;
        _pBlock = new uint8_t[blockSize];
        ::ZeroMemory(_pBlock, blockSize);
        _blockSize = blockSize;
    }

    // keep track of the current position in file for padding
    blockOriginInFile = _position;
    if (!Read(_pBlock, blockSize))
    {
        Error = ::GetLastError();
        std::cout << "Error while extracting " << blockName << " block: 0x" << std::hex << Error << std::dec << "\n";
        return false;
    }
    std::cout << "\n" << blockName << " block (" << blockSize << " bytes)\n";
    DumpBuffer(_pBlock, blockSize);

    return true;
}

For debugging sake, I’m displaying each “object” payload

thanks to the DumpBuffer helper.

To ease the memory access to the memory block content, my BlockParser will be used as a base class for each dedicated parsers:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


class BlockParser
{
public:
    BlockParser();
    bool Parse(uint8_t* pBlock, uint32_t bytesCount, uint64_t blockOriginInFile);
    void SetPointerSize(uint8_t pointerSize);

public:
    uint8_t PointerSize;

protected:
    virtual bool OnParse() = 0;

    // Access helpers
    bool Read(LPVOID buffer, DWORD bufferSize);
    bool ReadByte(uint8_t& byte);
    bool ReadWord(uint16_t& word);
    bool ReadDWord(uint32_t& dword);
    bool ReadLong(uint64_t& ulong);
    bool ReadDouble(double& d);
    bool ReadVarUInt32(uint32_t& val, DWORD& size);
    bool ReadVarUInt64(uint64_t& val, DWORD& size);
    bool ReadWString(std::wstring& wstring, DWORD& bytesRead);
    bool SkipBytes(uint32_t byteCount);

// shared fields
protected:
    bool _is64Bit;
    uint32_t _blockSize;
    uint32_t _pos;

private:
    uint8_t* _pBlock;
    uint64_t _blockOriginInFile;
};

The Parse function accepts the memory buffer containing an “object” payload, its size and its position since the beginning of the stream. The derived class will have to implement the OnParse function using the ReadXXX helpers.

The two ReadVarUintXXX functions are different from the other direct read helpers because they deal with some simple compression mechanisms used by the serialization of 32-bit and 64-bit numbers.

In the different types of “object” payloads, the strings are serialized as UTF16 strings ending with a “\0” wide character. Here is the implementation of the helper function used to read a std::wstring from a memory block:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


bool BlockParser::ReadWString(std::wstring& wstring, DWORD& bytesRead)
{
    uint16_t character;
    bytesRead = 0;  // in case of empty string
    while (true)
    {
        if (!ReadWord(character))
        {
            return false;
        }

        // protect against invalid UNICODE character (due to missing fields in ExceptionThrown event)
        if (character > 256)
        {
            // rewind the character
            _pos = _pos - sizeof(character);

            // this is only covering a missing string
            return (bytesRead == 0);
        }

        bytesRead += sizeof(character);

        // Note that an empty string contains only that \0 character
        if (character == 0) // \0 final character of the string
            return true;

        wstring.push_back(character);
    }
}

Note the check for character content in the loop: this is due to a serialization issue I will discuss later when the event “object” block will be detailed.

The Stack “object”

If you remember my previous post about retrieving call stacks for CLR events with TraceEvent, you might be wondering why there is a specific stack object since a ClrStackWalk event should contain the frames if the Stack keyword is enabled for the .NET provider. In fact, the current TraceEvent implementation is not using the stack object sent by the CLR (maybe to have the same code between ETW and EventPipe).

One stack “object” received in a nettrace stream contains one or more stacks. Each stack is identified by an id (more about this soon) and contains a list of instruction pointer addresses.

In the previous screenshot, the id of the first stack is 1 and the second is 2. In the next stack “object”, the FirstId field will be 3 and so on. This avoids storing the id in each call stack and saves space.

Note that even if this does not seem to make any sense, it might happen that the addresses list is empty.

These call stacks are stored in EventPipeSession as a per id cache:

1
2
3
4


// per stackID stack
    // only one will be used depending on the bitness of the monitored application
    std::unordered_map<uint32_t, EventCacheStack32> _stacks32;
    std::unordered_map<uint32_t, EventCacheStack64> _stacks64;

The frames are stored as addresses in a vector:

1
2
3
4
5
6


class EventCacheStack32
{
public:
    uint32_t Id;
    std::vector<uint32_t> Frames;
};

The CLR is sending one stack per unique callstack (i.e. at least one frame is different). As you will soon see, each event “object” contains a stack id corresponding to the chain of code from which it is sent.

The next episode will detail the Metadata and Event blocks to end the series.

Resources

Episode 1 — Digging into the CLR Diagnostics IPC Protocol in C#
Episode 2 — .NET Diagnostic IPC protocol: the C++ way
[Episode 3 ](/posts/2022-10-23_clr-events-go-for/ CLR events: go for the nettrace file format!
[Episode 4 ](/posts/2022-11-27_parsing-the-nettrace-stream/ Parsing the “nettrace” steam
Source code for the C++ implementation of CLR events listener
Diagnostics IPC protocol documentation

CLR events: go for the nettrace file format!

Sun, 23 Oct 2022 16:55:52 +0000

As shown in the previous post, the processing of ProcessInfo diagnostic commands is easy because you send a request and read the different fields from the response. This is different if you want to receive events from the CLR via EventPipe. In C#, the TraceEvent nuget package wraps everything under a nice event handler based model as shown in many of my previous posts.

Behind the scene, a StartSession command is sent (more details about the parameters later) and the response contains the numeric ID of the session. Then, the events will be read from the IPC channel as a binary stream of data with the “nettrace“ file format. The collection ends when the StopTracing command is sent.

The source code is available from my github repository.

Hidding the transport layer: IIpcEndoint

Unlike the previous post, to send the command and read the response back from the CLR , I’m wrapping the transport layer with the IIpcEndpoint interface:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


class IIpcEndpoint
{
public:
    virtual bool Write(LPCVOID buffer, DWORD bufferSize, DWORD* writtenBytes) = 0;
    virtual bool Read(LPVOID buffer, DWORD bufferSize, DWORD* readBytes) = 0;
    virtual bool ReadByte(uint8_t& byte) = 0;
    virtual bool ReadWord(uint16_t& word) = 0;
    virtual bool ReadDWord(uint32_t& dword) = 0;
    virtual bool ReadLong(uint64_t& ulong) = 0;

    virtual bool Close() = 0;
    virtual ~IIpcEndpoint() = default;
};

It abstracts the write and read accesses to the underlying transport layer. In addition, the base class accepts a “recorder” that allows me to store what is received from the CLR into any kind of storage (today only a file-based recorder that helped a lot to reproduce specific situations without the need to have a running process to connect to):

The PidEndpoint class accepts the process id of the running .NET application to monitor its CLR events and an optional recorder implementing the IIpcRecorder interface. The Create static factory implementation creates the expected named pipe on Windows (or the domain socket on Linux) and stores the handle into its _handle field:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42


PidEndpoint* PidEndpoint::CreateForWindows(int pid, IIpcRecorder* pRecorder)
{
    PidEndpoint* pEndpoint = new PidEndpoint(pRecorder);

    // build the pipe name as described in the protocol
    wchar_t pszPipeName[256];
    int nCharactersWritten = -1;
    nCharactersWritten = wsprintf(
        pszPipeName,
        L"\\\\.\\pipe\\dotnet-diagnostic-%d",
        pid
    );

    // check that CLR has created the diagnostics named pipe
    if (!::WaitNamedPipe(pszPipeName, 200))
    {
        auto error = ::GetLastError();
        std::cout << "Diagnostics named pipe is not available for process #" << pid << " (" << error << ")" << "\n";
        return nullptr;
    }

    // connect to the named pipe
    HANDLE hPipe;
    hPipe = ::CreateFile(
        pszPipeName,    // pipe name
        GENERIC_READ |  // read and write access
        GENERIC_WRITE,
        0,              // no sharing
        NULL,           // default security attributes
        OPEN_EXISTING,  // opens existing pipe
        0,              // default attributes
        NULL);          // no template file

    if (hPipe == INVALID_HANDLE_VALUE)
    {
        std::cout << "Impossible to connect to " << pszPipeName << "\n";
        return nullptr;
    }

    pEndpoint->_handle = hPipe;
    return pEndpoint;
}

The next step is to open a tracing session by sending the StartSession command.

The Trace diagnostic commands

Following the same object model provided by the Microsoft.Diagnostics.NETCore.Client nuget, my DiagnosticsClient class hides the transport layer. It also exposes high level functions such as OpenEventPipeSession to initiate a trace event session with the CLR:

1

EventPipeSession* OpenEventPipeSession(uint64_t keywords, EventVerbosityLevel verbosity);

If you remember from TraceEvent, you need a few parameters to create a session:

size of circular buffers used by the CLR to cache events (same as Perfview, use 16 MB as default)
netttrace format (i.e. value of 1)
if rundown events are needed
a list of providers (“Microsoft-Windows-DotNETRuntime” for the CLR in my case)
keywords
verbosity level
possible arguments (none here)

Here is the corresponding C++ description of the command type:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


const uint8_t DotnetProviderMagicLength = 32;
struct MagicProvider
{
    wchar_t Magic[DotnetProviderMagicLength];
};

// 32 wchar_t (including \0)
const MagicProvider DotnetProviderMagic = { L"Microsoft-Windows-DotNETRuntime" };

const uint32_t CircularBufferMBSize = 16;
const uint32_t NetTraceFormat = 1;

#pragma pack(1)
struct StartSessionMessage : public IpcHeader
{
    uint32_t CircularBufferMB;  // 16 MB
    uint32_t Format;            // 1 for NetTrace format
    uint8_t RequestRundown;     // 0 because don't want rundown

    // array of provider configuration
    uint32_t ProviderCount;     // 1 only: Microsoft-Windows-DotNETRuntime
    uint64_t Keywords;          // from EventKeyword
    uint32_t Verbosity;         // from EventPipeEventLevel
    uint32_t ProviderStringLen; // number of UTF16 characters = 32 (including last \0)
    union                       // dotnet provider name
    {
        MagicProvider _magic;
        uint8_t Provider[2 * DotnetProviderMagicLength];
    };
    uint32_t Arguments;         // 0 for empty string (no argument)
};

The code to fill up the command is straightforward:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


StartSessionMessage* CreateStartSessionMessage(uint64_t keywords, EventVerbosityLevel verbosity)
{
    auto message = new StartSessionMessage();
    ::ZeroMemory(message, sizeof(message));
    memcpy(message->Magic, &DotnetIpcMagic_V1, sizeof(message->Magic));
    message->Size = sizeof(StartSessionMessage);  
    message->CommandSet = (uint8_t)DiagnosticServerCommandSet::EventPipe;
    message->CommandId = (uint8_t)EventPipeCommandId::CollectTracing2;
    message->Reserved = 0;
    message->CircularBufferMB = CircularBufferMBSize;
    message->Format = NetTraceFormat;
    message->RequestRundown = 0;
    message->ProviderCount = 1;
    message->Keywords = (uint64_t)keywords;
    message->Verbosity = (uint32_t)verbosity;
    message->ProviderStringLen = DotnetProviderMagicLength;
    memcpy(message->Provider, &DotnetProviderMagic, sizeof(message->Provider));
    message->Arguments = 0;

    return message;
}

The provider list is defined with the ProviderCount field and string containing the list (only one here) follows the Verbosity field. To start the session, it is needed to send the StartSession message and read the session id from the response:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


bool EventPipeStartRequest::Process(IIpcEndpoint* pEndpoint, uint64_t keywords, EventVerbosityLevel verbosity)
{
    // send an StartSessionMessage and parse the response
    StartSessionMessage* pMessage = CreateStartSessionMessage(keywords, verbosity);

    DWORD writtenBytes = 0;
    if (!pEndpoint->Write(pMessage, pMessage->Size, &writtenBytes))
    {
        return false;
    }

    // analyze the response
    IpcHeader response = {};
    DWORD bytesReadCount = 0;
    if (!pEndpoint->Read(&response, sizeof(response), &bytesReadCount))
    {
        return false;
    }

    if (response.CommandId != (uint8_t)DiagnosticServerResponseId::OK)
    {
        return false;
    }

    // get the session ID from the payload
    uint16_t payloadSize = response.Size - sizeof(response);
    if (payloadSize < sizeof(uint64_t))
    {
        return false;
    }

    if (!pEndpoint->ReadLong(SessionId))
    {
        return false;
    }

    return true;
}

Once the StartSession command has been sent, the events corresponding to the given provider/keywords/verbosity (here the CLR runtime/gc+exception+contention/verbose)

1
2
3
4
5
6


auto pSession = pClient->OpenEventPipeSession(
        EventKeyword::gc |
          EventKeyword::exception |
          EventKeyword::contention,
        EventVerbosityLevel::Verbose  // required for AllocationTick
        );

will be read from the event pipe. Since this action will be synchronous, it is recommended to dedicate a thread to read and process the events:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


    if (pSession != nullptr)
    {
        DWORD tid = 0;
        auto hThread = ::CreateThread(nullptr, 0, ListenToEvents, pSession, 0, &tid);

        std::cout << "Press ENTER to stop listening to events...\n\n";
        std::string line;
        std::getline(std::cin, line);

        std::cout << "Stopping session\n\n";
        pSession->Stop();
        std::cout << "Session stopped\n\n";

        // test if it works
        ::Sleep(1000);

        ::CloseHandle(hThread);
    }

The ListenToEvents callback executed by the new thread is “simply” listening to the event pipe of the session:

1
2
3
4
5
6
7
8


DWORD WINAPI ListenToEvents(void* pParam)
{
    EventPipeSession* pSession = static_cast<EventPipeSession*>(pParam);

    pSession->Listen();

    return 0;
}

Before describing how to read the events, it is important to understand how to stop the flow. First, inside the EventPipeSession, the internal loop that reads events needs to exit thanks to the _stopRequested boolean:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


bool EventPipeSession::Stop()
{
    _stopRequested = true;

    if (_pid == -1)
        return true;

    // it is neeeded to use a different ipc connection to stop the Session
    DiagnosticsClient* pStopClient = DiagnosticsClient::Create(_pid, nullptr);
    pStopClient->StopEventPipeSession(SessionId);
    delete pStopClient;

    return true; 
}

In addition, a message with StopTracing command id from the EventPipe command set needs to be sent to tell the CLR to stop sending the events. This message must be sent through a different IPC channel (hence the pStopClient variable used in the previous code. The StopEventPipeSession helper function uses the EventPipeStopRequest wrapper:

1
2
3
4
5


bool DiagnosticsClient::StopEventPipeSession(uint64_t sessionId)
{
    EventPipeStopRequest request;
    return request.Process(_pEndpoint, sessionId);
}

The StopSession command accepts the session ID as single parameter:

1
2
3
4
5


#pragma pack(1)
struct StopSessionMessage : public IpcHeader
{
    uint64_t SessionId;
};

The processing of the stop request is to create such a message and send it through the IPC channel:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32


StopSessionMessage* CreateStopMessage(uint64_t sessionId)
{
    StopSessionMessage* message = new StopSessionMessage();
    ::ZeroMemory(message, sizeof(message));
    memcpy(message->Magic, &DotnetIpcMagic_V1, sizeof(message->Magic));
    message->Size = sizeof(StopSessionMessage);
    message->CommandSet = (uint8_t)DiagnosticServerCommandSet::EventPipe;
    message->CommandId = (uint8_t)EventPipeCommandId::StopTracing;
    message->Reserved = 0;
    message->SessionId = sessionId;

    return message;
}

bool EventPipeStopRequest::Process(IIpcEndpoint* pEndpoint, uint64_t sessionId)
{
    StopSessionMessage* pMessage = CreateStopMessage(sessionId);
    DWORD writtenBytes;
    if (!pEndpoint->Write(pMessage, pMessage->Size, &writtenBytes))
    {
        Error = ::GetLastError();
        std::cout << "Error while sending EventPipe Stop message to the CLR: 0x" << std::hex << Error << std::dec << "\n";
        delete pMessage;
        return false;
    }
    
    delete pMessage;

    ... // handle the response 

    return true;
}

When the stop command is received by the CLR, the remaining “data” (more on this in the next episode) is sent through the first IPC channel before being closed. This is how the code knows that the session can stop listening to the EventPipe.

The next episode will start to parse the nettrace stream of events.

Resources

Episode 1 — Digging into the CLR Diagnostics IPC Protocol in C#
Episode 2 — .NET Diagnostic IPC protocol: the C++ way
Source code for the C++ implementation of CLR events listener
Diagnostics IPC protocol documentation
Microsoft.Diagnostics.NETCore.Client source code

.NET Diagnostic IPC protocol: the C++ way

Sun, 18 Sep 2022 14:36:31 +0000

The previous post was describing the C# helpers to communicate with the diagnostic server in the CLR of a running .NET application.

If, like me, you must write native code (i.e not in C#), you will need to implement the transport and protocol yourself. And, as you will see, it is not that complicated thanks to the documentation but also by using the available C# code of the Microsoft.Diagnostics.NETCore.Client implementation as a guide.

EventPipe transport layer

The first step is to connect to the CLR of a running .NET process. On Linux, you connect to a domain socket named “{$TMPDIR}/dotnet-diagnostic-{%d:PID}-{%llu:disambiguation key}-socket”. For Windows, a named pipe called \.\pipe\dotnet-diagnostic-{%d:PID} needs to be accessed.

Here is the Windows implementation code to connect to the IPC named pipe:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


int BasicConnection(DWORD pid)
{
    wchar_t pszPipeName[256];

    // build the pipe name as described in the protocol
    int nCharactersWritten = -1;
    nCharactersWritten = wsprintf(
        pszPipeName,
        L"\\\\.\\pipe\\dotnet-diagnostic-%d",
        pid
    );

    // check that CLR has created the diagnostics named pipe
    if (!::WaitNamedPipe(pszPipeName, 200))
    {
        auto error = ::GetLastError();
        std::cout << "Diagnostics named pipe is not available for process #" << pid << " (" << error << ")" << "\n";
        return -1;
    }

    // connect to the named pipe
    HANDLE hPipe;
    hPipe = ::CreateFile(
        pszPipeName,    // pipe name
        GENERIC_READ |  // read and write access
        GENERIC_WRITE,
        0,              // no sharing
        NULL,           // default security attributes
        OPEN_EXISTING,  // opens existing pipe
        0,              // default attributes
        NULL);          // no template file

    if (hPipe == INVALID_HANDLE_VALUE)
    {
        std::cout << "Impossible to connect to " << pszPipeName << "\n";
        return -2;
    }

        // ... send a command...

    // don't forget to close the named pipe
    ::CloseHandle(hPipe);

    return 0;
}

Dig into the Command protocol

Once the connection is created, a command can be sent by writing to the pipe (or socket on Linux). The answer will be received by reading from the pipe (or socket on Linux). There is something important to remember: if you need to send different commands, it is needed to create one connection for each. You should not try to reuse a given connection that might also be closed after a command has been processed.

Let’s start with the IpcHeader

Both the command and the response share the same header format:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


#pragma pack(1)
struct IpcHeader
{
    union
    {
        MagicVersion _magic;
        uint8_t  Magic[14];  // Magic Version number; a 0 terminated ANSI char array
    };
    uint16_t Size;       // The size of the incoming packet, size = header + payload size
    uint8_t  CommandSet; // The scope of the Command.
    uint8_t  CommandId;  // The command being sent
    uint16_t Reserved;   // reserved for future use
};

const MagicVersion DotnetIpcMagic_V1 = { "DOTNET_IPC_V1" };

The Size field stores the size of the header (= 20) plus the size of the payload (if any). Next comes the CommandSet field that identifies the groups in which the command belongs to:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


enum class DiagnosticServerCommandSet : uint8_t
{
    // reserved = 0x00,
    Dump        = 0x01,
    EventPipe   = 0x02,
    Profiler    = 0x03,
    Process     = 0x04,

    Server      = 0xFF,
};

It is then followed by the ID of a command in that CommandSet :

For example, here is the memory layout of a ProcessInfo command:

Since there is no additional parameter that would need to be encoded in a payload, the Size field is set to 20 (= 0x14 in hexadecimal) which is the size of the header alone.

To make command handling easier, I have defined the corresponding headers:

1
2
3
4
5
6
7
8


const IpcHeader ProcessInfoMessage =
{
    { DotnetIpcMagic_V1 },
    (uint16_t)sizeof(IpcHeader),
    (uint8_t)DiagnosticServerCommandSet::Process,
    (uint8_t)DiagnosticServerResponseId::OK,
    (uint16_t)0x0000
};

All that makes the the code to send such a ProcessInfo command straightforward:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


bool ProcessInfoRequest::Send(HANDLE hPipe)
{
    // send the request
    IpcHeader message = ProcessInfoMessage;
    DWORD bytesWrittenCount = 0;
    if (!::WriteFile(hPipe, &message, sizeof(message), &bytesWrittenCount, nullptr))
    {
        Error = ::GetLastError();
        std::cout << "Error while sending ProcessInfo message to the CLR: 0x" << std::hex << Error << std::dec << "\n";
        return false;
    }

The next step is to read from the pipe to get the answer from the CLR. As mentioned earlier, the same IpcHeader is received; always with the Server (0xFF) CommandSet value. The CommandId field is 0 for a success and 0xFF in case of an error.

1
2
3
4
5
6


enum class DiagnosticServerResponseId : uint8_t
{
    OK = 0x00,
    // future
    Error = 0xFF,
};

For example, here is the memory layout of a ProcessInfo answer:

Note that numbers are little-endian encoded (hence the 0001 for the Size field: it means 0x0100 = 256 bytes)

The code to analyze the response follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


    // analyze the response
    // 1. get the header to know how large the buffer should be to get the payload
    message = {};
    DWORD bytesReadCount = 0;
    if (!::ReadFile(hPipe, &message, sizeof(message), &bytesReadCount, nullptr))
    {
        Error = ::GetLastError();
        std::cout << "Error while getting ProcessInfo response from the CLR: 0x" << std::hex << Error<< std::dec << "\n";
        return false;
    }

    if (message.CommandId != (uint8_t)DiagnosticServerResponseId::OK)
    {
        Error = message.CommandId;
        std::cout << "Error returned by the CLR in ProcessInfo response: 0x" << std::hex << Error<< std::dec << "\n";
        return false;
    }

In case of success, the size of the response payload is obtained from the Size field by subtracting the size of the header. The next step is to allocate a buffer and read the payload from the pipe into that buffer:

1
2
3
4
5
6
7
8
9


    uint16_t payloadSize = message.Size - sizeof(message);
    _buffer = new uint8_t[payloadSize];
    if (!::ReadFile(hPipe, _buffer, payloadSize, &bytesReadCount, nullptr))
    {
        Error = ::GetLastError();
        std::cout << "Error while getting ProcessInfo payload: 0x" << std::hex << Error << std::dec << "\n";
        return false;
    }
    // Note: bytesReadCount == payloadSize

How to access the response fields

Now that the response payload has been read into a memory buffer, it is time to look at the expected fields as described in the documentation. Here is the encoding of the different field types:

To continue with the ProcessInfo example, here are the expected fields:

So here is the memory layout for the response payload for my monitored Windows 64-bit simulator.exe test application:

To simplify the implementation, the class in charge of a command allocates a buffer corresponding to the payload and provides fields with values either copied from it or, in case of strings, pointing to the buffer (just after the 32-bit length):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


bool ProcessInfoRequest::ParseResponse(DWORD payloadSize)
{
    uint32_t index = 0;
    
    memcpy(&Pid, &_buffer[index], sizeof(Pid));
    index += sizeof(Pid);
    if (payloadSize < index) return false;

    memcpy(&RuntimeCookie, &_buffer[index], sizeof(RuntimeCookie));
    index += sizeof(RuntimeCookie);
    if (payloadSize < index) return false;

    PointToString(_buffer, index, CommandLine);
    if (payloadSize < index) return false;

    PointToString(_buffer, index, OperatingSystem);
    if (payloadSize < index) return false;

    PointToString(_buffer, index, Architecture);

    return true;
}

The PointToString helper code is straightforward:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


void PointToString(uint8_t* buffer, uint32_t& index, wchar_t*& string)
{
    // strings are stored as:
    //    - characters count as uint32_t
    //    - array of UTF16 characters followed by "\0"
    // Note that the last L"\0" IS COUNTED 
    uint32_t count;
    memcpy(&count, &buffer[index], sizeof(count));

    // skip characters count 
    index += sizeof(count);

    // empty string case
    // Note: could make it point to the "count" which is 0 in the buffer
    //       instead of returning nullptr
    if (count == 0)
    {
        string = nullptr;
        return;
    }

    string = (wchar_t*)&buffer[index];

    // skip the whole string (including last UTF16 '\0')
    index += count * (uint32_t)sizeof(wchar_t);
}

You now know how to send a command without parameter and analyze the expected response. The next post will show you how to listen to CLR events like dotnet trace.

Resources

Episode 1 — Digging into the CLR Diagnostics IPC Protocol in C#
Diagnostics IPC protocol documentation
Microsoft.Diagnostics.NETCore.Client source code

Troubleshooting CPU and exceptions issues with Datadog toolbox

Thu, 09 Jun 2022 16:09:27 +0000

Introduction

With the new 2.10 release of the Datadog .NET Tracer and Continuous Profiler available, it is time to update some investigation workflows I already introduced. New features have been added to help you diagnose performance issues in your .NET applications:

Linux support!
Code Hotspots: allow you to automatically navigate from lengthy spans and requests to profiles
CPU profiling: pinpoint high CPU consuming methods
Exceptions profiling: identify exceptions distributions
Profile sequence: easily profile an application startup

The goal of this post is to show you how all these features make your investigations easier. I would recommend reading the previous post; especially for the environment setup that I won’t repeat here.

It’s Linux showtime!

The .NET Continuous Profiler is now available for Linux. The only limitation is the presence of glibc 2.18+ in the distribution; for example, CentOS 7 is not supported. Beyond that, we provide features parity between Linux and Windows.

In terms of installation, download the .NET Tracer package that supports your operating system and architecture. Go to the documentation for the additional configuration steps.

From spans to profiles

When analysing lengthy requests, you usually start from looking at the corresponding spans in the APM Traces part of the UI. It is now possible to view the corresponding profiles by clicking the “View Profile” button in the “Code Hotspots” tab:

Before digging into the profiling information, you are already able to see that more than half of the time is spent in Buffer._Memmove that is called by Buggybits ProductsController.Index method:

From the profile view, it is also possible to come back to the traces:

Let’s see now what new features are available at the profiling side.

CPU profiling

The most demanded feature was the ability to analyse CPU consumption (a.k.a. CPU profiling). The idea is to be able to identify code that really consumes CPU usage and optimize it. This is particularly important in the context of cloud-based computing where what you pay is related to the consumed CPU.

In term of implementation, unlike Wall Time profiling, we look at the time spent by a thread on a CPU core and not the elapsed time since the last time we checked (every ~10ms). We also collect the call stack of a thread only if it is currently running on a core. Why? Because we want to only record call stacks corresponding to code paths that are consuming CPU. For example, ThreadPool threads are usually waiting (not interesting call stack) for a work item to process (interesting call stack).

In the last blog post, the code responsible for lengthy requests is doing too many string concatenations (look for += in the following code):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


public IActionResult Index()
{
    var sw = new Stopwatch();
    sw.Start();
    var products = dataLayer.GetAllProducts();
    var productsTable = "";    foreach (var product in products)
    {
        productsTable += $"";    }

    productsTable += "Product NameDescriptionPrice
{product.ProductName}{product.Description}{product.Price}
";
    sw.Stop();

    ViewData["ElapsedTimeInMs"] = sw.ElapsedMilliseconds;
    ViewData["ProductsTable"] = productsTable;
    return View();
}

The Wall Time view was very explicit about ProductsController.Index() calling String.Concat() culprit:

A simple solution is to use a StringBuilder to optimize the concatenations:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


public IActionResult Builder()
{
    var sw = new Stopwatch();
    sw.Start();
    var products = dataLayer.GetAllProducts();
    var productsTable = new StringBuilder(1000 * 80);
    productsTable.Append("");    foreach (var product in products)
    {
        productsTable.Append($"");    }

    productsTable.Append("Product NameDescriptionPrice
{product.ProductName}{product.Description}{product.Price}
");
    sw.Stop();

    ViewData["ElapsedTimeInMs"] = sw.ElapsedMilliseconds;
    ViewData["ProductsTable"] = productsTable;
    return View("Index");
}

The Wall Time view of the profile with the fix does not provide anything useful…

If you want to go deeper, you need to look at the CPU consumption:

The ProductController.Builder method is handling the request (like Index() in the String.Concat case) and calls DataLayer.GetAllProducts() where most of the CPU-related work is done.

Would it be interesting to continue optimizing the code? Notice that GetAllProducts() is “only” consuming 94ms and the other Number.* and String.Concat method around 350ms. So a gain might be neglectable compared to the total ~3 seconds CPU usage

Remember that you should not optimize for the sake of “optimizing”: you should have metrics that tell you when to start (too lengthy request processing) and when to stop.

Exceptions profiling

In the .NET world, exceptions are at the center of errors handling. It is now possible to get a sampled view of the exceptions that happened during an application lifetime; by type:

and by message:

At the implementation level, the new exceptions profiler is notified by the CLR when an exception is thrown. Since in special cases (such as network issue or invalid parsed data for example), an application could trigger thousands of exceptions in a very short period, it is needed to sample them. Otherwise, the impact on performances would be severe; especially if the call stack needs to be rebuilt for each exception.

First, at least one exception per type is kept, ensuring that weird specific exceptions are not lost in the flow. Second, exceptions are sampled over time based on a fixed number of exceptions per profile and the rate of appearance. For knowing the exact number of exceptions, feel free to leverage the Runtime Metrics package as explained in the previous blog post.

Profiling the application bootstrap

In some situations, you are interested in analysing an application bootstrap. In Datadog APM Profile Search UI, it means finding the first profile of the given service execution. However, even with the date and time column, it is not obvious to find the right one:

To help you find the initial profile of a service execution, a new “profile_seq” tag has been added to the HTTP request used to upload the profiles. It contains the count of generated profiles for a given execution of a service, starting from 0.

So now, in the Options of the profile list, add a “profile_seq” column:

The first profile is then easily spottable with a 0 value:

In the future, a more visual hint might be added to identify it without the need to add the column.

Major implementation refactoring

Finally, our implementation has benefited from a large code refactoring. As the previous post explained, the generation of .pprof files and their upload was done in C#. This has been replaced by using a rust library shared amongst different profiler libraries (native, Ruby, .NET).

It means that you should not anymore see these frames in the application call stacks:

It does not mean that one third of the processing has been removed! Just that no more C# code is running with performance gain. First, the managed implementation was allocating objects managed by the garbage collector; adding pressure that might trigger more collections. Second, with the native rust implementation, there is no need to duplicate data between the collecting native part of the continuous profiler and the managed code used to serialize it.

In addition, several optimizations have been done in the symbol’s resolution (i.e., type and method names) part of the code that also reduce memory consumption and CPU usage.

Happy profiling!

References

Datadog Tracer & Continuous Profiler .msi Installer and Linux tar.gz
Datadog Continuous Profiler documentation
Datadog Tracer documentation
Datadog Runtime metrics documentation
Tess Ferrandez repository for BuggyBits labs

Troubleshooting .NET performance issues with Datadog toolbox

Fri, 28 Jan 2022 12:55:40 +0000

Introduction

The beta of Datadog .NET Continuous Profiler is available!

This is a great opportunity to show how to use the different tools provided by Datadog to troubleshoot .NET applications facing performance issues. Tess Ferrandez updated her famous BuggyBits application to .NET Core. Among the different available scenarios, let’s see how to investigate the Lab 4 — High CPU Hang with Datadog. It will be completely different from Tess way: no need to analyze memory dump anymore.

Setup the environment

First, you need to download and run the .msi from our Tracer repository: it will install both the Tracer and the Profiler. The former allows you, among other things, to see how long it takes to process ASP.NET Core requests. The latter is in Beta today and provides wall time duration of your threads (more on this later). Look at the corresponding documentations for the details of enabling tracing and profiling once installed.

Next, ensure that .NET Runtime Metrics are installed for your organization:

Add DD_RUNTIME_METRICS_ENABLED=true environment variable for the application/service you want to monitor. Once enabled for your application, this package allows you to see the evolution of important metrics including some that you won’t find anywhere else such as GC pause time, thread contention time or count of exceptions per type.

Ensure that DogstatsD is setup for the Agent

Looking at the symptoms

In my example, the buggybits application is running under the datadog.demos.buggybits service name. This is how I can filter the related traces in the APM/Traces part of the Datadog portal:

In this screenshot, the Products/Index requests duration is around 6 seconds; which is way too long!

When clicking such a request, the details panel provides the exact URL in the Tags tab:

The Metrics tab shows CPU usage and other few metrics around the trace time:

So, in addition to having slow requests, the CPU usage seems to increase.

It is now time to go to the .NET runtime metrics dashboard and look at what is going on in more details. The first graph that shows up is the number of gen2 collections:

It means that during the 10 minutes test where very few requests are processed, almost 800 gen2 GC are happening every 10s (all runtime metrics are computed every 10 seconds).

The load test corresponding to these requests lasted 10 minutes between 4pm and 6+pm. Each time the requests were processed:

the CPU usage increased
the number of gen2 collections increased
the duration of pauses due to garbage collections increased
the threads contention time increased

In addition to being slow, it seems that the code that processes products/index HTTP requests has also an impact on the CPU (i.e. on the overall application and machine performances).

It would be great if we could see the callstacks corresponding to this processing. This is exactly what the .NET Wall time Continuous Profiler is all about: looking at the duration of methods through a flamegraph representation.

Here comes the profiler

Today, there is no direct way to jump from a trace to the profile containing the callstacks while the corresponding request was processed. We are currently working on this new feature called Code Hotspots.

However, it is easy to use the service name to filter the profiles and select the period of time from APM/Profile Search:

When you click a one-minute profile (the callstacks are gathered and sent every minute), a panel appears with the Performance tab selected. It shows a framegraph on the left and a list on the right.

Getting used to flamegraph

When you look at the wall time flamegraph, you see everything that happened during a single minute:

The previous screenshot highlights the groups of callstacks corresponding to the different threads of execution (from left to right):

the Main() entry point of the application
the code in the CLR responsible for sending counters
the code in the Profiler in charge of generating and sending (the very thin spike) the profile every minute
the Tracer code
the code that listens to the CLR events to generate the runtime metrics
…and the application code that processes the requests!

In the flamegraph, the width of each frame on a row represents the relative time during which the frame was found on a callstack. For example, in our tests, we have 4 threads simply calling Thread.Sleep; one for 10 seconds, one for 20 seconds, one for 30 seconds and a last one for 40 seconds. This is the expected result in a flamegraph (i.e. the widths are consistent with the 1/2/3/4 ratio):

This also applies to CPU-bound threads. For example, if 3 threads are computing the sum of numbers in a tight loop, this is the expected result (i.e. all OnCPUxxx have the same width)

These explanations should stop the fear that started to crawl inside your head about the “visible cost” of the Datadog Tracer and Profiler based on the previous screenshot. The large width of the Datadog threads frames is all about wall time, not CPU time: we are mostly sleeping or waiting but we don’t stop :^)

Investigate the performance issue

The next step is to focus on the stack frames corresponding to the request processing to better understand what is going on.

Basically, you would like to either remove a branch or keep only a branch. You simply have to move the mouse over a frame (i.e. ThreadPoolWorkQueue in the previous screenshot) and click the three dots that just appeared. Next, select Show From to keep only that branch in the flamegraph:

Now, scroll-down into the flamegraph and the flow of execution corresponding to processing the Products/Index request becomes more visible:

It seems that the Index() method of the ProductsController is spending most of its time calling String.Concat().

Let’s have a look at the source code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


// GET: Products
public IActionResult Index()
{
    var products = dataLayer.GetAllProducts();
    var productsTable = "";    foreach (var product in products)
    {
        productsTable += $"";    }
    productsTable += "Product NameDescriptionPrice
{product.ProductName}{product.Description}{product.Price}
";

    ViewData["ProductsTable"] = productsTable;
    return View();
}

But still no sign of String.Concat()… Well, this is because the C# compiler is hiding it from you with the += syntaxic sugar. Let’s have a look at the decompiled code as shown by IlSpy (without the string.Concat transformation):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


public Microsoft.AspNetCore.Mvc.IActionResult Index()
{
    var products = dataLayer.GetAllProducts();
    string productsTable = "";    var enumerator = products.GetEnumerator();
    try
    {
        while (enumerator.MoveNext())
        {
            BuggyBits.Models.Product product = enumerator.Current;
            string[] array = new string[8];
            array[0] = productsTable;
            array[1] = "";            productsTable = string.Concat(array);
        }
    }
    finally
    {
        ((System.IDisposable)enumerator).Dispose();
    }
    productsTable = string.Concat(productsTable, "Product NameDescriptionPrice
";
            array[2] = product.ProductName;
            array[3] = "";
            array[4] = product.Description;
            array[5] = "";
            array[6] = product.Price;
            array[7] = "
");
    base.ViewData["ProductsTable"] = productsTable;
    return View();
}

So now we can see the call to string.Concat() at the end of the while loop iteration.

Behind the scene, since a string object is immutable, string.Concat() will create a new string each time it is called and the previous string referenced by productsTable is no more rooted and will put more pressure on the GC. If I’m telling you that datalayer.GetAllProducts() returns 10.000 products, it means that string.Concat gets called 10.000 times.

As the string grows, it will reach the 85000 bytes limit and start to be allocated in the LOH, adding more pressure on GC that will trigger gen2 collections; hence the high number of gen2 collections seen in the runtime metrics dashboard.

Note that if the native frames were visible in the flamegraph (by the way, let me know if this is a feature that would make sense to add), you would see the methods of the CLR responsible for the GC.

Look at Tess Ferrandez post for a possible solution to this expensive code pattern (i.e. calling string.Contact in a large tight loop)

Different types of filters

Before leaving, I would like to quicky talk about the list shown on the right hand-side of the UI.

It allows you to see a wall time summary per method name, namespace (i.e. sum of methods from types in the same namespace, thread ID (i.e. internal Datadog unique ID), thread name or AppDomain Name.

First, the only methods listed here are “leaf” methods: they appear at the top of at least one callstack. If you would like to visually see some specific frames, you should use the filter box:

All other frames are faded out.

Second, the list is sorted with the largest wall time at the top: this could be an easy way to spot method “expensive” in term of CPU (i.e. will frequently be running so appear at the top of the stack). You simply need to skip the wait and sleep related methods like shown in the previous screenshot: String.Concat and Buffer._Memmove (used by string.Concat) were just in front of your eyes!

When you select an element of the list, the flamegraph is updated accordingly: only the callstacks containing this element will be visible (it could speed up the filtering process)

References

Datadog Tracer & Continuous Profiler .msi Installer
Datadog Continuous Profiler documentation
Datadog Tracer documentation
Datadog Runtime metrics documentation
Tess Ferrandez repository for BuggyBits labs

Decyphering methods signature with .NET profiling APIs

Tue, 12 Oct 2021 13:23:15 +0000

Introduction

After introducing the CLR profiling API by tracing managed methods calls, then dealing with assemblies and types, it is time to look at methods signatures. Remember that the starting point is the FunctionID received by the Enter callback each time a method is executed.

The question answered by this post is how to build the signature of the method given a FunctionID.

A method signature is built from its return value (or void), its name and a list of parameters. All these details are stored in the module metadata generated by the C# compiler. So the first step is to get the metadata token corresponding to a FunctionID thanks to ICorProfilerInfo::GetFunctionInfo:

1
2


mdToken token;
HRESULT hr = pInfo->GetFunctionInfo(functionId, &classId, &moduleId, &token);

Next, use the IMetaDataImport corresponding to the module to call GetMethodProps and pass the function metadata token:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


mdTypeDef type;
WCHAR name[260];
ULONG size;
ULONG attributes;
PCCOR_SIGNATURE pSig;
ULONG blobSize;
ULONG codeRva;
DWORD flags;
hr = pMetaDataImport->GetMethodProps(
        token, &type, name, ARRAY_LEN(name) - 1, &size, &attributes, &pSig, &blobSize, &codeRva, &flags);
In addition to the function name, you will be able to check the attributes to figure out if the function is static or not:
if ((attributes & mdStatic) == mdStatic)
{
   oss << " static ";
}

The return type and parameters type of the function are encoded in a binary format defined in the ECMA-335 specification. This binary blob is pointed to by the pSig parameter. Hopefully, you don’t have to implement a blob signature parser yourself. This has been done my Rico Mariani or Peter Sollich and it relies on low level helpers from cor.h such as CorSigUncompressData.

Here is an example of a signature blob for a non-static method returning void and accepting a float and a double as parameters:

The well-known types are encoded and available as ELEMENT_TYPE_xxx constants from corhdr.h.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


ELEMENT_TYPE_VOID,    "Void",          
ELEMENT_TYPE_BOOLEAN, "Boolean",       
ELEMENT_TYPE_CHAR,    "Char",          
ELEMENT_TYPE_I1,      "SByte",         
ELEMENT_TYPE_U1,      "Byte",          
ELEMENT_TYPE_I2,      "Int16",         
ELEMENT_TYPE_U2,      "UInt16",        
ELEMENT_TYPE_I4,      "Int32",         
ELEMENT_TYPE_U4,      "UInt32",        
ELEMENT_TYPE_I8,      "Int64",         
ELEMENT_TYPE_U8,      "UInt64",        
ELEMENT_TYPE_R4,      "Single",        
ELEMENT_TYPE_R8,      "Double",        
ELEMENT_TYPE_STRING,  "String",        
ELEMENT_TYPE_I,       "IntPtr",        
ELEMENT_TYPE_U,       "UIntPtr",       
ELEMENT_TYPE_OBJECT,  "Object",

For custom types identified as ELEMENT_TYPE_CLASS for reference types or ELEMENT_TYPE_VALUETYPE for value types, the metadata token of the type is “compressed” as part of the signature (see CorSigUncompressToken in cor.h for implementation details). If the type is defined in the same assembly as the method, you get a TypeDef token (starting with 02) used to call IMetaDataImport::GetTypeDefProps. If not, it will be a TypeRef token (starting with 01) used to call IMetaDataImport::GetTypeRefProps.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


case ELEMENT_TYPE_CLASS:
case ELEMENT_TYPE_VALUETYPE:
{
   mdToken token;
   char classname[260];

   classname[0] = '\0';
   signature += CorSigUncompressToken(signature, &token);
   if (typeToken != NULL)
   {
      *typeToken = token;
   }

   HRESULT hr;
   WCHAR zName[260];
   if (TypeFromToken(token) == mdtTypeRef)
   {
      mdToken resScope;
      ULONG length;
      hr = pMDImport->GetTypeRefProps(token,
         &resScope,
         zName,
         260,
         &length);
   }
   else
   {
      hr = pMDImport->GetTypeDefProps(token,
         zName,
         260,
         NULL,
         NULL,
         NULL);
   }
…

Generic methods have more complicated signatures to compute

The following figure shows how to handle generic methods:

The main change for such a generic method is that, in the signature blob, you will get the number of generic arguments just after the total parameters count and the first “calling convention” data will be IMAGE_CEE_CS_CALLCONV_GENERIC. The other difference is how to deal with generic parameters in the blob that will all share the same ELEMENT_TYPE_MVAR value followed by a position (starting from 0). This is the position in the array returned by ICorProfilerInfo2::GetFunctionInfo2 for the ClassID.

The final code should look like the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36


DWORD bIsValueType;
ULONG currentGenericParam = 0;
for (ULONG i = 0;
   (SUCCEEDED(hr) && (pSigBlob != NULL) && (i < (argCount)));
   i++)
{
   // get the parameter name
   hr = pMetaData->GetParamProps(paramDefs[i], NULL, &pos, name, ARRAY_LEN(name)-1, &length, &attributes, &bIsValueType, NULL, NULL);
   // note that we need to convert from WCHAR* to char* for the name

   // get the parameter type
   buffer[0] = '\0';

   // in case of generic function, get the type details from the runtime and not from the metadata
   // !! we don't know in advance which parameter is a generic parameter and this is given by elementType == ELEMENT_TYPE_MVAR
   pSigBlob = ParseElementType(pMetaData, pSigBlob, classTypeArgs, methodTypeArgs, &elementType, buffer, ARRAY_LEN(buffer) - 1);
   if ((methodTypeArgs != NULL) && (elementType == ELEMENT_TYPE_MVAR))
   {
      ModuleID moduleId;
      mdTypeDef mdType;
      hr = pInfo->GetClassIDInfo2(methodTypeArgs[currentGenericParam], &moduleId, &mdType, NULL, 0, NULL, NULL);
      if (SUCCEEDED(hr))
      {
         WCHAR paramTypeName[260];
         IMetaDataImport2* pImport2;
         hr = pInfo->GetModuleMetaData(moduleId, ofRead, IID_IMetaDataImport2, reinterpret_cast<IUnknown**>(&pImport2));
         ULONG sigBlobLen = 0;

         // NOTE: get elementType from type name because the metadata can't give us the instanciated generic in the blob signature
         hr = pImport2->GetTypeDefProps(mdType, paramTypeName, ARRAY_LEN(paramTypeName)-1, 0, NULL, NULL);
         pImport2->Release();
      }

      currentGenericParam++;
   }
}

One last detail before looking for the parameters value in the next post: for ALL reference types, the same implementation is generated by the JIT compiler. If you think about it, there is no need to implement a List in a different way than a List of any other reference type: they all deal with references (i.e. addresses). The name picked by the CLR team to identify this “generic reference type” is System.__Canon that stands for “canonical”. So expect to receive that type name a lot!

References

Start a journey into the .NET Profiling APIs

Sat, 07 Aug 2021 12:21:39 +0000

Introduction

When I want to dig into a new API, I implement a real world scenario. This is exactly what I did for the .NET native profiling API. I want to know how to get parameters and return value of any method call during any .NET application life. The expected result would be something like :

Enter PublicClass::ClassParamReturnClass
this = 0x6f97e190 (8)
ClassType obj = 0x6f97e488 (8)
| int32 k__BackingField = 84
| int32 intField = 42
| String stringField = 43
ClassType obj = 0x0000023A475BBAD8

Leave PublicClass::ClassParamReturnClass
| int32 k__BackingField = 170
| int32 intField = 85
| String stringField = 86
returns 0x0000023A475BBBB0

when the following method is executed:

1
2
3
4


public ClassType ClassParamReturnClass(ClassType obj)
{
   return new ClassType(obj.IntProperty + 1);
}

You will have to write native C/C++ code to leverage the .NET Profiling API as John Robbins explained in his 2003 Debugging Applications book. Even though Microsoft is providing a code sample for that, it does just show how to get notified when a function is called or exited but nothing about its type, its name, what are its parameters value and its return value. This series will detail both the .NET profiling API and the metadata API (i.e. the native reflection API).

Let’s start with the basics of .NET Profiling. As shown in the following figure from the Microsoft Profiling documentation, with the right environment configuration, the CLR will load a COM-like object implementing ICorProfilerCallback interface to notify almost everything happening in a .NET application from startup to shutdown.

Since .NET was launched, more and more notifications have been added by versioning the interface up to ICorProfilerCallback9. Welcome to the usual COM world! I recommend that you watch Pavel Yosifovich session about writing a CLR Profiler in an hour to get an overview. I will use his sample solution as a starting point.

For performance sake, you tell the runtime which events you are interested in; i.e. which ICorProfilerCallback functions will be called and you simply have to return S_OK from all other functions. This setup is done in your implementation of ICorProfilerCallback::Initialize. This first function called by the CLR provides a parameter from which you need to QueryInterface a version of ICorProfilerInfo interface (the current one is ICorProfilerInfo10). This interface provides functions to query information about parameters passed to your ICorProfilerCallback functions (such as AppDomain, assembly, type, function, thread and so on).

The first ICorProfilerInfo function you will use is SetEventMask to filter the ICorProfilerCallback functions that will be called by the runtime. It accepts a flag combination of values from the COR_PRF_MONITOR enumeration.

Assembly code is needed for Enter/Leave/TailCall

To get notified when a managed method is called or exits, you should pass:

COR_PRF_MONITOR_ENTERLEAVE | 
COR_PRF_ENABLE_FUNCTION_ARGS | 
COR_PRF_ENABLE_FUNCTION_RETVAL | 
COR_PRF_ENABLE_FRAME_INFO

to ICorProfilerInfo::SetEventMask. The first flag tells the runtime to call static callbacks (i.e. not exposed as functions of ICorProfilerCallback) when a managed method gets executed or returns. The other three flags ensure that these callbacks will receive enough information to extract method arguments and return value.

Unlike the other notifications that end up calling your ICorProfilerCallback functions, you need to register three special callbacks to the runtime via ICorProfilerInfo3::SetEnterLeaveFunctionHooks3WithInfo. For performance reasons, the .NET team asks you to write the prolog and epilog (i.e. saving/restoring CPU registers on/from the stack) yourself in assembly code instead of relying on well-defined calling conventions supported by the C/C++ compiler.

This is why you have to call ICorProfilerInfo3:: SetEnterLeaveFunctionHooks3WithInfo and pass pointers to these “naked” functions. The Microsoft ELTProfiler sample implements the stubs both for x86 (as inlined assembly embedded in a C++ file) and x64 (defined in .asm file). In x64, you need to update your project file to add the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


 Label = "ExtensionSettings">
    Project = "$(VCTargetsPath)\BuildCustomizations\masm.props" / >


 Include = "../DotNext.Profiler.Shared/asm/windows/nakedcallbacks.asm"
	Condition = "'$(Platform)' == 'x64'" / >

 Label = "ExtensionTargets">
     Project = "$(VCTargetsPath)\BuildCustomizations\masm.targets" / >

The nakedcallbacks.asm file contains the assembly code to call stub functions wrapped by the expected prolog and epilog written in assembly code. Here is the signature of the functions from where you will be able to start working in C++:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


PROFILER_STUB EnterStub(FunctionIDOrClientID functionId, COR_PRF_ELT_INFO eltInfo)
{
   ...
}

PROFILER_STUB LeaveStub(FunctionID functionId, COR_PRF_ELT_INFO eltInfo)
{
   ...
}

PROFILER_STUB TailcallStub(FunctionID functionId, COR_PRF_ELT_INFO eltInfo)
{
   ...
}

A function is identified by a FunctionID and this is where you start your adventure (I will come back later to the COR_PRF_ELT_INFO parameter). Note that for the EnterStub function, you need to get the FunctionID from the FunctionIDOrClientID.functionId field.

Once your hook callbacks have been registered via ICorProfilerInfo3:: SetEnterLeaveFunctionHooks3WithInfo, it is still possible to decide whether or not a managed method call should trigger them. For that, it is needed to register a “mapper” function that is called once per managed method with one of these functions:

HRESULT ICorProfilerInfo::SetFunctionIDMapper([in] FunctionIDMapper* pFunc); with UINT_PTR __stdcall Mapper(FunctionID functionId, BOOL* pHookFunction)
HRESULT ICorProfilerInfo3::SetFunctionIDMapper2([in] FunctionIDMapper2* pFunc, [in] void* clientData); with UINT_PTR __stdcall Mapper2(FunctionID functionId, void* clientData, BOOL* pHookFunction)

The latter allows you to pass some “client data” to the mapper function such as a helper class to manipulate the received FunctionID or your profiler state.

If pHookFunction is set to TRUE, your enter/leave functions will be called and the returned UINT_PTR will be passed as the FunctionID parameter. This allows you to handle function name or signature computation at one single place outside of the real profiling work done each time a method is called. If pHookFunction is set to FALSE, the enter/leave functions will never be called for that FunctionID. The mapper callback is called only once per FunctionID: this could be a good way to avoid performance impact if you just want to profile a small subset of methods.

How to debug your profiler

Before going any further, it is needed to know how to debug your C++ profiler code with Visual Studio. The first step is to write a simple .NET application to execute the method calls you want to intercept. The second natural step would be to setup the environment variables needed to inject your profiler:

and also check the Enable native code debugging option:

If you start a debug session, the Visual Studio debugger with use both managed and native debugging APIs. Unfortunately, the managed debugging API does not allow breakpoints sets in ICorProfilerCallback functions.

Instead, you need to Debug | Start Without Debugging (CTRL+F5), not Debug | Start Debugging (F5) the C# test application with the same environment variables and attach the debugger via Debug | Attach to Process. Select the process and click Select button in the Attach to section:

To make attachment simple, add a Console.ReadLine in a console application before starting the method calls to test.

The next post will show you how to extract information from a FunctionID.

References

Writing a .NET Core cross platform profiler in an hour video and the corresponding source code from Pavel Yosifovich
Microsoft Profiling documentation
Microsoft Enter/Leave code sample

Memory Anti-Patterns in C#

Thu, 01 Jul 2021 13:56:00 +0000

In the context of helping the teams at Criteo to clean up our code base, I gathered and documented a few C# anti-patterns similar to Kevin’s publication about performance code smell. Here is an extract related to good/bad memory patterns.

Even though the garbage collector is doing its works out of the control of the developers, the less allocations are done, the less the GC will impact an application. So the main goal is to avoid writing code that allocates unnecessary objects or references them too long.

Finalizer and IDisposable usage

Let’s start with a hidden way to referencing an object: implementing a “finalizer”. In C#, you write a method whose name is the name of the class prefixed by ~. The compiler generates an override for the virtual Object.Finalize method. An instance of such a type is treated in a particular way by the Garbage Collector:

after it is allocated, a reference is kept in a** Finalization** internal queue
after a collection, if it is no more referenced, this reference is moved into another fReachable internal queue and treated as a root until a dedicated thread calls its finalizer code

As Konrad Kokosa details in one of his free GC Internals video, instances of a type implementing a finalizer stay much longer in memory than needed; waiting for the next collection of the generation in which the previous collection left it (i.e. gen1 if it was in gen0 or even worse, gen2 if it was in gen1).

So the first question people are usually asking is: do you really need to implement a finalizer? Most of the time, the answer should be no. The code of a finalizer is responsible for cleaning up ONLY resources that are NOT managed. It usually means “stuff” received from COM interop or P/Invoke calls to native functions such as handles, native memory or memory allocated via Marshal helpers. If your class has IntPtr fields, it is a good sign that their lifetime finishes in a finalizer via Marsal helpers or P/Invoke cleanup calls. Look for SafeHandle-derived class if you need to manipulate kernel object handles instead of raw IntPtr and avoiding finalizers. So in 99.9% of the cases, you don’t need a finalizer.

The second question is how implementing a finalizer relates to implementing IDisposable? Unlike a finalizer, implementing the unique Dispose() method of IDisposable interface in a class means nothing for the Garbage Collector. So there is no side effect to extend the lifetime of its instances. This is only a way to allow the users of instances of this class to explicitly cleanup such an instance at a certain point in time instead of waiting for a garbage collection to be triggered.

Let’s take an example: when you want to write to a file, behind the scene, .NET will call native APIs that operate on real file (via kernel object handles on Windows) with limited concurrent access (i.e. two processes can’t corrupt a file by writing different things at the same time — this is a very high level view of the situation but valid enough for this discussion). Another class would allow access to databases via a limited number of connections that should be released as soon as possible. In all these scenarios, as a user of these classes, you want to be able to “release” the resources used behind the scene as quickly as possible when you don’t need to access them anymore. This translates into the well known **using **pattern in C#:

1
2
3
4


using (var disposableInstance = new MyDisposable())
{
   DoSomething(disposableInstance);
}; // the instance will be cleanup and its resources released

that is transformed by the C# compiler into:

1
2
3
4
5
6
7
8
9


var disposableInstance = new MyDisposable();
try
{
   DoSomething(disposableInstance);
}
finally
{
   disposableInstance?.Dispose();
}

So when should you implement IDisposable? My answer is simple: when the class owns fields of classes that implement IDisposable and if it implements a finalizer (for the good reasons already explained). Don’t use IDisposable.Dispose for other reasons such as logging (like what we used to do in C++ destructor): prefer to implement another explicit interface dedicated to that purpose.

In term of implementation, I have to say that I never understood why Microsoft decided to provide such a confusing implementation in its documentation. You have to implement the following method to “free” unmanaged and managed resources. It should be called by both the finalizer and IDisposable.Dispose():

1
2
3
4


protected virtual void Dispose(bool disposing)
{
   // free unmanaged and managed resources
}

You also need to have a _disposed field to allow IDisposable.Dispose() to be called more than once without problem. In all methods and properties of the class, don’t forget to throw an ObjectDisposedException if _disposed is true to catch usage of already disposed objects.

Ask a group of developers when disposing should be true or false: half will say when called from the finalizer and the other half from Dispose (and I’m not counting those who are not sure). Why giving the same name to the method that already exists in IDisposable? Why picking “disposing” as parameter name? I don’t think it could been possible to find a more confusing solution: too many “dispose” kills the pattern…

Here is my own implementation that does exactly the same thing but with much less confusion:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


class DisposableMe : IDisposable
{
    private bool _disposed = false;

    // 1. field that implements IDisposable
    // 2. field that stores "native resource" (ex: IntPtr)

    ~DisposableMe()
    {
        Cleanup("called from GC" != null);
    }           // = true

    public void Dispose()
    {
        Cleanup("not from GC" == null);
    }           // = false
    
    ...
}

I also rename Dispose(bool disposing) into Cleanup(bool from GC):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


 private void Cleanup(bool fromGC)
 {
     if (_disposed)
         return;

     try
     {
         // always clean up the NATIVE resources
         if (fromGC)
             return;

         // clean up managed resources ONLY if not called from GC
     }
     finally
     {
         _disposed = true;

         if (!fromGC)
             GC.SuppressFinalize(this);
     }
 }

The rules you have to keep in mind are simple:

native resources (i.e. IntPtr fields) must always be cleaned up
managed resources (i.e. IDisposable fields) should be disposed when called from Dispose (not from GC)

The _disposed boolean field is used to cleanup resources only once. In this implementation, it is set to true even if an exception happens because I’m assuming that if it just happened, it will also happen if called another time.

Last but not least, the call to GC.SuppressFinalize(this) simply tells the GC to remove the disposed object from the Finalization internal queue:

it is only meaningful when called from Dispose (not from GC) to avoid extending its lifetime.
it means that the finalizer will never be called. If it were, it would have called Cleanup that would have returned immediately because _disposed is true.

The rest of the post describes typical anti-patterns. However, as usual with performance related topic, remember that the impact might not be noticeable if it does not run in a hot path. Always balance between readability/ease of maintenance/understanding and performance gain.

Provide list capacity when possible

It is recommended to provide a capacity when creating a List or a collection instance. The .NET implementation of such classes usually stores the values in an array that need to be resized when new elements are added: it means that:

A new array is allocated
The former values are copied to the new array
The former array is no more referenced

In the following example, the capacity of resultList is otherList.Count

1
2
3
4
5


var resultList = new List<...>();
foreach (var item in otherList)
{ 
   resultList.Add(...);
}

Prefer StringBuilder to +/+= for string concatenation

Creating temporary objects will increase the number of garbage collections and impact performances. Since the string class is immutable, each time you need to get an updated version of a string of characters, the .NET framework ends up creating a new string.

For string concatenation, avoid using Concat, + or +=. This is especially important in loop or methods called very often. For example in the following code, a StringBuilder should be used:

1
2
3
4
5
6


var productIds = string.Empty;
while (match.Success)
{
   productIds += match.Groups[2].Value + "\n";
   match = match.NextMatch();
}

Again in loops, avoid creating temporary string such as in the following code where SearchValue.ToUpper() do not change in the loop:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


if (SelectedColumn == Resources.Journaux.All && !String.IsNullOrEmpty(SearchValue))
    source = model.DataSource.Where(x => x.ItemId.Contains(SearchValue)
        || x.ItemName.ToUpper().Contains(SearchValue.ToUpper())
        || x.ItemGroupName.ToUpper().Contains(SearchValue.ToUpper())
        || x.CountingGroupName.ToUpper().Contains(SearchValue.ToUpper()));
 
 
if (SelectedColumn == Resources.Journaux.ItemNumber)
    source = model.DataSource.Where(x => x.ItemId.ToUpper().Contains(SearchValue.ToUpper()));
 
 
if (SelectedColumn == Resources.Journaux.ItemName)
    source = model.DataSource.Where(x => x.ItemName.ToUpper().Contains(SearchValue.ToUpper()));
 
 
if (SelectedColumn == Resources.Journaux.ItemGroup)
    source = model.DataSource.Where(x => x.ItemGroupName.ToUpper().Contains(SearchValue.ToUpper()));
 
if (SelectedColumn == Resources.Journaux.CountingGroup)
    source = model.DataSource.Where(x => x.CountingGroupName.ToUpper().Contains(SearchValue.ToUpper()));

The effect is even worse due to the Where() clause that create a new temporary upper string for each element of the sequence!

This recommendation also applies to types that provides string-based direct access to characters such as in the following code:

1

if (!uriBuilder.ToString().EndsWith(".", true, invCulture))

where ToString() is not needed because it is possible to directly access the last character:

1

if (uriBuilder[uriBuilder.Length - 1] != '.')

Caching strings and interning

Prefer static cache of read-only objects to recreating them in each call such as in the following example:

1
2
3


var allCampaignStatuses = 
   ((CampaignActivityStatus[])Enum.GetValues(typeof(CampaignActivityStatus)))
   .ToList();

(Replace by a static list since the enumeration elements won’t change)

Last but not least, when string keys (with only a few different values) are used, you could “intern” them (i.e. ask the CLR to cache a value and always return the same reference). Read the corresponding Microsoft Docs for more details.

Don’t (re)create objects

The first pattern to use is the static classes with static methods to avoid the creation of temporary objects just to call fields-less methods. It is also recommended to pre-compute read-only list instead of re-creating it each time a method gets called like in the following example :

1
2
3
4


var allCampaignStatuses = 
   ((CampaignActivityStatus[])Enum.GetValues(typeof(CampaignActivityStatus)))
   .ToList();
   // use allCampaignStatuses in the rest of the method

This list could have been computed once as a static field of the class because the enumeration will not change during the application lifetime.

Avoid repeated calls and keep values in local variables when used in a loop; this is particularly easy to forget when dealing with string ToLower() and ToUpper().

1
2
3
4
5
6
7


var found = elements.Any(
// ToLower() is called in each test
k => string.Compare(
       k.ToLower(), 
       key.ToLower(), 
       StringComparison.OrdinalIgnoreCase
       ) == 0);

(a new temporary string will be created by key.ToLower() by each test)

Prefer String.Compare(…, StringComparison.OrdinalIgnoreCase) to avoid calling ToLower()/ToUpper() just for string comparison such as in the following example:

1

if (transactionIdAsString != null && transactionIdAsString.ToLowerInvariant() == "undefined")

becomes:

1

if (transactionIdAsString != null && string.Compare(transactionIdAsString, "undefined", StringComparison.OrdinalIgnoreCase) == 0)

Best practices with LINQ

The LINQ syntax is used extensively all over the source code. However, several patterns are found very often and might impact overall performance.

Prefer IEnumerable to IList

Most of the methods are iterating on sequences represented by IEnumerable either via foreach() or thanks to System.Linq.Enumerable extension methods. IList should be used only when sequence modification is required:

It is also recommended to use IEnumerable instead of IList as method parameters if there is no need to add/remove elements to the sequence. That way, the client code don’t have to use ToList() before calling the method. The same comment applies to return types that should be IEnumerable rather than IList because most of the time, the sequence will simply be iterated via a foreach statement.

FirstOrDefault and Any are your friends… but might not be needed

First, there is no need to call Any (or even worse ToList().Count > 0) before foreach such as in the following code:

1
2
3
4
5


if (sequence != null && sequence.Any())
{
   foreach (var item in sequence)
   ...
}

Avoid unnecessary ToList()/ToArray() calls

LINQ queries are supposed to defer their execution until the corresponding sequence is iterated such as with a **foreach **statement. This is also the case when ToList() or ToArray() are called on such a query:

1
2
3
4
5
6
7
8
9


var resourceNames = resourceAssembly
.GetManifestResourceNames()
.Where(r => r.StartsWith($"{resourcePath}.i18n"))
.ToArray();

foreach (var resourceName in resourceNames)
{
   ...
}

The ToList() method builds a List<> instance that contains all elements of the given sequence. It should be used carefully because the cost of creating a list from a large sequence of objects could be high both in term of memory and performance due to the implementation of element addition in List<>.

The only recommended usages are:

optimization sake to avoid executing the underlying query several times when it is expensive
removing/adding elements from a sequence
storing the result of a query execution in a class field

However, most of the times, you don’t need to call ToList() to iterate on a IEnumerable. If you do so, you hurt the runtime execution both in term of memory consumption (because of the unneeded List that is just temporary) and in term of performance because the sequence gets iterated twice.

The base of LINQ to Object is the IEnumerable interface used to iterate on a sequence of objects. All LINQ extension methods are taking IEnumerable instances as parameter in addition to foreach constructs. It is also not needed to call ToList() when an IEnumerable is expected (this is a good reason to prefer IEnumerable to IList/List/[] in method signatures)

Some methods are calling ToList() before Where clauses are applied to an IEnumerable sequence: it is more efficient to stack the Where clauses and call ToList() at the end.

Last but not least, it is not needed to call ToList() to get the number of elements in a sequence such as in the following code sample:

1
2
3
4
5


productInfos
  .Select(p => p.Split(DisplayProductInfoSeparator)[0])
  .Distinct()
  .ToList()
  .Count;

becomes:

1
2
3
4


productInfos
  .Select(p => p.Split(DisplayProductInfoSeparator)[0])
  .Distinct()
  .Count();

Prefer IEnumerable<>.Any to List<>.Exists

When manipulating IEnumerable, it is recommended to use Any instead of ToList().Exists() such as in the following code:

1

if (sequence.ToList().Exists(…))

becomes:

1

if (sequence.Any(...))

Prefer Any to Count when checking for emptiness

The **Any **extension methods should be preferred to count computation on IEnumerable because the iteration on the sequence stops as soon as the condition (if any) is fulfilled without allocating any temporary list:

1
2
3
4
5


var nonArchivedCampaigns = 
   campaigns
   .Where(c => c.Status != CampaignActivityStatus.Archived)
   .ToList();
if (nonArchivedCampaigns.Count == 0)

becomes:

1

if (!campaigns.Where(c => c.Status != CampaignActivityStatus.Archived).Any())

Note that it is also valid to use if (!campaigns.Any(filter))

Order in extension methods might matter

When operators are applied to sequences (i.e. IEnumerable), their order might have an impact on the performance of the resulting code. One important rule is to always filter first so the resulting sequences get smaller and smaller to iterate. This is why it is recommended to start a LINQ query by Where filters.

With LINQ, the code you write to define a query might be misleading in term of execution. For example, what is the difference between:

1
2
3
4


var filteredElements = sequence
  .Where(first filter)
  .Where(second filter)
  ;

and:

1
2
3


var filteredElements = sequence
  .Where(first filter && second filter)
  ;

It depends on the query executor. For LINQ for Objects, it seems that there is no difference in term of the filters execution: the first and second filters will be executed the same number of times as shown by the following code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


 var integers = Enumerable.Range(1, 6);
 var set1 = integers
 .Where(i => IsEven(i))
 .Where(i => IsMultipleOf3(i));

 foreach (var current in set1)
 {
     Console.WriteLine($"--> {current}");
 }

 Console.WriteLine("--------------------------------");

 var set2 = integers
 .Where(i => IsEven(i) && IsMultipleOf3(i))
 ;

 foreach (var current in set2)
 {
     Console.WriteLine($"--> {current}");
 }

When you run it, you get the exact same lines in the console:

IsEven(1)
IsEven(2)
   IsMultipleOf3(2)
IsEven(3)
IsEven(4)
   IsMultipleOf3(4)
IsEven(5)
IsEven(6)
   IsMultipleOf3(6)
--> 6
--------------------------------
IsEven(1)
IsEven(2)
   IsMultipleOf3(2)
IsEven(3)
IsEven(4)
   IsMultipleOf3(4)
IsEven(5)
IsEven(6)
   IsMultipleOf3(6)
--> 6

However, when you run it under Benchmark.NET,

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42


 private int[] _myArray;

 [Params(10, 1000, 10000)]
 public int Size { get; set; }

 [GlobalSetup]
 public void Setup()
 {
     _myArray = new int[Size];

     for (var i = 0; i < Size; i++)
         _myArray[i] = i;
 }

 [Benchmark(Baseline = true)]
 public void Original()
 {
     var set = _myArray
         .Where(i => IsEven(i))
         .Where(i => IsMultipleOf3(i))
         ;

     int i;
     foreach (var current in set)
     {
         i = current;
     }
 }

 [Benchmark]
 public void Merged()
 {
     var set = _myArray
         .Where(i => IsEven(i) && IsMultipleOf3(i))
         ;

     int i;
     foreach (var current in set)
     {
         i = current;
     }
 }

the results are significantly better for the single “merged” Where clause:

After looking at the implementation in the .NET Framework with my colleague Jean-Philippe, the additional cost seems to be related to the underlying IEnumerator corresponding to the first Where.

Remember to never assume and always measure.

Interesting in joining the team? Check out our latest job posts:

Senior Site Reliability Engineer - PRE Team (remote flexibility with base in France) job in Paris careers.criteo.com Senior Site Reliability Engineer - PRE - Performance (remote flexibility with base in France) careers.criteo.com

How to ease async callstacks analysis in Perfview

Tue, 02 Mar 2021 09:35:37 +0000

Introduction

In the previous post, I described why you might get weird reversed callstacks in Visual Studio when analyzing or debugging async/await code. And if you are using Perfview to profile the same application, you should also get the same reverse continuation flow:

The rest of the post describes how to easily profile with Perfview and more interestingly, how to leverage grouping/folding features to get much more readable asynchronous callstacks.

Perfview 101

Here are the different steps to get the previous tree-like representation of a profiling session results.

First, start a data collection by clicking the Start Collection button from the Collect | Collect dialog box and check Kernel Base, CPU Samples, and .NET boxes:

Stop the collection when the application ends and double-click the CPU Stacks node :

After selecting the application in the Select Process Window

click the CallTree tab:

Before entering the dreaded yellow/white CPU Stacks window, let’s spend some time detailing its vast toolbar in the following figure:

The most powerful elements are the *Pats combo-boxes. Each of them supports a “simple” matching pattern syntax for different purposes (don’t worry, you will see how to use them in many more examples later):

GroupPats: merge sibling matching frames into one.
FoldPats: matching frames are folded into parent frame.
IncPats: non matching frames are removed (used for process filtering for example).
ExcPats: matching frames are excluded.

Let’s see what we get with all combo-box set as empty for the CallTree tab:

For server applications, we are usually not interested in making any difference between threads so it would be nice to group all threads under a single AllThreads node. This is exactly what the first choice of the GroupPats combo-box provides:

The effect is simple: all lines at the same level containing “Thread” in the text are merged into a new line with “AllThreads” as new text

You can now get the same kind of tree based representation as in Visual Studio: the difference is that you need to open each node by clicking a checkbox (or right-click + Expand All to see the whole tree)

Most columns meaning are quite self-explicit except maybe the When column that provides the CPU usage graph over time in a “textual” way:

The CallTree representation obviously displays the frames sorted by Inc% column.

Going further with Perfview

When expanding the calltree, you usually get lost in the async/await implementation details. The following screenshot shows the signal/noise ratio for my simple test application where we don’t really care about the blue lines!

This is where the different Perfview combo-boxes are coming to the rescue. Some frames and their children are clearly not interesting such as the last two coreclr!ThePreStub. In that case, select the frame and copy the text from the status bar (yes: this is possible and so handy!)

and paste it into the ExcPats combo-box

to make the corresponding frames disappear.

Unfortunately, you can’t do the same for the other Task-related frames because these ExcPats matching frames are completely removed with their children where your async method calls appear.

Folding patterns are your friends

This time, the FoldPats combo-box will be your friend: each frame that maps one of its ; separated substring will disappear and its occurrence count will be added to its parent frame. Since all these Task-related frame do not appear a lot, the impact in the Inc/Exc columns of the parent frames should be minimal. After I used the following substrings:

Tasks.Task+DelayPromise.CompleteTimedOut(;Tasks.Task.FinishContinuations(;Tasks.Task.RunOrQueueCompletionAction;Tasks.Task.RunContinuations(;Tasks.Task+DelayPromise+;Tasks.AwaitTaskContinuation.RunOrScheduleAction;Runtime.CompilerServices.AsyncTaskMethodBuilder`;CompilerServices.AsyncTaskMethodBuilder.Start(;CompilerServices.AsyncMethodBuilderCore.Start(;ExecutionContext.RunInternal(;CompilerServices.AsyncTaskMethodBuilder.SetResult(;Tasks.VoidTaskResult].TrySetResult;.TrySetResult(System.Threading.Tasks.;Tasks.Task.TrySetResult(

the callstack was much more readable:

Morph the frames

The final step is to transform the frames text into something more meaningful thanks to the GroupPats combo-box. At the beginning of this post, I picked the predefined [fold threads] Thread -> AllThreads grouping pattern. The starting text between [] is used as a title by Perfview to allow the user to more easily figure out what its role is. The rest of the string defines how parts of each frame should match and be grouped. The corresponding contextual help has already been shown earlier when the toolbar was detailed.

Here, I don’t want to group all sibling frames into a group but rather morph the text into something more readable. The part before -> or => is used as matching pattern to be replaced by the part after the sign. It is also possible to “extract” elements between {} from the matching pattern to be used to build the replacement string. Each matched element is identified as $1, $2,… based on its position in the pattern.

In my example, I would like to apply the following transformation:

To write the pattern, you should focus on the separators (!, +< and >d__ in this case):

{%}!{%}+<{%}>d__*

And the items to extract are what is left in between; identified as {%}.

The building of the replacement string is simply counting the matching item position (starting at 1):

($1) $2 ~~~async back to~~~ $3()

And here is the corresponding final result:

Don’t lose your xxxPats!

It is interesting to note that you could define your own patterns preset via the Preset menu item

As you can see here, I have defined my own Criteo Arbitrage preset. If you want to reuse the content of GroupPats, FoldPats, and Fold% combo-boxes, click the Save As Preset (or even Set As Startup Preset to get them when you start Perfview) and pick a name

Feel free to use the Manage Presets dialog for easier preset manipulation:

I hope that, now, you better understand the value of Perfview to analyze complicated callstacks.

Read more from Christophe on our Medium blog!

Consul Streaming: What’s behind it? *Let’s look at new hidden feature for Consul large or very dynamic clusters of Consul 1.9: Streaming.*medium.com

Like what you are reading? Join us and make an impact!

Careers at Criteo | Criteo jobs *Find opportunities everywhere. Choose your next challenge. Find the job opportunities at Criteo in Product, research &…*careers.criteo.com

Understanding “reversed” callstacks in Visual Studio and Perfview with async/await code

Tue, 19 Jan 2021 17:16:08 +0000

Introduction

With my colleague Eugene, we spent a long time analyzing performances of one of Criteo main applications with Perfview. The application is processing thousand of requests in an asynchronous pipeline full of async/**await **calls. During our research, we ended up with weird callstacks that looked kind of “reversed”. The goal of this post is to describe why this could happen (even in Visual Studio).

Let’s see the result of profiling in Visual Studio

I wrote a simple .NET Core application that simulates a few async/**await **calls:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


static async Task Main(string[] args)
{
    Console.WriteLine($"pid = {Process.GetCurrentProcess().Id}");
    Console.WriteLine("press ENTER to start...");
    Console.ReadLine();
    await ComputeAsync();

    Console.WriteLine("press ENTER to exit...");
    Console.ReadLine();
}

private static async Task ComputeAsync()
{
    await Task.WhenAll(
        Compute1(),
        ...
        Compute1()
        ); 
}

ComputeAsync is starting a bunch of tasks that will await other **async **methods:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


private static async Task Compute1()
{
    ConsumeCPU();
    await Compute2();
    ConsumeCPUAfterCompute2();
}

private static async Task Compute2()
{
    ConsumeCPU();
    await Compute3();
    ConsumeCPUAfterCompute3();
}

private static async Task Compute3()
{
    await Task.Delay(1000);
    ConsumeCPUinCompute3();
    Console.WriteLine("DONE");
}

Unlike the Compute1 and Compute2 methods, the last Compute3 is waiting 1 second before consuming some CPU with square root computation in CompusumeCPUXXX helpers:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


[MethodImpl(MethodImplOptions.NoInlining)]
private static void ConsumeCPUinCompute3()
{
    ConsumeCPU();
}

[MethodImpl(MethodImplOptions.NoInlining)]
private static void ConsumeCPUAfterCompute3()
{
    ConsumeCPU();
}

[MethodImpl(MethodImplOptions.NoInlining)]
private static void ConsumeCPUAfterCompute2()
{
    ConsumeCPU();
}

private static void ConsumeCPU()
{
    for (int i = 0; i < 1000; i++)
        for (int j = 0; j < 1000000; j++)
        {
            Math.Sqrt((double)j);
        }
}

From Visual Studio, profile the CPU usage of this test program via Debug | Performance Profiler…

In the summary result panel, click the Open Details… link

And pick the Call Tree view

You should see two paths of execution:

If you open the last one, you should see the expected chain of calls:

… if the methods were synchronous; which is not the case. So Visual Studio did a great job in dealing with the implementation details of async/**await **to present a nice call stack.

However, if you open the first node, you get something more disturbing:

… if you don’t know how async/**await **is implemented. My Compute3 code is definitively not calling Compute2 which is not calling Compute1! This is where Visual Studio smart frame/callstack reconstruction brings more confusion than anything else. So what’s going on?

Understanding async/await implementation

Unlike Visual Studio that is hiding real calls, you should be able to see what methods are really called when analyzing a memory dump with dotnet-dump and the pstacks command:

If you follow the arrows from the bottom to the top, you should see the following synchronous (because as frame in thread callstacks) calls:

a timer callback is calling d__4.MoveNext() : this corresponds to the end of the Task.Delay inCompute3 method.
d__3.MoveNext() gets called to continue the code after await Compute3
d__.MoveNext() gets called to continue the code after await Compute2
ConsumeCPUAfterCompute2() gets called as expected
ComputeCPU()** **or ConsumeCPUInCompute3() get called as expected

All the fancy methods names are due to “state machine” types that is generated by the C# compiler when you (1) define **async **methods that (2) await other **async **methods (or any “awaitable” object). Their role is to manage a “state machine” to execute code synchronously up to an await call, and again up to the next await call, and again and again until the method returns.

All these d__* types contains fields corresponding to each async method local variables and parameters if any. For example, here is what is generated for the ComputeAsync** **and Compute1/2/3 async methods without any local or parameter:

The integer <>1__state field keeps track of the “execution state” of the machine. For example, after the state machine is created in Compute1, this field is set to -1:

I don’t want to dig into the builder details but just let’s just say that the MoveNext method of the state machine d__2 gets executed (by the same thread).

Before looking at the MoveNext implementation corresponding to the Compute1 method (without exception handling), keep in mind that it has to :

run all code up to an **await **call,
change the “execution state” (more on this later)
do some magic to execute that code in another thread (if needed — more on this later)
come back to continue the execution of the code after the await call
and do that up to the next await call again and again

Since <>1__state is -1, the first “synchronous” part of the code is executed (i.e. calling ComsumeCPU method).

The Compute2 method is then called to get the corresponding awaitable object (here a Task). If the task runs immediately (i.e. no await call such as a simple Task.FromResult() in the async method), IsCompleted() will return true and the code after the await call will be run by the same thread. Yes it means that async/await calls could be run synchronously by the same thread: why creating a thread when it is not needed?

If the Task is passed to the ThreadPool to be executed by a worker thread, the <>1__state value is set to 0 (so the next time MoveNext is called, the next “synchronous” part (i.e. after the await call) will be executed). The code now calls awaitUnsafeOnCompleted to do its magic: adding a continuation to the Compute2 task (the first awaiter parameter) so that MoveNext will be called on that same state machine (the second this parameter) when the task ends. The current thread then quietly returns.

So when the Compute2 task ends, its continuation runs to call MoveNext this time with <>1__state as 0 so the last two lines are executed: awaiter.GetResult() returns immediately because the Task returned by Compute2 already ended and the last CinsumeCPUAfterCompute2 method is now called.

Here is a summary of what is happening:

Each time you see an async method, the C# compiler is generating a dedicated state machine type with a MoveNext method that is responsible for executing your code synchronously between await calls
each time you see an await call, it means that a continuation will be added to the Task wrapping the async method to be executed. That continuation code will call the MoveNext method of the state machine of the calling method to execute the next piece of code up to its next await call.

This is why Visual Studio, trying to smartly match each async method state machine MoveNext frame to the method itself, shows reversed callstacks: the shown frames are the ones corresponding to the continuations after the await calls (in green in the previous figure).

Note that I described in more details how async/await is working and the action of AwaitUnsageOnCompleted during a DotNext conference session with Kevin so feel free to watch the recording at that particular time if you want to go deeper.

The next post will describe what to do in Perfview to get more readable callstacks.

Stay tuned!

Check out our latest posts on Medium:

Top Applications of Graph Neural Networks 2021 *GNNs have come a long way in academia. But do we have good applications of them in industry?*medium.com Build your own .NET CPU profiler in C# *After describing memory allocation profiling it is now time to dig into the CPU sample profiling in C#!*medium.com

Join the crowd!

Careers at Criteo | Criteo jobs *Find opportunities everywhere. Choose your next challenge. *careers.criteo.com

How to write your own commands in dotnet-dump (2/2)

Mon, 09 Nov 2020 16:27:38 +0000

In the previous post, I presented the new commands that were added to dotnet-dump and how to use them. It is now time to show how to implement such a command.

But before jumping into the code, you should first ensure that you have a valid use case that the Diagnostics team is not currently working on. I recommend to create an issue in the Diagnostics repository to explain what is missing for which scenario and propose to implement the corresponding command.

What is a dotnet-dump command?

Here is the directory structure related to the dotnet-dump tool in the Diagnostics repository:

The built binaries are generated under artifacts\bin\dotnet-dump\netcoreapp2.1 folder if you need to test them outside of Visual Studio.

The eng folder contains the versions.props file that lists the versions for nuget dependencies. In my case, I had to reference the ParallelStacks.Runtime nuget so I added the following line:2.0.1

And in the dotnet-dump.csproj, this nuget is referenced with the same variable: `

Missed the first part of the story? Read it here:

How to extend dotnet-dump (1/2) — What are the new commands? *This first post describes the new commands, when to use them, and the git setup I used to implement them.*medium.com

Want to work with Christophe or other teams? Check out our open positions:

Careers at Criteo | Criteo jobs *Find opportunities everywhere. Choose your next challenge. Find the job opportunities at Criteo in Product, research &…*careers.criteo.com

The .NET Core Journey at Criteo

Fri, 31 Jul 2020 15:55:50 +0000

Introduction

When I arrived at Criteo in late 2016, I joined the .NET Core “guild” (i.e. group of people from different teams dedicated to a specific topic). The first meeting I attended included Microsoft folks led by Scott Hunter (head of .NET program management) and including David Fowler (SignalR and ASP.NET Core). The goal for Criteo was simple: Moving a set of C# applications from Windows/.NET Framework to Linux/.NET Core. I guess that for Microsoft we were a customer with workloads that could be interesting to support with .NET Core. At that time, I did not realize how strong their commitment to work with us was. Our Open Source mindset was the selling point.

How complicated could it be? Well… this post will show you the challenges that we had to face to run, monitor and debug our applications.

Try it

Once we got a build of all .NET Core assemblies (more on this in a forthcoming blog post), it was time to run a few applications. The first issues that we faced were related to missing features between .NET Framework and .NET Core. For example, we need cryptography support of 3DES and AES with cypher mode CFB but it is (still) not available in .NET Core for Linux. Thanks to the Open Source status of .NET Core, we were able to add it to CoreFx. However, since we did not implement it on MacOS/Windows as Microsoft requested for our change to be accepted as a Pull Request, we had to keep our Criteo-forked branch.

The second class of runtime problems we had to solve were due to differences between Windows and Linux but also with the “containerization” of the runtime environment. Let’s take two examples involving the .NET Garbage Collector. First, our containers were using Linux cgroups to manage quotas including memory and number of CPU cores usable by applications. However, at CLR startup, the GC was counting the total count of CPU cores to compute the number of heaps to allocate instead of the one defined at the cgroup level: We ended up with instant Out Of Memory automatic killing. This time our fix was done and merged in the CLR repository.

The second example is related to a GC optimization: During background generation 2 collections, the CLR threads working underneath are affinitized to each different CPU core to avoid locks. We were lucky enough to welcome Maoni Stephens (Lead Dev on the GC) in our Paris office early 2018 to share our weird allocation patterns that impacted the GC. During her stay, she was kind enough to help us investigate a behavior on our servers: When SysInternals ProcessExplorer was running, the garbage collections were taking more time than usual. Maoni found out ProcessExplorer had an affinitized high priority thread conflicting with GC threads. During investigations related to longer response time on Linux compared to Windows. We realized that GC threads were not affinitized like it was the case on Windows and the issue was fixed by Jan Vorlicek.

Here is our lesson: Sometimes fixes are merged into the official release and sometimes they are not. If your workloads are pushing .NET to its limits, you will probably have to build and manage your own Core fork and make it available to your deployments.

Monitor it

At Criteo, our Grafana dashboards measuring .NET Framework application health were based on metrics computed from Windows performance counters. Even without going to Linux, .NET Core is no more exposing performance counters so we had to entirely rebuild our metrics collection system!

Based on Microsoft feedbacks, we decided to listen to CLR events emitted via ETW on Windows and LTTng on Linux. In addition to work for both Operating Systems, these events are also providing accurate details about thread contention, exceptions and garbage collections not available with Performance counters. Please refer to our series of blog posts for more details and reusable code samples to integrate these events into your own systems.

Our first Linux metrics collection implementation was based on LTTng and we presented our journey during the Tracing Summit in 2017. Microsoft already built TraceEvent, an assembly allowing .NET code to parse CLR events for both Windows and Linux. Unfortunately for us, the Linux part was only able to load traces files but we needed live session like on Windows where you can listen to events emitted by running applications. Since this code is Open Source, Gregory was able to add the live session feature to TraceEvent.

With .NET Core 3.0 Microsoft provided a way to exchange events common to Linux and Windows called EventPipes. So… we moved our collection implementation from LTTng to EventPipe (look at our blog series and DotNext conference session for more details and reusable code sample). With the new EventPipe implementation in the CLR came performance issues not seen by Microsoft. The reason is simple: Some of our applications are running hundreds of threads to process thousands of requests per second and allocate memory like crazy. In that kind of context, the CLR has a lot to do and so, has a lot of events to generate and emit via LTTng or EventPipes.

The initial implementation was lacking some filtering and too many events were generated or expensive event payload was created even though the events were not emitted. Based on our feedback, the Microsoft Diagnostic team was very responsive and quickly fixed the problem.

Microsoft did not “just” move to Open Source, the teams are working deeply integrated with the issue/pull request model of GitHub. So don’t be shy and if you find a problem, create an issue with a detailed reproduction and even better, provide a pull request with the fix. Everyone in the community will benefit!

Run it

With these metrics, we started to investigate some performance differences (mostly response time) between Windows and containerized Linux.

We saw a huge performance difference on Linux: Both response time (x2) and scalability (timeout increase with QPS). Our team spent a lot of time to improve the situation up to the point where it was possible to send the applications to production.

In the new containerized environment we faced the same kind of noisy neighbor symptoms that we had with Process Explorer. If the CPU cores are not dedicated to a container (as it was for us at the beginning), this scenario happens a lot. So we updated the scheduling system to dedicate CPU cores to containers.

On a totally different area, we found out that the way .NET Core handles network I/O continuation had an impact on our main application. To give a bit of context, this application has to handle a lot of requests and is response-time driven. During the processing of a request, the current thread might have to send an HTTP request before continuing its processing. Since this is done asynchronously, the thread is now available to process more incoming requests and this is good for throughput. However, it means that when the inner HTTP request comes back, all available threads might be processing new incoming requests and it will take time to complete the old one. The net effect is to increase the median response time and this is not something we want!

The .NET Core implementation is relying on the .NET ThreadPool that shares its threads with all the async/await magic and the incoming requests processing (The .NET Framework implementation is using a totally different implementation based on I/O completion ports on Windows). To solve the issue, Kevin implemented a custom thread pool to handle network I/O and we keep on optimizing it. When you work on this kind of deep area of code-shared by so many different workloads, you realize that it is impossible to find the silver bullet.

Debug it

What would you do if something would go wrong in an application? On Windows, with Visual Studio, we are able to remote debug a rogue application to set a breakpoint, look at fields and properties or even have a high-level view of what threads are doing with the ParallelStacks view. In the worst case, SysInternals procdump allows us to take a snapshot of the application and analyze it on our developer’s machine with WinDBG or Visual Studio.

In terms of remote debugging a Linux application, Microsoft provides an SSH-based solution to attach to a running application. However, for security reasons, it is not allowed to run an SSH server in our Criteo containers. The solution was to implement the communication protocol with VsDbg for Linux on top of WebSockets.

Well… this was not enough. Hosting architecture (Marathon and Mesos in our case) ensures that applications in containers are running smoothly by sending requests to health check endpoints. If the application replies that everything is fine, then the container is safe. If the application does not answer as expected (including retries), then Marathon/Mesos kills the application and cleans up the container. Now think about what will happen if you set a breakpoint in the application and you dig into the data structures content in Visual Studio Watch/Quick Watch panels for a few minutes. Behind the scene, the debugger has to freeze all application threads, including the ones from the thread pool responsible to answer health checks. As you have probably guessed already, the debugging session will not end well.

This is why the previous figure shows an arrow between Marathon and the Remote Debugger which acts as a proxy for the application health check. When a debugging session starts (i.e. when the WebSockets code executes the protocol), the Remote Debugger knows that it should answer OK instead of calling the application endpoint that might never answer.

When remote debugging is not enough, how do you take a memory snapshot of the application? For example, if the health check does not answer after a series of retry, the Remote Debugger is calling the createdump tool installed with the .NET Core runtime to generate a dump file. Again, since the memory dump creation of 40+ GB applications could take several minutes, the same health check proxy mechanism has been put in place.

Once the dump file is created, the remote debugger let Marathon kill the application. But wait! This is not enough because in that case, the container will be cleaned up and the disk storage will disappear. Not a problem, after a dump has been generated by createdump, the file is sent to a “Dump Navigator” application (one per data center). This application is providing a simple HTML user interface to get high-level details of the application state such as thread stacks or managed heap content.

On Windows, we have built our own set of extension commands that allow us to investigate memory, threadpool starvation, thread contention, or timer leak scenarios in a Windows memory dump with WinDBG as shown during this NDC Oslo conference session. Note that they are also usable with LLDB on Linux. These commands are leveraging the ClrMD Microsoft library that gives you access to a live process or a memory dump in C#. Thanks to the Linux support that has been added to this library by Microsoft developers, it was easy to reuse the code into our Dump Navigator application. I definitively recommend to look at the API provided by ClrMD to automate and build your own tools. The long Criteo blog series is a good start in addition to my DotNext conference session.

Conclusion

Even though some of our main applications moved to .NET Core running on containerized Linux with a large set of monitoring/debugging tools, the journey is not over. We are now testing the preview of .NET Core 5.0 (like we did for 3.0) to check if it supports Criteo specific needs. If this is not the case, we will figure out why and find solutions to integrate into the code. Same for the tools: I have started to add our extension commands to Microsoft dotnet-dump CLI tool used to analyze both Windows and Linux dumps.

At least we could say that we not only helped ourselves but also Microsoft to understand how far .NET Core could go and even the whole .NET Windows and Linux community. This is where Open Source shines!

Stay tuned for the next article in our mini-series. Don’t forget to head over to our previous articles of this journey:

Migrating Arbitrage to Apache Mesos *Lessons learned from migrating our largest application to our container platform.*medium.com Moving .NET to Linux at Scale *The story of a multi-year migration: How we changed Criteo’s whole foundation.*medium.com

Interested in joining the challenge? Head over to our career site!

Product, Research & Development | Criteo Careers careers.criteo.com

Build your own .NET memory profiler in C# — call stacks (2/2–2)

Fri, 19 Jun 2020 09:32:16 +0000

In the past two episodes of this series I have explained how to get a sampling of .NET application allocations and one way to get the call stack corresponding to the allocations; all with CLR events. In this last episode, I will detail how to transform addresses from the stack into methods name and possibly signature.

From managed address to method signature

In order to transform an address on the stack into a managed method name, you need to know where in memory (i.e. at which address) is stored the method JITted assembly code and what is its size:

For each JITted method, the MethodLoadVerbose/MethodDCStartVerboseV2 events are providing this information in addition to 3 properties to rebuild the full method name and signature (more on this later). I’m storing each method description as a MethodInfo into a MethodStore per process.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


public class PerProcessProfilingState : IDisposable
{
    ...
    private readonly Dictionary<int, MethodStore> _methods = new Dictionary<int, MethodStore>();

public class MethodStore : IDisposable
{
    // JITed methods information (start address + size + signature)
    private readonly List<MethodInfo> _methods;
    ...

The only interesting part of the MethodInfo class is the computation of the full method name stored in the _fullName field:

1
2
3
4
5
6


public class MethodInfo
{
    private readonly ulong _startAddress;
    private readonly int _size;
    private readonly string _fullName;
    ...

The ComputeFullName helper merges together the 3 properties given by the MethodxxxVerbose events including special processing for constructors:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


private string ComputeFullName(ulong startAddress, string namespaceAndTypeName, string name, string signature)
{
    var fullName = signature;

    // constructor case: name = .ctor | namespaceAndTypeName = A.B.typeName | signature = ...  (parameters)
    // --> A.B.typeName(parameters)
    if (name == ".ctor")
    {
        return $"{namespaceAndTypeName}{ExtractParameters(signature)}";
    }

    // general case: name = Foo | namespaceAndTypeName = A.B.typeName | signature = ...  (parameters)
    // --> A.B.Foo(parameters)
    fullName = $"{namespaceAndTypeName}.{name}{ExtractParameters(signature)}";
    return fullName;
}

private string ExtractTypeName(string namespaceAndTypeName)
{
    var pos = namespaceAndTypeName.LastIndexOf(".", StringComparison.Ordinal);
    if (pos == -1)
    {
        return namespaceAndTypeName;
    }
    
    // skip the .
    pos++;

    return namespaceAndTypeName.Substring(pos);
}

Only the parameters (not the return type) are extracted from the “return type SPACE SPACE (parameters)” signature format:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


private string ExtractParameters(string signature)
{
    var pos = signature.IndexOf("  (");
    if (pos == -1)
    {
        return "(???)";
    }

    // skip double space
    pos += 2;

    var parameters = signature.Substring(pos);
    return parameters;
}

With the starting address and the size of each JITted methods, it is easy to find the one corresponding to a given address on the stack: look for the MethodInfo where this address could be between the start address and the start address + the code size:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


public string GetFullName(ulong address)
{
    if (_cache.TryGetValue(address, out var fullName))
        return fullName;
    
    // look for managed methods
    for (int i = 0; i < _methods.Count; i++)
    {
        var method = _methods[i];
        
        if ((address >= method.StartAddress) && (address < method.StartAddress + (ulong)method.Size))
        {
            fullName = method.FullName;
            _cache[address] = fullName;
            return fullName;
        }
    }

    // look for native methods
    fullName = GetNativeMethodName(address);
    _cache[address] = fullName;

    return fullName;
}

For performance sake, the _cache dictionary property speeds up the process by keeping track of the address/full name mappings.

It is now time to look at the details of the GetNativeMethodName helper that takes care of the native functions scenario.

The native part of the symbols story

Unlike for JITted methods, the CLR does not send events to describe native functions even for the CLR itself. Instead, you need to find a way to map a call stack address to a native function by yourself. Unlike Perfview, I will be using the dbghelp native API instead of DIA mostly because my scenario is to get the stacks while the applications are still running:

After reading the march 2002 MSDN article about DBGHELP by Matt Pietrek, the updated symbols related Microsoft Docs and the dbghelp.h include a file from the Windows SDK, I wrote a C# wrapper around the dbghelp function needed to get a method name from an address in a process address space:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


internal static class NativeDbgHelp
{
    // from C:\Program Files (x86)\Windows Kits\10\Debuggers\inc\dbghelp.h
    public const uint SYMOPT_UNDNAME = 0x00000002;
    public const uint SYMOPT_DEFERRED_LOADS = 0x00000004;

    [StructLayout(LayoutKind.Sequential)]
    public struct SYMBOL_INFO
    {
        public uint SizeOfStruct;
        public uint TypeIndex;      // Type Index of symbol
        private ulong Reserved1;
        private ulong Reserved2;
        public uint Index;
        public uint Size;
        public ulong ModBase;       // Base Address of module containing this symbol
        public uint Flags;
        public ulong Value;         // Value of symbol, ValuePresent should be 1
        public ulong Address;       // Address of symbol including base address of module
        public uint Register;       // register holding value or pointer to value
        public uint Scope;          // scope of the symbol
        public uint Tag;            // pdb classification
        public uint NameLen;        // Actual length of name
        public uint MaxNameLen;
        [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 1024)]
        public string Name;
    }

    [DllImport("dbghelp.dll", SetLastError = true)]
    public static extern bool SymInitialize(IntPtr hProcess, string userSearchPath, bool invadeProcess);

    [DllImport("dbghelp.dll", SetLastError = true)]
    public static extern uint SymSetOptions(uint symOptions);

    [DllImport("dbghelp.dll", SetLastError = true, CharSet = CharSet.Ansi)]
    public static extern ulong SymLoadModule64(IntPtr hProcess, IntPtr hFile, string imageName, string moduleName, ulong baseOfDll, uint sizeOfDll);

    // use ANSI version to ensure the right size of the structure 
    // read https://docs.microsoft.com/en-us/windows/win32/api/dbghelp/ns-dbghelp-symbol_info
    [DllImport("dbghelp.dll", SetLastError = true, CharSet = CharSet.Ansi)]
    public static extern bool SymFromAddr(IntPtr hProcess, ulong address, out ulong displacement, ref SYMBOL_INFO symbol);

    [DllImport("dbghelp.dll", SetLastError = true)]
    public static extern bool SymCleanup(IntPtr hProcess);
}

Note that you will need to download the dbghelp.dll (SymSrv.dll if needed) from the Windows SDK and copy it next to your memory profiler binaries.

The usage of the dbghelp API is straightforward. First, for each new process, call SymSetOptions/SymInitialize** **with a handle of the process:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


private bool SymInitialize(IntPtr hProcess)
{
    // read https://docs.microsoft.com/en-us/windows/win32/api/dbghelp/nf-dbghelp-symsetoptions for more details
    // maybe SYMOPT_NO_PROMPTS and SYMOPT_FAIL_CRITICAL_ERRORS could be used
    NativeDbgHelp.SymSetOptions(
        NativeDbgHelp.SYMOPT_DEFERRED_LOADS |   // performance optimization
        NativeDbgHelp.SYMOPT_UNDNAME            // C++ names are not mangled
        );

    // https://docs.microsoft.com/en-us/windows/win32/api/dbghelp/nf-dbghelp-syminitialize
    // search path for symbols:
    //   - The current working directory of the application
    //   - The _NT_SYMBOL_PATH environment variable
    //   - The _NT_ALTERNATE_SYMBOL_PATH environment variable
    //
    // passing false as last parameter means that we will need to call SymLoadModule64 
    // each time a module is loaded in the process
    return NativeDbgHelp.SymInitialize(hProcess, null, false);
}
private IntPtr BindToProcess(int pid)
{
    try
    {
        _process = Process.GetProcessById(pid);

        if (!SymInitialize(_process.Handle))
            return IntPtr.Zero;

        return _process.Handle;
    }
    catch (Exception x)
    {
        Console.WriteLine($"Error while binding pid #{pid} to DbgHelp:");
        Console.WriteLine(x.Message);
        return IntPtr.Zero;
    }
}

In the case of protected processes, Process.GetProcessById might throw an exception. The _hProcess field storing the process handle will be cleaned up in the IDisposible.Dispose implementation of the MethodStore:

1
2
3
4
5
6
7
8


public void Dispose()
{
    if (_hProcess == IntPtr.Zero)
        return;
    _hProcess = IntPtr.Zero;

    _process.Dispose();
}

After a process has been bound, each time one of its modules is loaded, SymLoadModule64 must be called. You can be notified of such a loaded module by enabling the Kernel provider with the ImageLoad keyword.

1
2
3
4
5


session.EnableKernelProvider(
    KernelTraceEventParser.Keywords.ImageLoad |
    KernelTraceEventParser.Keywords.Process,
    KernelTraceEventParser.Keywords.None
);

The handler attached to the ImageLoaded event will be called each time a dll gets loaded.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


private void SetupListeners(ETWTraceEventSource source)
{
    ...

    // get notified when a module is load to map the corresponding symbols
    source.Kernel.ImageLoad += OnImageLoad;
}

const int ERROR_SUCCESS = 0;
private void OnImageLoad(ImageLoadTraceData data)
{
    if (FilterOutEvent(data)) return;

    GetProcessMethods(data.ProcessID).AddModule(data.FileName, data.ImageBase, data.ImageSize);
}
public void AddModule(string filename, ulong baseOfDll, int sizeOfDll)
{
    var baseAddress = NativeDbgHelp.SymLoadModule64(_hProcess, IntPtr.Zero, filename, null, baseOfDll, (uint)sizeOfDll);
    if (baseAddress == 0)
    {
        // should work if the same module is added more than once
        if (Marshal.GetLastWin32Error() == ERROR_SUCCESS) return;

        Console.WriteLine($"SymLoadModule64 failed for {filename}");
    }
}

Now everything is in place to get a native function name from an address on the stack:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


private string GetNativeMethodName(ulong address)
{
    var symbol = new NativeDbgHelp.SYMBOL_INFO();
    symbol.MaxNameLen = 1024;
    symbol.SizeOfStruct = (uint)Marshal.SizeOf(symbol) - 1024;   // char buffer is not counted
    // the ANSI version of SymFromAddr is called so each character is 1 byte long

    if (NativeDbgHelp.SymFromAddr(_hProcess, address, out var displacement, ref symbol))
    {
        var buffer = new StringBuilder(symbol.Name.Length);

        // remove weird "$##" at the end of some symbols
        var pos = symbol.Name.LastIndexOf("$##");
        if (pos == -1)
            buffer.Append(symbol.Name);
        else
            buffer.Append(symbol.Name, 0, pos);

        // add offset if any
        if (displacement != 0)
            buffer.Append($"+0x{displacement}");

        return buffer.ToString();
    }

    // default value is the just the address in HEX
    return $"0x{address:x}";
}

Note that I needed to remove some unexpected $## strings are the end of some symbols.

This is the last episode of the series about building your own memory profiler in C#. In case you missed the first episodes, check them out on Medium:

Build your own .NET memory profiler in C# — call stacks (2/2–1) *This post explains how to get the call stack corresponding to the allocations with CLR events.*medium.com Build your own .NET memory profiler in C# *This post explains how to collect allocation details by writing your own memory profiler in C#.*medium.com

Resources

Source code available on Github.
Download Debugging Tools for Windows for dbghelp.dll and SymSrv.dll
Matt Pietrek article in MSDN Magazine about DBGHELP
Dbghelp samples -http://www.debuginfo.com/examples/dbghelpexamples.html

Join the crowd!

Careers at Criteo | Criteo jobs *Find opportunities everywhere. Choose your next challenge. Find the job opportunities at Criteo in Product, research &…*careers.criteo.com

Build your own .NET memory profiler in C# — call stacks (2/2–1)

Mon, 18 May 2020 12:07:01 +0000

In the previous episode of this series, you have seen how to get a sampling of .NET application allocations thanks to the AllocationTick and GCSampleObjectAllocation(High/Low) CLR events. However, this is often not enough to investigate unexpected memory consumption: you would need to know which part of the code is triggering the allocations. This post explains how to get the call stack corresponding to the allocations, again with CLR events.

Introduction

If you look carefully at the payload of the TraceEvent object mapped by Microsoft TraceEvent library (not my fault if they have the same name) for each CLR event, you won’t see anything related to a call stack. However, in the TraceEvent sample 41, the following line looks promising:

var callStack = data.CallStack();

with data being a TraceEvent object received for each CLR event!

This CallStack method is an extension method provided by the TraceLog special kind of event source. You might not have noticed but I have used it in the AllocationTick code sample from the previous post. This class (and many more helper classes) is doing a lot of work to :

“attach” a call stack to each CLR event; i.e. a list of addresses of assembly code
to translate addresses into string symbols (method names or full signatures), listen to a bunch of JIT related events for managed methods (more on this later), using COM-based Debug Interface Access (a.k.a. DIA) and MetadataReaderProvider** **for native functions

Notice that since events from all managed processes on the machine are handled by TraceLog, the internal cache for JITted methods description could consume a lot of memory. During my tests with two Visual Studio running, my test profiler consumed more than 500 MB before even handling call stacks. If you are in such an environment with multiple .NET processes, I will show how to “manually” get the same stacks (+ symbols in the next episode) with CLR events and a few methods from dbghelp.dll in a cheaper way.

The new provider (more on ClrRundown later), keywords and events need to be received to make all this work:

TraceLog: the easy way

As you have seen in the previous posts, the TraceEventSession class exposes a Source property of ETWTraceEventSource type. This source has event parsers properties from which you register handler methods that will be called when CLR events are received. Instead of directly using this source, you should wrap it with a TraceLogEventSource object that provides the same event parsers.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


await Task.Factory.StartNew(() =>
{
    using (_session)
    {
        SetupProviders(_session);

        using (TraceLogEventSource source = TraceLog.CreateFromTraceEventSession(_session))
        {
            SetupListeners(source);

            source.Process();
        }
    }
});

What’s new with providers?

The code for mySetupProviders method is a little bit different from the previous post even though no new event listeners are needed:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


private void SetupProviders(TraceEventSession session)
{
    // Note: the kernel provider MUST be the first provider to be enabled
    // If the kernel provider is not enabled, the callstacks for CLR events are still received 
    // but the symbols are not found (except for the application itself)
    // TraceEvent implementation details triggered when a module (image) is loaded
    session.EnableKernelProvider(
        KernelTraceEventParser.Keywords.ImageLoad |
        KernelTraceEventParser.Keywords.Process,
        KernelTraceEventParser.Keywords.None
    );

    session.EnableProvider(
        ClrTraceEventParser.ProviderGuid,
        TraceEventLevel.Verbose,    // this is needed in order to receive AllocationTick_V2 event
        (ulong)(
        // required to receive AllocationTick events
        ClrTraceEventParser.Keywords.GC |
        ClrTraceEventParser.Keywords.Jit |                      // Turning on JIT events is necessary to resolve JIT compiled code 
        ClrTraceEventParser.Keywords.JittedMethodILToNativeMap |// This is needed if you want line number information in the stacks
        ClrTraceEventParser.Keywords.Loader |                   // You must include loader events as well to resolve JIT compiled code.
        ClrTraceEventParser.Keywords.Stack
        )
    );

    // this provider will send events of already JITed methods
    session.EnableProvider(ClrRundownTraceEventParser.ProviderGuid, TraceEventLevel.Informational,
    (ulong)(
        ClrTraceEventParser.Keywords.Jit |              // We need JIT events to be rundown to resolve method names
        ClrTraceEventParser.Keywords.JittedMethodILToNativeMap | // This is needed if you want line number information in the stacks
        ClrTraceEventParser.Keywords.Loader |           // As well as the module load events.  
        ClrTraceEventParser.Keywords.StartEnumeration   // This indicates to do the rundown now (at enable time)
        ));

}

The kernel provider needs to be enabled with the ImageLoad and Process keywords in order to let TraceEvent detect when a process loads “images” (i.e. dlls) and at which address (needed to convert Relative Virtual Addresses (RVA) to addresses in the address space). Note that this provider must be enabled before any other provider or your code will trigger an exception.
The CLR provider needs to be enabled with Jit, JittedMethodILToNativeMap, and Loader (in addition to the usual GC one).
The Stack keyword has to be set on the same CLR provider to receive call stacks events for “normal” CLR event (more on this later)
The CLR Rundown provider is enabled with the same Jit, JittedMethodILToNativeMap, and Loader keywords. That way, JIT events corresponding to already JITted methods will be received (not only the new ones). This is important because otherwise, you won’t be able to map these methods with the address in memory of their JITted native code in the case of processes that have been started before the profiler. This is the case for my AllocationTickProfiler sample.

Callstacks and symbols

Now, when an AllocationTick event is received, calling the CallStack extension method on the GCAllocationTickTraceData argument returns a TraceCallStack object. This class is a linked list of TraceCodeAddress representing each stack frame (i.e. address in assembly code). These classes are at the heart of TraceEvent and Perfview callstack management. The method names and signatures are retrieved behind the scene thanks to JIT events and the SymbolReader class that digs into .pdb files.

You first need to initialize a SymbolReader instance:

Set the path to find the .pdb; including the Microsoft HTTP endpoint for public .NET versions symbols,
Allow pdb next to the executable to be loaded.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


// By default a symbol Reader uses whatever is in the _NT_SYMBOL_PATH variable.  However you can override
// if you wish by passing it to the SymbolReader constructor.  Since we want this to work even if you 
// have not set an _NT_SYMBOL_PATH, so we add the Microsoft default symbol server path to be sure/
var symbolPath = new SymbolPath(SymbolPath.SymbolPathFromEnvironment).Add(SymbolPath.MicrosoftSymbolServerPath);
_symbolReader = new SymbolReader(_symbolLookupMessages, symbolPath.ToString());

// By default the symbol reader will NOT read PDBs from 'unsafe' locations (like next to the EXE)  
// because hackers might make malicious PDBs. If you wish ignore this threat, you can override this
// check to always return 'true' for checking that a PDB is 'safe'.  
_symbolReader.SecurityCheck = (path => true);

Then, displaying a TraceCallStack from a received CLR event in a human-readable format is simple:

Get one frame after the other from the linked list,
If the CodeAddress field is not cached yet, load the symbols for its module,
Display the FullMethodName field of the frame (or the address if not found).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


private void DumpStack(TraceCallStack callStack)
{
    while (callStack != null)
    {
        var codeAddress = callStack.CodeAddress;
        if (codeAddress.Method == null)
        {
            var moduleFile = codeAddress.ModuleFile;
            if (moduleFile == null)
            {
                Debug.WriteLine($"Could not find module for Address 0x{codeAddress.Address:x}");
            }
            else
            {
                codeAddress.CodeAddresses.LookupSymbolsForModule(_symbolReader, moduleFile);
            }
        }
        if (!string.IsNullOrEmpty(codeAddress.FullMethodName))
            Console.WriteLine($"     {codeAddress.FullMethodName}");
        else
            Console.WriteLine($"     0x{codeAddress.Address:x}");
        callStack = callStack.Caller;
    }
}

Note that the first frame in the linked list is the last on the stack (i.e. last executed method).

As I mentioned at the beginning of the post, I have been facing OutOfMemory errors due to the TraceEvent symbols management large memory usage when a few other .NET applications were running. Let’s see how to get the call stacks in a less memory consuming way.

Manually rebuilding the allocations call stack

Instead of using the call stack and symbol management provided by TraceLog in TraceEvent, I would prefer to manually get them. If you remember the last post, thanks to GCSampledObjectAllocation CLR events, it is possible to have a sampling of the allocation size and count per process and per type. What I would like to add to the type picture is the list of call stacks leading to these allocations.

How to manually get CLR events call stack

The first step is to understand how to get the CLR events call stacks. If you use the TraceLog-based code just presented, you should see the following kind of call stack:

The ETWCallout CLR helper function is in charge of sending a special event containing the call stack of other normal events from the four supported CLR providers. If you set the Stack keyword to the CLR provider, each time an event is sent by a thread, a ClrStackWalk event will be sent just after. It means after each SampleObjectAllocation event, a ClrStackWalk event containing the call stack will be immediately received. In fact, since an application will probably be using more than one thread, it is required to do the mapping between the two events on a per-thread basis.

Each allocation event received by the OnSampleObjectAllocation handler contains the ThreadID property so it is easy to keep track of the last received allocation event per thread. In my case, the ProcessAllocations class stores this information in its _perThreadLastAllocation field:

1
2
3
4
5


public class ProcessAllocations
{
    ...
    private readonly Dictionary<string, AllocationInfo> _allocations;
    private readonly Dictionary<int, AllocationInfo> _perThreadLastAllocation;

Now, each time a SampleObjectAllocation event is received, the id of the sending thread is passed to the updatedProcessAllocations.AddAllocation() method:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


public AllocationInfo AddAllocation(int pid, ulong size, ulong count, string typeName)
{
    if (!_allocations.TryGetValue(typeName, out var info))
    {
        info = new AllocationInfo(typeName);
        _allocations[typeName] = info;
    }

    info.AddAllocation(size, count);

    // the last allocation is still here without the corresponding stack
    if (_perThreadLastAllocation.TryGetValue(pid, out var lastAlloc))
    {
        Console.WriteLine("no stack for the last allocation");
    }

    // keep track of the allocation for the given thread
    // --> will be used when the corresponding call stack event will be received
    _perThreadLastAllocation[pid] = info;

    return info;
}

The _perThreadLastAllocation dictionary stores the AllocationInfo per thread. If an allocation happens, it is added into the dictionary. When a ClrStackWalk event is received for a given thread, the stack will be associated with the last AllocationInfo and removed from the dictionary. If some events are missed (it never happens during my tests but who knows), error message could be logged.

The ClrStackWalkTraceData argument received by the ClrStackWalk listener has a FrameCount property that returns the number of frames in the call stack. In addition, its InstructionPointer() method takes a frame position in the stack (starting at 0) and returns the address (in assembly code) at this position on the call stack.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


private void OnClrStackWalk(ClrStackWalkTraceData data)
{
    if (FilterOutEvent(data)) return;

    var callstack = BuildCallStack(data);
    GetProcessAllocations(data.ProcessID).AddStack(data.ThreadID, callstack);
}
private AddressStack BuildCallStack(ClrStackWalkTraceData data)
{
    var length = data.FrameCount;
    AddressStack stack = new AddressStack(length);

    // frame 0 is the last frame of the stack (i.e. last called method)
    for (int i = 0; i < length; i++)
    {
        stack.AddFrame(data.InstructionPointer(i));
    }

    return stack;
}

The AddressStack class returned by BuildCallStack stores the frames as a list of addresses so it can be stored in AllocationInfo.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36


public class AddressStack
{
    // the first frame is the address of the last called method 
    private readonly List<ulong> _stack;

    public AddressStack(int capacity)
    {
        _stack = new List<ulong>(capacity);
    }

    // No need to override GetHashCode because we don't want to use it as a key in a dictionary
    public override bool Equals(object obj)
    {
        if (obj == null) return false;

        var stack = obj as AddressStack;
        if (stack == null) return false;

        var frameCount = _stack.Count;
        if (frameCount != stack._stack.Count) return false;

        for (int i = 0; i < frameCount; i++)
        {
            if (_stack[i] != stack._stack[i]) return false;
        }

        return true;
    }

    public IReadOnlyList<ulong> Stack => _stack; 

    public void AddFrame(ulong address)
    {
        _stack.Add(address);
    }
}

This class overrides the Equals method for a single reason: I want to be able to detect when the “same” stack (i.e. with the exact same frame addresses) is received for a given type allocation. That way, I just need to keep a counter for each different AddressStack and not all call stacks in AllocationInfo. Remember that AllocationInfo is used to keep track of allocations per type:

1
2
3
4
5
6


public class AllocationInfo
{
    private readonly string _typeName;
    private ulong _size;
    private ulong _count;
    private List<StackInfo> _stacks;

The StackInfo class contains an AddressStack and how many times it led to this type of allocation.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


public class StackInfo
{
    private readonly AddressStack _stack;
    public ulong Count;

    internal StackInfo(AddressStack stack)
    {
        Count = 0;
        _stack = stack;
    }

    public AddressStack Stack => _stack;
}

So, when a stack event is received, AddStack is called on the last AllocationInfo for the same thread:

1
2
3
4
5
6
7
8
9


public void AddStack(int tid, AddressStack stack)
{
    if (_perThreadLastAllocation.TryGetValue(tid, out var lastAlloc))
    {
        lastAlloc.AddStack(stack);
        _perThreadLastAllocation.Remove(tid);
        return;
    }
}

The job of AllocationInfo.AddStack() the method is to check if a previous allocation was made with the same call stack (hence the Equals override). If this is the case, just increment the corresponding StackInfo count. Otherwise, create a new StackInfo for this call stack with a count set to 1.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


internal void AddStack(AddressStack stack)
{
    var info = GetInfo(stack);
    if (info == null)
    {
        info = new StackInfo(stack);
        _stacks.Add(info);
    }

    info.Count++;
}

private StackInfo GetInfo(AddressStack stack)
{
    for (int i = 0; i < _stacks.Count; i++)
    {
        var info = _stacks[i];
        if (stack.Equals(info.Stack)) return info;
    }

    return null;
}

Knowing the address in code of each frame for all events call stack is nice but it would be much more useful to translate them into method names… You have to deal with two different cases: managed and native methods. I will cover these topics in the next episode.

Resources

Source code available on Github.
TraceEvent sample 41 source code.

Missed the first part of this story? Check this out:

Build your own .NET memory profiler in C# *This post explains how to collect allocation details by writing your own memory profiler in C#.*medium.com

Interested in joining our journey? Check this out:

Product, Research & Development | Criteo Careers careers.criteo.com

Build your own .NET memory profiler in C# — Allocations (1/2)

Sat, 18 Apr 2020 09:48:53 +0000

In a previous post, I explained how to get statistics about the .NET Garbage Collector such as suspension time or generation sizes. But what if you would need more details about your application allocations such as how many times instances of a given type were allocated and for what cumulated size or even the allocation rate? This post explains how to get access to such information by writing your own memory profiler. The next one will show how to collect each sampled allocation stack trace.

Introduction

I have already used commercial tools to get detailed information about allocated type instances in an application; Visual Studio Profiler, dotTrace, ANTS memory profiler, or Perfview to name a few. With these tools in mind, I started to look at the .NET profiler API documentation and it reminded me the first time I read about the .NET profiler API. It was in December 2001 in Matt Pietrek’s MSDN Magazine article (I still have the paper version). When your application is starting, based on an environment variable, the .NET Framework (and now .NET Core) runtime is loading a profiler COM object that implements a specific ICorProfilerCallback interface (today, runtimes are supporting the 9th version ICorProfilerCallback9 interface). The methods of this interface will be called by the runtime at specific moments during the application lifetime. For example, the ObjectAllocated method is called each time an instance of a class is allocated: perfect for the job but it requires going back to COM and writing native code. Don’t be scared: I won’t go that way :^)

However, if you would like to get more details about writing your own .NET profiler in C or C++, I would recommend looking at the Microsoft ClrProfiler initial .NET Framework implementation and also Pavel Yosifovich DotNext session about Writing a .NET Core cross platform profiler in an hour with the corresponding (more recent and cross platform) source code.

Instead, several events that are emitted by the CLR are providing interesting details:

The GCSampledObjectAllocation events payload provides a type ID instead of a plain text type name. In order to retrieve the type name given its ID, we need to listen to TypeBulkType event that contains the mapping as I described in my post about finalizers. This is why the last two GCHeapAndTypeNames and Type keywords are needed.

Remember that if both GCSampledObjectAllocationLow and GCSampledObjectAllocationHigh keywords are set, an event will be received for EACH allocation. This could be a performance issue both for the monitored application and the profiler. I would recommend starting with either low or high (more on this later).

Last but not least, enabling at least one of these keywords is also switching the CLR to use “slower” allocators. This is why you should check that it does not impact your application performance. These slower allocators are also used when your ICorProfilerCallback.Initialize method calls SetEventMask with COR_PRF_ENABLE_OBJECT_ALLOCATED flag to receive allocation notifications.

When you use Perfview for memory investigation, you are relying on these events without knowing it. In the Collect/Run dialog, three checkboxes are defining how to get the memory profiling details:

.NET Alloc: use a custom native C++ ICorProfilerCallback implementation (noticeable impact on the profiled application performance).
.NET SampAlloc: use the same custom native profiler but with sampled events.
ETW .NET Alloc: use GCSampledObjectAllocationHigh events

In all cases, the profiled application needs to be started after the collection begins.

How to listen to allocation events

As I have already explained in previous posts, the Microsoft TraceEvent nuget helps you listening to CLR events. First, you create a TraceEventSession and setup the providers you want to receive events from:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


session.EnableProvider(
    ClrTraceEventParser.ProviderGuid,
    TraceEventLevel.Verbose,    // this is needed in order to receive AllocationTick_V2 event
    (ulong)(

    // required to receive AllocationTick events
    ClrTraceEventParser.Keywords.GC |

    // the CLR source code indicates that the provider must be set before the monitored application starts
    ClrTraceEventParser.Keywords.GCSampledObjectAllocationLow | 
    //ClrTraceEventParser.Keywords.GCSampledObjectAllocationHigh | 

    // required to receive the BulkType events that allows 
    // mapping between the type ID received in the allocation events
    ClrTraceEventParser.Keywords.GCHeapAndTypeNames |   
    ClrTraceEventParser.Keywords.Type |
);

Second, you set up the handlers for the events you are interested in:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


private void SetupListeners(TraceLogEventSource source)
{
    source.Clr.GCAllocationTick += OnAllocationTick;

    source.Clr.GCSampledObjectAllocation += OnSampleObjectAllocation;

    // required to receive the mapping between type ID (received in GCSampledObjectAllocation)
    // and their name (received in TypeBulkType)
    source.Clr.TypeBulkType += OnTypeBulkType;
}

And lastly, the processing of received events is done in a dedicated thread until the session is disposed of:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


await Task.Factory.StartNew(() =>
{
    using (_session)
    {
        SetupProviders(_session);
        SetupListeners(_session.Source);
        
        _session.Source.Process();
    }
});

Now let’s see the difference between the two sets of events.

The AllocationTick way

My first idea was to use the AllocationTick event because it seemed easy: one sampled event with a size, a type name, and LOH/ephemeral kind. However, how this sampling works makes it impossible to get an exact per type allocated size. Let’s have a look at this list of events received from a WPF test application:

Small | 105444 : FreezableContextPair[]
Small | 111908 : FreezableContextPair[]
Small | 106720 : System.String
Small | 102488 : System.String
Small | 107028 : System.TimeSpan[]
Small | 106100 : System.String

All allocations were small (i.e. not in the LOH: < 85.000 bytes) and the second column gives the cumulated size of all allocations to reach the 100 KB threshold but not for this particular type! There is no easy way to make a valid guess of the specific last allocation size for which we get the type name.

For example, the first array of FreezableContextPair triggered the event for a cumulated size of 105.444 bytes. But how big was this array? We don’t know: could have been 100.000 because only 5444 bytes were allocated before or only 10444 bytes because 95.000 were allocated before. It would have been so useful that the size of the last allocated object would be passed in the event payload…

It is a little bit different (but not that better) for objects allocated in LOH because they have to be at least 85.000 bytes long. For example, allocate 4-byte arrays, each one 85.000 bytes long and let’s see the corresponding events:

Large | 170064 : System.Byte[]
Large | 170064 : System.Byte[]

Two AllocationTick events are received with 170064 as cumulated size. Still hard to figure out what was the size of the last allocated array: the only thing we know is that it was larger (or equal) to 85.000 bytes because it was allocated in LOH.

For larger objects, it might seem a little bit more accurate. Let’s allocate 2 byte arrays, each one 110.000 bytes long:

Large | 195064 : System.Byte[]
Large | 110032 : System.Byte[]

There are ~85.000 bytes difference between the two events even though the same 110.000 bytes were allocated. You could remove 85.000 bytes from the value and have an approximation of the LOH allocated object: the larger the allocation the less the error. But still: could be 85.000 size error…

So we won’t be able to rely on the size provided by the AllocationTick event; only the type name. In addition, you get a view of objects allocated in LOH. Maybe the other events will provide better results.

The GCSampledObjectAllocation way

When an object is allocated by the GC allocator, a GCSampledObjectAllocation event is emitted under certain conditions:

Both GCSampledObjectAllocationLow and GCSampledObjectAllocationHigh keywords are set on the CLR provider,
The object size is larger than 10.000 bytes,
At least 1000 instances of the type have been allocated,
Just before the application exits, current statistics for all types are flushed,
A complicated piece of code decides based on time since the last event and the type allocation rate.

Picking one or the other keyword changes the maximum number of milliseconds between two events for a given type:

High (10 ms) : 100 events / second
Low (200 ms) : 5 events / second

You should use low or high depending on the monitored application memory allocation workload to avoid impacting too much the profiler (and even the monitored application performance)

The interesting feature of these events is that, for a given type, the payload contains both the number of allocated instances since the last event and the cumulated size of these instances. Let’s take the same allocation of 4 arrays of byte, each 85000 long:

226 | 103616 : System.Byte[]
  1 |  85012 : System.Byte[]
  1 |  85012 : System.Byte[]
  1 |  85012 : System.Byte[]

This time, we get the exact count in the first column (ObjectCountForTypeSample) and the exact cumulated size in the second column (TotalSizeForTypeSample). If the count is 1, we have the exact size of that allocation and if it is bigger than 85000 bytes, we know it has been allocated in the LOH. Same accuracy for the 2-byte array of 110.000 elements:

198 | 123552 : System.Byte[]
  1 | 110012 : System.Byte[]

Sounds good. However, you have to remember that profiled applications need to be started after the session was created: it means that you can’t write a tool that will listen to a specific process ID like with AllocationTick. Three dictionaries are used by PerProcessProfilingState to keep track of per type allocations, type ID mappings, and process names:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


public class PerProcessProfilingState
{
    private readonly Dictionary<int, string> _processNames = new Dictionary<int, string>();
    private readonly Dictionary<int, ProcessTypeMapping> _perProcessTypes = new Dictionary<int, ProcessTypeMapping>();
    private readonly Dictionary<int, ProcessAllocationInfo> _perProcessAllocations = new Dictionary<int, ProcessAllocationInfo>();

    public Dictionary<int, string> Names => _processNames;
    public Dictionary<int, ProcessTypeMapping> Types => _perProcessTypes;
    public Dictionary<int, ProcessAllocationInfo> Allocations => _perProcessAllocations;
}

The SampledObjectAllocationMemoryProfiler class uses it for the events processing:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


public class SampledObjectAllocationMemoryProfiler
{
    private readonly TraceEventSession _session;
    private readonly PerProcessProfilingState _processes;
    
    // because we are not interested in self monitoring
    private readonly int _currentPid;

    private int _started = 0;

    public SampledObjectAllocationMemoryProfiler(TraceEventSession session, PerProcessProfilingState processes)
    {
        _session = session;
        _processes = processes;
        _currentPid = Process.GetCurrentProcess().Id;
    }

The constructor of the profiler keeps track of its own process ID in _currentPid to skip its own events.

Gathering type mapping

The processing of TypeBulkType events is quite straightforward: store the type ID/name association into a per-process dictionary:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


private void OnTypeBulkType(GCBulkTypeTraceData data)
{
    if (FilterOutEvent(data)) return;

    ProcessTypeMapping mapping = GetProcessTypesMapping(data.ProcessID);
    for (int currentType = 0; currentType < data.Count; currentType++)
    {
        GCBulkTypeValues value = data.Values(currentType);
        mapping[value.TypeID] = value.TypeName;
    }
}

private ProcessTypeMapping GetProcessTypesMapping(int pid)
{
    ProcessTypeMapping mapping;
    if (!_processes.Types.TryGetValue(pid, out mapping))
    {
        AssociateProcess(pid);

        mapping = new ProcessTypeMapping(pid);
        _processes.Types[pid] = mapping;
    }
    return mapping;
}

Remember that I choose to skip events from the current process detected by FilterOutEvent().

How to get process names

Even though each event contains the ID of the emitting process, it would be better to display its name instead. You could use Process.GetProcessById(pid).ProcessName when analyzing the details but the process might be long gone at that time.

Another solution would be to enable the Kernel ETW provider and listen to the ProcessStart event. The ImageFileName field of the payload contains the process filename with the extension. However, it is obviously not working on Linux.

The easiest solution is to use GetProcessById but just when you receive the first type mapping for a given process. This is the role of the AssociateProcess method called in GetProcessTypesMapping shown previously:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


private void AssociateProcess(int pid)
{
    try
    {
        _processes.Names[pid] = Process.GetProcessById(pid).ProcessName;
    }
    catch (Exception)
    {
        Console.WriteLine($"? {pid}");
        // we might not have access to the process
    }
}

It is now time to process allocation events.

Collecting allocation details

The GCSampledObjectAllocationTraceData payload contains the size and count of instances since the last event. We just need to store them for the corresponding process:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


private void OnSampleObjectAllocation(GCSampledObjectAllocationTraceData data)
{
    if (FilterOutEvent(data)) return;
    
    GetProcessAllocations(data.ProcessID)
        .AddAllocation(
            (ulong)data.TotalSizeForTypeSample, 
            (ulong)data.ObjectCountForTypeSample, 
            GetProcessTypeName(data.ProcessID, data.TypeID)
            );
}
private string GetProcessTypeName(int pid, ulong typeID)
{
    if (!_processes.Types.TryGetValue(pid, out var mapping))
    {
        return typeID.ToString();
    }

    var name = mapping[typeID];
    return string.IsNullOrEmpty(name) ? typeID.ToString() : name;
}

The AddAllocation() helper method is simply accumulating these numbers for a given type in the ProcessAllocationInfo associated to the related process.

Displaying the results

When the profiling session ends, it is easy to show the allocated count and size per type:

The code is using a Linq syntax to get top allocations sorted either by count or by size:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


private static void ShowResults(string name, ProcessAllocationInfo allocations, bool sortBySize, int topTypesLimit)
{
    Console.WriteLine($"Memory allocations for {name}");
    Console.WriteLine();
    Console.WriteLine("---------------------------------------------------------");
    Console.WriteLine("    Count        Size   Type");
    Console.WriteLine("---------------------------------------------------------");
    IEnumerable<AllocationInfo> types = (sortBySize)
        ? allocations.GetAllocations().OrderByDescending(a => a.Size)
        : allocations.GetAllocations().OrderByDescending(a => a.Count)
        ;
    if (topTypesLimit != -1)
        types = types.Take(topTypesLimit);

    foreach (var allocation in types)
    {
        Console.WriteLine($"{allocation.Count,9} {allocation.Size,11}   {allocation.TypeName}");
    }
}

Another usage could be a long-running monitoring system that shows the allocation rate: a nice complement to the other GC metrics. However, compared to the other profilers, one important feature is missing: if an unexpected number of instances are created, how to know which part of the code is responsible for the spike?

The next post will explain how to enhance such a sampled memory profiler with call stacks per sampled allocation.

Resources

Source code available on Github.
Spying on .NET Garbage Collector with TraceEvent
Pavel Yosifovich — Writing a .NET Core cross-platform profiler in an hour
Original Microsoft ClrProfiler source code and documentation

Like what you read? Don’t forget to check out part 2 on this topic:

Build your own .NET memory profiler in C# — call stacks (2/2–1) *This post explains how to get the call stack corresponding to the allocations with CLR events.*medium.com

Interested in joining our journey? Check this out:

Product, Research & Development | Criteo Careers *Product, Research & Development at Criteo. At Criteo, come and meet our teams and join our R & D and also enjoy…*careers.criteo.com

Debugging Wednesday at Criteo — Cancel this task!

Fri, 21 Feb 2020 08:18:08 +0000

Introduction

Last Wednesday was a great day for Kevin and myself: We spent a lot of time investigating the reasons why a test was failing. Let’s share with you these frustrating but interesting minutes.

One of our colleagues came to us because an integration test would get stuck in some specific conditions. Here is the simplified code of the service that is supposed to do some background processing until it is stopped:

1
2
3
4
5
6
7
8


public class Service
{
    private CancellationTokenSource _cancellationSource;
    private Task _backgroundProcessing;
    private Task _cleanup;

    ...
}

In the test, the service is created: Two Task instances are created in the constructor (for background processing irrelevant for this discussion) that (1) are started when the service starts and (2) are canceled when the service is stopped. A CancellationTokenSource is used to cancel the tasks if needed.

1
2
3
4
5
6
7
8
9


public Service()
{
    _cancellationSource = new CancellationTokenSource();
    var cancellationToken = _cancellationSource.Token;
    _backgroundProcessing = new Task(DoStuffInTheBackground,
        cancellationToken, TaskCreationOptions.LongRunning);
    _cleanup = new Task(DoStuffInTheBackground, 
        cancellationToken, TaskCreationOptions.LongRunning);
}

However, in the specific conditions of this test, the Start method would never be called, skipping straight to the Stop method.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


public void Start()
{
    _backgroundProcessing.Start();
    _cleanup.Start();
}

public async Task Stop()
{
    _cancellationSource.Cancel();
    await _backgroundProcessing;
    await _cleanup;
}

The task is never started because the test skips the Service.Start() method, but it should transition to “Cancelled” state as soon as the CancellationTokenSource associated with the CancellationToken passed to the Task gets canceled. In that particular situation, we expect the await _backgroundProcessing code to immediately return in the Stop() method.

In our colleague Visual Studio, the debugger never came back from await _backgroundProcessing, indicating that the task never completed…

Reproduce the problem

I wanted to un-validate any side effect related to our test framework or custom Criteo libraries and confirm my understanding of task cancellation, so I wrote a small Console application with the same 4.7.2 version of .NET Framework with the following code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


static async Task ReproduceWorking()
{
    var source = new CancellationTokenSource();
    var token = source.Token;
    var t = new Task(
        () => Console.WriteLine("Done"), token, TaskCreationOptions.LongRunning);

    source.Cancel();

    try
    {
        await t;
    }
    catch (TaskCanceledException x)
    {
        Console.WriteLine(x.ToString());
    }
}

And guess what? I got the expected TaskCancellationException:

System.Threading.Tasks.TaskCanceledException: A task was canceled. at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.GetResult() at TaskCancellation.Program.d__2.MoveNext()

The next step was to double-check the understanding of how tasks are working related to cancellation. So I set a breakpoint after the task is instantiated

And its status is Created.

Doing the same after Cancel() is called on the CancellationTokenSource, gives the expected Canceled status.

So, let’s do the same when debugging the test code:

It gives the same Created status after the task creation but after canceling the source:

…the status does not switch to Canceled like in my Console repro!

Looking for cancellation

The only thing that came to my mind was: maybe the cancellation token is not taken into account. However, a CancellationToken is just a struct that keeps a reference to its CancellationTokenSource. It should be easy in Visual Studio debugger to double-check that our task keeps track of the token somewhere and goes back to the cancellation source. Well… the token is not kept as a field of the task but stored inside the m_contingentProperties field deep during the task construction code path.

Let’s look at the value in the Quick Watch after the cancellation source gets canceled:

It sounds like the cancellation token is not canceled… But if we look at the source Token property of our canceled CancellationTokenSource,

we don’t have the same value for IsCancellationRequested!

It’s like we don’t look at the same cancellation source… To find out, we just have to compare the reference to our CancellationTokenSource with the one we see in the m_contingentProperties token of the task. To achieve that, we could copy the expression from QuickWatch, paste it into the Debug | Windows | Memory… pane and press ENTER to get the address where the object is stored in memory

After having done the same with _cancellationSource, I did not get the same address:

It means that we were dealing with two different instances of CancellationTokenSource.

But this might be too C++ish for you… Kevin prefers leveraging the Make Object ID feature of the C# debugger. You simply right-click the Data Tip of the cancellation source and select Make Object ID:

Once this is done, a numeric identifier is displayed for this instance in Data Tip and any Watch window (#1 in this screenshot):

When we looked at the cancellation source of the token stored by the task,

we didn’t see any ID so it was not the same object.

So we decided to use Make Object ID on this m_source that became marked as #2.

And when we looked at the m_source of the second cleanup Task, we realized that it was the same object but not the one we created!

We started to think that we were becoming crazy. So, let’s restart from the beginning and follow the CancellationTokenSource from its creation because we are sure that we passed a valid cancellation token (linked to this source) to the task constructor. Or… Did we? The QuickWatch gives a different answer just after the task gets created compared to what we’ve seen already: a token with an empty source property now!

I closed the QuickWatch pane and reopened it for Kevin to confirm. And like in a nightmare, the source was not null anymore…

Visual Studio must be responsible for that weird behavior!

Kevin remembered the ”Enable property evaluation” settings in the Options dialog. If it is checked (which is the default), it means that the Debugger would fetch the value of an instance field and then call the Getter of each property in order to display its value.

However, if you uncheck it, only the fields are displayed. So in our case, we then always got a null m_contingentProperties field (and, as expected, all property would not be displayed):

The m_contingentProperties is initialized in EnsureContingentPropertiesInitialized() when called by AssignCancellationToken() from the TaskContructorCore() helper used by the Task constructor but it did not seem to be the case because it was definitively null…

Kevin decided to stop at the CancellationTokenSource constructor with a new breakpoint (more on how to set a breakpoint on a .NET Framework method soon) to see where the one shown in the Debugger was created but the breakpoint was never hit. So the CancellationTokenSource #2 must have been created even before our own was created by our code. In fact, a static CancellationTokenSource is created and is set to m_source when InitializeDefaultSource() gets called by one of the Getter. This explains why we saw the same instance #2 in both tasks token.

To sum up, we were now sure that the passed token was not “received” by the Task.

Eureka!

Maybe there is a magic trick done by the .NET Framework to lazily set the token source after the creation of the task. However, we did not find such a code in the .NET Framework and this is not what we see in our repro.

Back to the basics: are we sure that we are executing the code we think is executed? We looked for mscorlib in the Debug | Windows | Modules pane,

and we opened it with a decompiler: the code of the methods called during the Task** **construction was the same as the one shown in https://referencesource.microsoft.com/#mscorlib/system/threading/Tasks/Task.cs.

Next, in order to better follow the execution and the passing of parameters (including our token), we decided to set breakpoints on Task private method responsible for its initialization.

internal void TaskConstructorCore(object action, object state, CancellationToken cancellationToken, TaskCreationOptions creationOptions, InternalTaskOptions internalOptions, TaskScheduler scheduler)

In the Debug | Windows | Breakpoints pane, click New | Function Breakpoint…

and type the full name of the method. This is working even for a method of a class defined in the .NET Framework assembly for which you do not have the source code:

We checked that the breakpoints were well set (i.e. no typo in the full name) by looking at the filled red circle:

The cancellationToken parameter should contain the token that we passed at Task creation. Unfortunately, the QuickWatch pane displayed a “cannot read memory” error that we never saw in Visual Studio before!

At that time, we thought we were doomed but we looked at the Call Stack pane and we realized that the code was calling the wrong Task constructor:

public Task(Action action, object state, TaskCreationOptions creationOptions)

Its signature is compatible with our code:

_backgroundProcessing = new Task(DoStuffInTheBackground, cancellationToken, TaskCreationOptions.LongRunning);

and this is why the compiler did not complain.

Our CancellationToken was passed as the state parameter is given directly to our DoStuffInTheBackground Action: the created Task had no idea that it was supposed to be its CancellationToken.

Note that if we had noticed the Auto Completion (Ctrl + Shift + Space) hint, we might have figured out the root cause much sooner…

The fix was straightforward; just using the right constructor:

public Task(Action action, object state, CancellationToken cancellationToken, TaskCreationOptions creationOptions)

that accepts both a state for the callback and a CancellationToken for the Task to create:

_backgroundProcessing = new Task(DoStuffInTheBackground, cancellationToken, cancellationToken, TaskCreationOptions.LongRunning);

Under the debugger, we validated that the source was now the expected one:

and the test did not hang anymore.

If, from the beginning, we would have been able to step into .NET Framework compiled code as we do with Jetbrains Resharper integration in Visual Studio, we would have found the issue almost immediately. Thankfully, Microsoft has just announced decompilation of C# code made easy with Visual Studio.

We wish we had it last Wednesday…

Interested in reading more about Christophe’s & Kevin’s work? Check out their latest articles:

Build your own .NET memory profiler in C# *This post explains how to collect allocation details by writing your own memory profiler in C#.*medium.com Switching back to the UI thread in WPF/UWP, in modern C# Leveraging the async machinery to transparently switch to the UI thread when neededmedium.com

If you are looking for a change and would love to work with these two, head over to our careers page and let us know if there is something that sounds like you!

Product, Research & Development | Criteo Careers *Come and meet our teams …*careers.criteo.com

WSL + Visual Studio = attaching/launching a Linux .NET Core application on my Window 10

Thu, 21 Nov 2019 15:57:01 +0000

This post shows how to attach to a .NET Core process running on Linux with WSL and also how to start a Linux process with Visual Studio debugger

Coming from the Windows world, I don’t find that easy to develop .NET Core applications for Linux. I’m used to code and debug in Visual Studio. Now, I need to build on Windows (due to our Criteo continuous integration), deploy an artifact to Marathon in order to get an application running inside a Mesos container. At Criteo, we had to build a whole set of services to allow remote debugging or memory dump analysis.

But what if I just wanted to test and debug a small scenario on my beloved Windows machine? Windows Subsystem for Linux (aka WSL) is perfect for running a Linux .NET Core application on Windows. However, how to attach to it or even start a debugging session from Visual Studio? In the rest of this post, I’ll explain how to setup your Windows 10 machine to achieve these miracles.

Linux on Windows: welcome WSL

It is obviously not the place to dig into Windows Subsystem for Linux. You just need to know that once installed, it allows you open up a Linux shell on your Windows machine without any virtual machine kind of technology. It is also possible to share folders between Windows (where you want to build your application) and Linux (where you want to execute the built assemblies).

The first step is to turn on WSL feature in your Windows:

After a reboot, you are able to start a WSL prompt:

You can also install your favorite Linux distro if you want.

The next step is to install .NET Core runtime or SDK so you are able to run your application on Linux.

It is now time to look at your hard drive from a Linux perspective:

Your drives are mapped under /mnt without the “:”. In my case, I created a wsl folder under my d: drive to do my experiments. With Visual Studio, I generated a TestConsole application with the following code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


using System;
 
namespace TestConsole
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Enter x to EXIT...");
            while(true)
            {
                var cmd = Console.ReadLine();
                if (cmd.ToLower() == "x")
                    return;
 
                Console.WriteLine($"> {cmd}");
            }
        }
    }
}

I’m publishing the application to get all needed assemblies:

In a WSL prompt, type dotnet with the name of your application assembly et voila!

A Linux .NET Core application is running on your Windows machine.

How to attach to a running Linux application

Let’s say that I’m detecting a problem and I want to debug the Linux application with Visual Studio. When attaching the Visual Studio debugger to a process, several connection types are available:

The SSH connection type will be used with WSL with the following kind of communications architecture:

The Visual Studio debugger is sending commands to the remote Linux debugger vsdbg via an SSH channel. Here are the steps to follow to install the missing components:

By default, an SSH server is installed with WSL. However, I was not able to make the whole pipeline work with it so I had to uninstall and reinstall it: sudo apt-get remove openssh-server sudo apt-get install openssh-server
The SSH configuration needs also to be changed in order to allow username/password kind of security needed by Visual Studio (if you prefer key-based security, look at the end of the post for available resources). If you don’t know how to use vi efficiently to simply edit a file, install nano (thanks @kookiz for the tips :^) sudo apt-get install nano
In /etc/ssh/sshd_config, change the PasswordAuthentication settings sudo nano /etc/ssh/sshd_config PasswordAuthentication yes
Restart the ssh server sudo service ssh start
You need to install unzip in order to get vsdbg sudo apt-get install unzip curl -sSL https://aka.ms/getvsdbgsh | bash /dev/stdin -v latest -l ~/vsdbg

You are now ready to select SSH as connection type and enter your machine name before clicking the Refresh button. At that time, a new dialog should pop up for you to enter your WSL credentials:

After you click the Refresh button, the list at the bottom should contain the Linux processes running in WSL

Select your .NET Core application and click Attach to select the Managed debugger:

Now, if you set a breakpoint in the code and trigger it with an appropriate action in your WSL prompt

then the debugger will break as expected:

Note that when you stop your debugging session, the Linux application is not stopped; just detached from the debugger and keeps on running.

Showtime for F5!

Attaching to a running Linux application is nice but it would be even better to start a Linux process from Visual Studio debugger. In order to achieve this goal you need to add another piece to the architecture puzzle:

As detailed in this WIKI page, it is possible to tell Visual Studio to execute debugging actions thanks to a launch.json file such as the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


{
  "version": "0.2.0",
  "adapter": "c:\\tools\\plink.exe",
  "adapterArgs": "-ssh -pw  chrisnas@DBGDQPZ1 -batch -T ~/vsdbg/vsdbg --interpreter=vscode",
  "configurations": [
    {
      "name": ".NET Core Launch",
      "type": "coreclr",
      "cwd": "/mnt/d/wsl/TestConsole/TestConsole/bin/Debug/netcoreapp2.1/publish",
      "program": "TestConsole.dll",
      "request": "launch"
    }
  ]
}

The plink tool from putty will be used as an adapter for Visual Studio to communicate with vsdbg running in WSL. The adapterArgs property gives the same SSH/machine/user/password information that you provide via Visual Studio UI in the Attach scenario. The configuration section defines which request (“launch” instead of attach and which folder/assembly to start) will be sent to vsdbg.

Once this file is created (in my case in d:\wsl\vs folder), you just need to type the following command in the Immediate pane of Visual Studio:

DebugAdapterHost.Launch /LaunchJson:d:\wsl\vs\launch.json

and if you had set a breakpoint on the first line of your application, the debugger should break there:

In a WSL prompt, you can see the expected 2 new spawned processes:

But wait!

I have a problem now: I don’t have any prompt in which typing input for my console application… However, as always in Linux, you simply need to write to a file to fix this. The stdin stream of your application is accessible under /proc//fd/0.

So, when I type the following command:

echo "Launching a Linux app is not a problem!" > /proc/18341/fd/0

my breakpoint is hit in Visual Studio:

Also note that everything that is sent to the console appears in Visual Studio Output pane:

Note that, unlike the Attach scenario, if you stop the debugging session, the Linux application (and vsdbg) will be terminated.

Resources

During my investigations I’ve found a few resources you might find useful (especially about configuring SSL with keys instead of passing clear user/password)

Basic VS + WSL
Good description about how to use VS 2017 to attach to .NET Core app running in WSL
Debugging .NET Core from VS 2017 and WSL
VS/C++ with WSL (describe how to install WSL and setup open ssh server)
Setup SSH on WSL
Wiki for VSDBG
Great description about how to setup your linux dev environment with VS Code and WSL

How to expose your custom counters in .NET Core

Thu, 17 Oct 2019 12:42:17 +0000

This post of the series explains how to implement your own counters.

Part 1: Replace .NET performance counters by CLR event tracing.

Part 2: Grab ETW Session, Providers and Events.

Part 3: CLR Threading events with TraceEvent.

Part 4: Spying on .NET Garbage Collector with TraceEvent.

Part 5: Building your own Java GC logs in .NET

Part 6: Spying on .NET Core Garbage Collector with .NET Core EventPipes

Part 7: .NET Core Counters internals: how to integrate counters in your monitoring pipeline

Introduction

The** EventPipe** counters are the .NET Core replacement for Windows performance counters. In the previous post, I’ve explained how to listen to CLR event pipes to get the counter’s value over time both on Windows and Linux. This post shows you how easy it is to provide your counters via the same infrastructure.

The example I’m using is based on a real-world case we had to investigate at Criteo. We needed to correlate request duration with garbage collections, so we decided to add new metrics to our testing dashboard: number and duration of requests but split between those processed without being interrupted by a GC and the others.

For the sake of the ASP.NET Core code example, a dedicated middleware is created: it simply measures the time spent to process a request and if the count of garbage collections has changed before and after the request is processed:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51


public class RequestMetricsMiddleware
{
    private readonly RequestDelegate _next;

    public RequestMetricsMiddleware(RequestDelegate next)
    {
        _next = next;
    }

    public async Task InvokeAsync(HttpContext context)
    {
        // get the count of GCs before processing the request
        var collectionCountBeforeProcessingTheRequest = GetCurrentCollectionCount();

        var sw = Stopwatch.StartNew();

        try
        {
            // Call the next delegate/middleware in the pipeline
            await _next(context);
        }
        finally
        {
            // compare the counter of GCs after processing the request
            // if the count changed, a garbage collection occurred during the processing 
            // and might have slowed it down and maybe reaching SLA limit: this could 
            // explain 9x-percentile in slow requests for example
            if (GetCurrentCollectionCount() - collectionCountBeforeProcessingTheRequest != 0)
            {
                // update with collection metric
               RequestCountersEventSource.Instance.AddRequestWithGcDuration(sw.ElapsedMilliseconds);
            }
            else
            {
                // update without collection metric
                RequestCountersEventSource.Instance.AddRequestWithoutGcDuration(sw.ElapsedMilliseconds);
            }
        }
    }

    private int GetCurrentCollectionCount()
    {
        int count = 0;
        for (int i = 0; i < GC.MaxGeneration; i++)
        {
            count += GC.CollectionCount(i);
        }

        return count;
    }
}

The interesting part is in the RequestCountersEventSource implementation.

Use an EventSource Luke!

As explained in the previous post, an EventSource instance is used as the “server” part of the EventPipe communication channel. It exposes a name that is used to identify it, but more important to listen to it with dotnet-trace, dotnet-counters, or your own listener as the provider name.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


[EventSource(Name = RequestCountersEventSource.SourceName)]
public class RequestCountersEventSource : EventSource
{
    // this name will be used as "provider" name with dotnet-counters
    // ex: dotnet-counters monitor -p  Sample.RequestCounters
    //
    const string SourceName = "Sample.RequestCounters";

    public RequestCountersEventSource() 
        : base(RequestCountersEventSource.SourceName, EventSourceSettings.EtwSelfDescribingEventFormat)
    {
        // create the counters: they'll be bound to this event source + CounterGroup
        CreateCounters();
    }

This name is exposed via an EventSourceAttribute that decorates your EventSource-derived class (you could also pass it to the constructor). The counters are created in the constructor through the CreateCounters helper.

Pick the right Counter class

Before looking at the implementation of the CreateCounters method, you need to understand what kind of counters are available for you. In the previous post, I mentioned that the CLR was using Mean (that provides mean, max, and min values over the update interval) and Sum (to increment a single value) kinds of counters. Note that dotnet-counter will only show the mean value for Mean counters. In addition, the counters could either automatically poll the value from a callback (the method used by the CLR today), or your code could change a counter value by calling the WriteMetric method. The EventCounter class provides this helper and it does its best to compute the min/max/mean in a lock-free way.

The next question to answer is which one should you use.

In the case of the request with(out) GC example, I want to expose different metrics:

Request count: a PollingCounter will be used in addition to an int field incremented when a request is received.
Request count delta: an IncrementingCounter associated with the same int value will provide the delta (i.e., number of requests processed during an interval)
Request with GC and without GC counts: two PollingCounter instances based on two int fields incremented when a request with (or without respectively) GC are processed.
Duration of requests with and without GC: two EventCounter instances updated when requests are processed.

Here is the implementation ofCreateCounters:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


private void CreateCounters()
{
    // the same request count can be used for two counters:
    // - raw request counter that will always increase
    // - increment counter that will automatically compute the delta
    //   between the current value and the value when the counter
    //   was previously sent
    _requestCount ??= new PollingCounter("request-count", this,
        () => _requestCountValue)
    { DisplayName = "Requests count" };
    _requestCountDelta ??= new IncrementingPollingCounter("request-count-delta", this,
        () => _requestCountValue)
    { DisplayName = "New requests", DisplayRateTimeScale = new TimeSpan(0, 0, 1) };

    // split the request counts between those for which a GC occured or not 
    // during their processing
    _noGcRequestCount ??= new PollingCounter("no-gc-request-count", this,
        () => _noGcRequestCountValue)
    { DisplayName = "Requests (processed without GC) count" };
    _withGcRequestCount ??= new PollingCounter("with-gc-request-count", this,
        () => _withGcRequestsCountValue)
    { DisplayName = "Requests (processed during a GC) count" };

    // request duration counters (with or without GC happening during the processing)
    _noGcRequestDuration ??= new EventCounter("no-gc-request-duration", this)
    { DisplayName = "Requests (processed without GC) duration in milli-seconds" };

    _withGcRequestDuration ??= new EventCounter("with-gc-request-duration", this)
    { DisplayName = "Requests (processed during a GC) duration in milli-seconds" };
}

The request processing code of the ASP.NET Core middleware is relying on the following helper methods to update the counters:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


internal void AddRequestWithoutGcDuration(long elapsedMilliseconds)
{
    IncRequestCount();
    Interlocked.Increment(ref _noGcRequestCountValue);

    // compute min/max/mean
    _noGcRequestDuration?.WriteMetric(elapsedMilliseconds);
}

internal void AddRequestWithGcDuration(long elapsedMilliseconds)
{
    IncRequestCount();
    Interlocked.Increment(ref _withGcRequestsCountValue);

    // compute min/max/mean
    _withGcRequestDuration?.WriteMetric(elapsedMilliseconds);
}

private void IncRequestCount()
{
    Interlocked.Increment(ref _requestCountValue);
}

And that’s it!

How to get these custom counters?

The controller of the ASP.NET Core sample application is triggering (or not) garbage collections based on the parameters passed via the url:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


[Route("api/[controller]")]
[ApiController]
public class RequestController : ControllerBase
{

    // GET: api/Request/5
    [HttpGet("{id}")]
    public string Get(int id)
    {
        if (id == -1)
        {
            return $"pid = {Process.GetCurrentProcess().Id}";
        }
        else
        if ((id >= 0) && (id <= 2))
        {
            GC.Collect(id);
            return $"triggered GC {id}";
        }
        else
        if (id <= 10)
        {
            // trigger a given number of GCs up to 10
            TriggerGCs(id);
            return $"triggered {id} garbage collections";
        }

        return $"value = {id}";
    }

    private void TriggerGCs(int count)
    {
        for (int current = 0; current < count; current++)
        {
            GC.Collect(0);
        }
    }

As explained earlier, it is possible to see the counter values with dotnet-counters by using the event source name as a provider with the following command line:

dotnet counters monitor -p Sample.RequestCounters

Then if you trigger a few requests with and without GC, you should see the numbers change:

The code available on Github has been updated to provide the middleware and the event source classes that demonstrate how to expose custom .NET Core counters.

.NET Core Counters internals: how to integrate counters in your monitoring pipeline

Tue, 23 Jul 2019 15:30:37 +0000

This post of the series digs into the implementation details of the new .NET Core counters.

Part 1: Replace .NET performance counters by CLR event tracing.

Part 2: Grab ETW Session, Providers and Events.

Part 3: CLR Threading events with TraceEvent.

Part 4: Spying on .NET Garbage Collector with TraceEvent.

Part 5: Building your own Java GC logs in .NET

Part6: Spying on .NET Core Garbage Collector with .NET Core EventPipes

Introduction

As explained in a previous post, .NET Core 2.2 introduced the EventListener class to receive in-proc CLR events both on Windows and Linux. Starting with .NET Core 3.0 Preview 6, the EventPipe-based infrastructure makes it now possible to get these events from another process. The diagnostics repository contains the cross-platform tools leveraging this infrastructure:

dotnet-dump: take memory snapshot and allow analysis based on most SOS commands
dotnet-trace: collect events emitted by the Core CLR and generate trace file to be analyzed with Perfview
dotnet-counters: collect the metrics corresponding to some performance counters that used to be exposed by the .NET Framework

At Criteo, our metrics are exposed in Grafana dashboards and it is interesting to figure out how the new counters are implemented and see how to fetch them via the EventPipe infrastructure. With this knowledge in hand, I’ve implemented helpers to let you get counters in less than 10 lines of code:

1
2
3
4
5
6
7


_counterMonitor = new CounterMonitor(_pid, GetProviders());
_counterMonitor.CounterUpdate += // receive the value of one counter after the other

Task monitorTask = new Task(() => {
    _counterMonitor.Start();
});
monitorTask.Start();

At the end of this post you will be able to very easily integrate any counter to your own monitoring pipeline!

.NET Core replacement for .NET Framework Performance Counters

With .NET Core being cross-platform, performance counters were gone and, as explained in the previous posts of the series, CLR events were the only way to get metrics about how your .NET Core applications were behaving. However, with .NET Core 3.0, it is now possible to view a few metrics thanks to the dotnet-counters tool.

You can download and install the tools automatically if you have installed .NET Core SDK 2.1+. Microsoft is currently working to provide other ways to directly download the tools binaries without having to install the SDK or recompile the diagnostics repository.

Use the following command line to install dotnet-counters: dotnet tool install --global dotnet-counters --version 3.0.0-preview7.19365.2

Note that you need to have the same version both for the Core CLR runtime and for the tools because, as you will soon see, the monitoring and the monitored applications are communicating via a dedicated protocol (that have changed between previews) on top of a transport layer different between Windows and Linux.

After the installation, use the following command line dotnet counters monitor -p and you get a 1 second auto-refreshed view of counters.

These counters are exposed by the System.Runtime provider and are detailed with the list argument:

This list is currently hard-coded in the CreateKnownProviders method. However, you are free to create your own provider and expose your application metrics as shown in this tutorial (and in the next forthcoming post). In addition, if you are using ASP.NET Core, starting from Preview 7, then you could get a few counters from the “Microsoft.AspNetCore.Hosting” provider defined in HostingEventSource.cs.

What are these “counters”

Even though it is nice to have a console-based cross-platform tool to see the values of counters change, what would be the cost to get them into your own monitoring pipeline? For example, at Criteo, we are pushing our metrics to Graphite in order to get nice Grafana dashboards. These graphical representations allow us to have a visual representation of the evolution of metrics over time. In addition, it is also possible to define alerts based on threshold for some metrics values (when CPU > 85% for more than 5 seconds for example).

In a nutshell, dotnet-counters tool is listening to another application via EventPipe. Unlike .NET Framework performance counters that are polled by the monitoring application, the counters are pushed by the monitored .NET Core process.

In term of implementation, these counters are values that you could get via .NET internal or public APIs if you were running in-proc as shown in RuntimeEventSource.cs:

Unlike most of the events that previous posts of this series presented, counters are metrics that are computed by the CLR in the monitored application. They are supposed to provide a set of values changing over time in the monitored application without impacting the performance nor flooding the listener client. I highly recommend to take a look at this issue for a deeper discussion about EventCounters compared to regular events.

As of Preview 7, two types of counters are used:

Mean: supposed to contain a mean of all values during the polling interval with its min and max values. However, based on the current implementation, all contain only the current value.
Sum: contains an increment between the previous value and the current one

The question is now to figure out how to get the values of the counters.

How to receive the counters?

Like the Perfview tool that relies on TraceEvent library, dotnet-counters uses an API exposed by Microsoft.Diagnostics.Tools.RuntimeClient assembly. Note that it is currently not (yet) available from nuget so you need to recompile it with the diagnostics git repo.

To receive counters, you need to create an EventPipe session that communicates via IPC (named pipes on Windows and domain sockets on Linux) with the CLR of the monitored process. Here is an excerpt of the CounterMonitor.StartMonitoring implementation that connects and listens to counter events:

1
2
3
4
5
6
7
8
9


var configuration = new SessionConfiguration(
    circularBufferSizeMB: 1000,
    outputPath: "",
    providers: Trace.Extensions.ToProviders(providerString));

var binaryReader = EventPipeClient.CollectTracing(_processId, configuration, out _sessionId);
EventPipeEventSource source = new EventPipeEventSource(binaryReader);
source.Dynamic.All += ProcessEvents;
source.Process();

The important method call is call is EventPipeClient.CollectTracing() that returns a Stream from which an EventPipeEventSource instance gets created. This class has been added to TraceEvent so you can now leverage the event parsing infrastructure on top of EventPipe! As shown in a previous post, it is easy to attach a listener to the source All .NET event and get notified each time an event is received after the Process method is called.

A few parameters are given to CollectTracing via the SessionConfiguration object: the size of the circular buffer used by the CLR and no file path because we want a live session. The last one is supposed to filter which providers and counters you would like to listen to: it expects a list of Provider instances. This struct is created with a few parameters:

1
2
3
4
5
6


    public struct Provider
    {
        public Provider(string name, ulong keywords = ulong.MaxValue, 
                        EventLevel eventLevel = EventLevel.Verbose, 
                        string filterData = null)
        { ... }

As we have already mentioned, the name of the provider is “System.Runtime” for the Core CLR counters. The keywords and event level are expected to have these max values. The filter data string starts with “EventCounterIntervalSec=” followed by the refresh interval in seconds. Internally, the CLR in the monitored application is creating a timer with that frequency to push the counters via EventPipe (more on this later).

Here is a helper class to easily create your providers:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


public class CounterHelpers
{
    public static Provider MakeProvider(string name, int refreshIntervalInSec)
    {
        var filterData = BuildFilterData(refreshIntervalInSec);
        return new Provider(name, 0xFFFFFFFF, EventLevel.Verbose, filterData);
    }

    private static string BuildFilterData(int refreshIntervalInSec)
    {
        if (refreshIntervalInSec < 1)
            throw new ArgumentOutOfRangeException(nameof(refreshIntervalInSec), $"must be at least 1 second");

        return $"EventCounterIntervalSec={refreshIntervalInSec}";
    }
}

Note that dotnet-counters allows you to pass a subset of the counters with the System.Runtime[counter1,counter2,counter2] syntax: events for all System.Runtime counters will be received but only these three will be displayed in the console.

Show time for counter events!

Next, the important part of the job takes place in the EventSourc.All event listener. Each new counter value is received in the payload of an event named “EventCounters”.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


private void ProcessEvents(TraceEvent data)
{
    if (data.EventName.Equals("EventCounters"))
    {
        IDictionary<string, object> countersPayload = (IDictionary<string, object>)(data.PayloadValue(0));
        IDictionary<string, object> kvPairs = (IDictionary<string, object>)(countersPayload["Payload"]);

        var name = string.Intern(kvPairs["Name"].ToString());
        var displayName = string.Intern(kvPairs["DisplayName"].ToString());

        var counterType = kvPairs["CounterType"];
        if (counterType.Equals("Sum"))
        {
            OnSumCounter(name, displayName, kvPairs);
        }
        else
        if (counterType.Equals("Mean"))
        {
            OnMeanCounter(name, displayName, kvPairs);
        }
        else
        {
            throw new InvalidOperationException($"Unsupported counter type '{counterType}'");
        }
    }
}

The Name and DisplayName values are self-explanatory. The Sum/Mean type is retrieved from CounterType.

The value for each counter type is retrieved from the payload with “Increment” (Sum type) or “Mean” (*Mean *type) keys.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


 private void OnSumCounter(string name, string displayName, IDictionary<string, object> kvPairs)
{
    double value = double.Parse(kvPairs["Increment"].ToString());

    // send the information to your metrics pipeline
}

private void OnMeanCounter(string name, string displayName, IDictionary<string, object> kvPairs)
{
    double value = double.Parse(kvPairs["Mean"].ToString());

    // send the information to your metrics pipeline
}

The CounterMonitor class has been added on my Github to expose a CounterUpdate C# event when a counter event is received:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


public class CounterMonitor
{
    ...
    public event Action<CounterEventArgs> CounterUpdate;
    private void OnSumCounter(string name, string displayName, IDictionary<string, object> kvPairs)
    {
        double value = double.Parse(kvPairs["Increment"].ToString());

        // send the information to your metrics pipeline
        CounterUpdate(new CounterEventArgs(name, displayName, CounterType.Sum, value));
    }

    private void OnMeanCounter(string name, string displayName, IDictionary<string, object> kvPairs)
    {
        double value = double.Parse(kvPairs["Mean"].ToString());

        // send the information to your metrics pipeline
        CounterUpdate(new CounterEventArgs(name, displayName, CounterType.Mean, value));
    }
}

The event argument contains the expected properties but other could be added if needed such as the timestamp for example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


public class CounterEventArgs : EventArgs
{
    internal CounterEventArgs(string name, string displayName, CounterType type, double value)
    {
        Counter = name;
        DisplayName = displayName;
        Type = type;
        Value = value;
    }

    public string Counter { get; set; }
    public string DisplayName { get; set; }
    public CounterType Type { get; set; }
    public double Value { get; set; }
}

public enum CounterType
{
    Sum = 0,
    Mean = 1,
}

Let’s show some graphs!

With these helpers in hand, it is easy to integrate any counter to your monitoring pipeline. As an example, let’s see how to generate a .csv file used to create visual representations in Excel.

With a refresh rate of 1 second, one line containing the value of the CLR counters should be added to the .csv file every second. Since we get one event per counter, we need to know which is the “last” counter event sent by the CLR for a given 1 second counters push.

As mentioned earlier the RuntimeEventSource class defines the CLR counters. Each one is an instance of a type derived from the DiagnoticCounter class that associates its instances to a CounterGroup also bound to the RuntimeEventSource. The CounterGroup class will setup a repeating timer responsible for creating the payload for its DiagnosticCounter-derived instances and ask the event source to send each to the monitoring application via EventPipe.

So we can rely on the order defined by the counters creation code in RuntimeEventSource: for a given push of counters, the name of the last one will be “assembly-count”. Beware that in a case of new counters (such as for ASP.NET Core), you would need to check what would be the last one of the counters series. Another way to work around would be to rely on the timestamps of each event but this could become flaky over time. It would have been great if a “CounterSeries”event containing the list of counter names would have been sent before any “EventCounters” of a series push (good idea for a pull request :^)

The CsvCounterListener class wraps the few lines of code needed to handle the events and add a line into the .csv file each time a series of counters is received:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115


public class CsvCounterListener
{
    private readonly string _filename;
    private readonly int _pid;
    private CounterMonitor _counterMonitor;
    private List<(string name, double value)> _countersValue;

    public CsvCounterListener(string filename, int pid)
    {
        _filename = filename;
        _pid = pid;
        _countersValue = new List<(string name, double value)>();
    }

    public void Start()
    {
        if (_counterMonitor != null)
            throw new InvalidOperationException($"Start can't be called multiple times");

        _counterMonitor = new CounterMonitor(_pid, GetProviders());
        _counterMonitor.CounterUpdate += OnCounterUpdate;

        Task monitorTask = new Task(() => {
            try
            {
                _counterMonitor.Start();
            }
            catch (Exception x)
            {
                Environment.FailFast("Error while listening to counters", x);
            }
        });
        monitorTask.Start();
    }

    private void OnCounterUpdate(CounterEventArgs args)
    {
        _countersValue.Add((args.DisplayName, args.Value));

        // we know that the last CLR counter is "assembly-count"
        if (args.Counter == "assembly-count")
        {
            SaveLine();
            _countersValue.Clear();
        }
    }

    bool isHeaderSaved = false;
    private void SaveLine()
    {
        if (!isHeaderSaved)
        {
            File.AppendAllText(_filename, GetHeaderLine());
            isHeaderSaved = true;
        }

        File.AppendAllText(_filename, GetCurrentLine());
    }

    private string GetHeaderLine()
    {
        StringBuilder buffer = new StringBuilder();
        foreach (var counter in _countersValue)
        {
            buffer.AppendFormat("{0}\t", counter.name);
        }

        // remove last tab
        buffer.Remove(buffer.Length - 1, 1);

        // add Windows-like new line because will be used in Excel
        buffer.Append("\r\n");

        return buffer.ToString();
    }

    private string GetCurrentLine()
    {
        StringBuilder buffer = new StringBuilder();
        foreach (var counter in _countersValue)
        {
            buffer.AppendFormat("{0}\t", counter.value.ToString());
        }

        // remove last tab
        buffer.Remove(buffer.Length - 1, 1);

        // add Windows-like new line because will be used in Excel
        buffer.Append("\r\n");

        return buffer.ToString();
    }

    public void Stop()
    {
        if (_counterMonitor == null)
            throw new InvalidOperationException($"Stop can't be called before Start");

        _counterMonitor.Stop();
        _counterMonitor = null;

        _countersValue.Clear();
    }

    private IReadOnlyCollection<Provider> GetProviders()
    {
        var providers = new List<Provider>();

        // create default "System.Runtime" provider with a refresh every second
        var provider = CounterHelpers.MakeProvider("System.Runtime", 1);
        providers.Add(provider);

        return providers;
    }
}

What’s next?

You have seen how easy it is to be notified of CLR counters update. The integration to your own monitoring system should not be more complicated. However, you need to pay attention to the meaning of counter types between *Mean *and Sum. For example, the value you get for gen-0-count (Sum) counters is a difference between now and the previous computation. It means that you can’t have the “current” number of gen 0 collection at a given time.

This is not a problem in the Excel example because you can “rebuild” a column that will contain the “current” count based on the previous value + the diff returned by the counter.

Here is the resulting graph:

In other cases, you might need to feed your monitoring system with real count values and benefit from advanced charting such as non derivative computation to show a rate based on a series of values. At the end of the day, it is just a question of initial value from which rebuild a count. And if you think about it, you are often more interested in unexpected variations (i.e. differences returned by counters) when monitoring your application.

In addition to your business metrics, .NET Core Counters are usually enough to monitor the health of your applications. However, in order to investigate situations where counters value are showing weird results, you often need more details. For example spikes in garbage collections count might not be a problem if the pause time is not too long. Listening to specific CLR events as shown in previous posts of this series is a great way to unveil important metrics such as GC pause time, contentions duration or exception names without performance hit.

The code available on Github has been updated to provide the CounterMonitor and CsvCounterListener classes that demonstrates how to get .NET Core counters and generate .csv file usable in Excel.

Spying on .NET Garbage Collector with .NET Core EventPipes

Tue, 28 May 2019 07:38:15 +0000

This post of the series shows how to generate GC logs in .NET Core with the new event pipes architecture and details the events emitted by the CLR during a collection.

Part 1: Replace .NET performance counters by CLR event tracing.

Part 2: Grab ETW Session, Providers and Events.

Part 3: CLR Threading events with TraceEvent.

Part 4: Spying on .NET Garbage Collector with TraceEvent.

Part 5: Building your own Java GC logs in .NET

Introduction

The previous episode of the series introduced the notion of “GC log”, well known in the Java world and how to implement it in .NET thanks to ETW and TraceEvent on Windows. This solution is easy but requires to create an ETW session (and to remember to close it)… and is also not supported on Linux. However, .NET Core 2.2 introduced the EventListener class as the best way to receive CLR events both on Windows and Linux but only from inside the process itself. As of today, TraceEvent is not supporting live session with EventPipe/EventListener, only a file-based constructor is available. This is unfortunate because it means that you can’t rely on the huge work done by TraceEvent to parse the CLR events; especially those related to garbage collections. The rest of the post will explain how to decipher raw events.

In addition, there is a bigger problem: the current .NET Core 2.2 implementation is not working for all CLR events. Long story short, the EventPipe class relies on specific Thread Local Storage slot that is not set by GC background worker threads: the events are not emitted in that case. In addition, there is no per event timestamp information in 2.2. The implementation presented in this post relies on tests done with ETW traces and on the Pull Request that fixes the issue for .NET Core 3.0, available in Preview 5.

Back to the basics: what events are emitted by the GC?

The previous posts of the series were based on C# events raised by the TraceEvent parser with names different from the original CLR events and the corresponding Microsoft Docs. When you implement your EventListener-derived class, each event is received as an EventWrittenEventArgs object in the OnEventWritten override. The EventId and EventName properties allow you to figure out which event is received. If you have worked with TraceEvent before, you might be using the Opcode property but even if a property with the same name exists in EventWrittenEventArgs, the value is completely different and should not be used.

The CLR is versioning the emitted events to be able to add information over time. For example, the EventId of the “GCStart” event is 1 but the EventName could be GCStart, GCStart_V1 or GCStart_V2 even though the Microsoft Docs seems to be stuck on version 1. The following table lists the interesting GC events for .NET Core 2.2/3.0:

Look at the documentation related to each event.

If you go back to this previous article of the series, you notice that all details provided by the TraceGC argument are available except for the objects size before and after the collection. These values are embedded in the workload of the GCPerHeapHistory event by the GC code. Unfortunately, these details are not marshalled by the current EventPipe implementation to your OnEventWritten override (read https://github.com/dotnet/coreclr/issues/24506 for more details and when it will be fixed).

There is no strongly typed EventArgs per event and you need to know the name of the field you are interested in to get its index. From this index, you get its corresponding value from the Payload property of the received EventWrittenArgs. The following helper method is doing the heavy lifting for you:

1
2
3
4
5
6
7
8
9


private T GetFieldValue<T>(EventWrittenEventArgs e, string fieldName)
{
    // this is not very optimum in term of performance but should not be a problem
    var index = e.PayloadNames.IndexOf(fieldName);
    if (index == -1)
        return default(T);

    return (T) e.Payload[index];
}

Now that all interesting events are known, it is time to figure out what is the sequence of events emitted during a garbage collection: a new line with the details should be added to the GC log file when the last event is received.

What is the exact sequence of GC events

So let’s go back to the main phases of a garbage collection with the related CLR events as shown in the following figure (with Konrad Kokosa courtesy from his book)

This is the expected events for the most complicated case: a background collection with possible foreground ephemeral (gen0 and gen1) collections while the GC threads are concurrently sweeping. However, it is not possible to rely on this specific order of events because the order changes, depending on workstation/background mode and generation 2/ephemeral. Each type of collection triggers events in different order as shown below:

Gen0/Gen1 and Gen 2 (non concurrent)

Gen 2 (background)

Here is a more visual view of what could happen (dark blue is gen 2 and light blue are ephemeral gen0/1):

When exactly does a GC start…

The GCTriggered event notifies that a new collection will start except in the case of foreground ephemeral gen0/gen1 collections triggered during a background gen2. In that case, you could rely on the GCStart event and check if a background gen2 is running. This GCStart event provides the condemned generation in its Depth property. So I keep track of both the current background GC (if any) and the foreground GC (if any) in a GCInfo object:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


internal class GCInfo
{
    ...

    // When a background garbage collection (BGC) is started,
    // other foreground garbage collection (FGC) for gen 0 and 1 could happen
    // before the original BGC ends
    //
    public GCDetails CurrentBGC { get; set; }

    // this could contain a FGC after a BGC has started
    // or a non-concurrent gen0/gen1/gen2 collection
    public GCDetails GCInProgress { get; set; }
}

The GCDetails class keeps tracks of all the details gathered during a garbage collection:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


internal class GCDetails
{
    public DateTime TimeStamp { get; set; }
    public double PauseDuration { get; set; }
    public int Number { get; set; }
    public GCReason Reason { get; set; }
    public GCType Type { get; set; }
    public int Generation { get; set; }
    public bool IsCompacting { get; set; }
    public HeapDetails Heaps;
}

The HeapDetails stores the size of each generation after a collection:

1
2
3
4
5
6
7


public struct HeapDetails
{
    public long Gen0Size { get; set; }
    public long Gen1Size { get; set; }
    public long Gen2Size { get; set; }
    public long LOHSize { get; set; }
}

The GCDetails instance is created when the GCStart event is received:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32


private void OnGcStart(EventWrittenEventArgs e)
{
    // This event is received after a collection is started
    var newGC = BuildGCDetails(e);

    // If a BCG is already started, FGC (0/1) are possible and will finish before the BGC
    //
    if (
        (GetFieldValue<uint>(e, "Depth") == 2) && 
        ((GCType)GetFieldValue<uint>(e, "Type") == GCType.BackgroundGC)
        )
    {
        _gcInfo.CurrentBGC = newGC;
    }
    else
    {
        _gcInfo.GCInProgress = newGC;
    }

    // forthcoming expected events for gen 0/1 collections are GCGlobalHeapHistory then GCHeapStats
}

private GCDetails BuildGCDetails(EventWrittenEventArgs e)
{
    return new GCDetails()
    {
        Number = (int)GetFieldValue<uint>(e, "Count"),
        Generation = (int)GetFieldValue<uint>(e, "Depth"),
        Type = (GCType)GetFieldValue<uint>(e, "Type"),
        Reason = (GCReason)GetFieldValue<uint>(e, "Reason")
    };
}

This is where it is important to remember if either a background or foreground GC is starting. In the former case, the CurrentBGC field is set and the GCInProgress field is set otherwise with a new GCDetails instance.

That way, when either of GCGlobalHistory or GCHeapStarts is received, it is easy to know what is the GC in progress; i.e. if a foreground GC is in progress, an event happens in its context (until the last one **GCHeapStats **that will clean the GCInProcess field):

1
2
3
4
5
6
7
8
9


private GCDetails GetCurrentGC(GCInfo info)
{
    if (info.GCInProgress != null)
    {
        return info.GCInProgress;
    }

    return info.CurrentBGC;
}

… suspend, pause application threads and end of ephemeral collections

The suspension and pause time are not that complicated to compute. The garbage collector code is relying on the SuspendEE and RestartEE methods provided by the .NET Execution Engine to suspend and restart the application threads respectively. Each of these methods emits a pair of GCxxxBegin and GCxxxEnd events. After GCSuspendEEBegin is emitted, the Execution Engine waits for the application threads to suspend their execution. When all threads are suspended, GCSuspendEEEnd gets emitted.

The GCRestartEEBegin event is emitted when the applications threads begin to resume their execution. When all application threads are resumed, GCRestartEEEnd gets emitted. The elapsed time between the GCSuspendEEEnd and GCRestartEEBegin events is counted as suspension time. However, for simplicity sake, my current implementation sums both the time spent by the Execution Engine to suspend the threads and the pause time due to the GC work.

The suspension start time is kept in GCInfo:

1
2
3
4


// time when SuspendEEBegin is received for this process
// --> from here, all app threads will be suspended until RestartEEStop is received
// Note that we don't know yet what will be the triggered GC
public DateTime? SuspensionStart { get; set; }

It will be set when the GCSuspendEEBegin event is received:

1
2
3
4
5
6


private void OnGcSuspendEEBegin(EventWrittenEventArgs e)
{
    // we don't know yet what will be the next GC corresponding to this suspension
    // so it is kept until next GCStart 
    _gcInfo.SuspensionStart = e.TimeStamp;
}

This implementation decision does not provide the same level of suspension details (no fine grain suspension time for inner foreground collections) as the one provided by the TraceEvent parsing.

The sibling GCRestartEEEnd event is used to (1) compute the total pause time and (2) detect when gen0/gen1/non concurrent gen2 collections end:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36


private void OnGcRestartEEEnd(EventWrittenEventArgs e)
{
    var currentGC = GetCurrentGC(_gcInfo);
    if (currentGC == null)
    {
        // this should never happen, except if we are unlucky to have missed a GCStart event
        return;
    }

    // compute suspension time
    double suspensionDuration = 0;
    if (_gcInfo.SuspensionStart.HasValue)
    {
        suspensionDuration = (e.TimeStamp - _gcInfo.SuspensionStart.Value).TotalMilliseconds;
        _gcInfo.SuspensionStart = null;
    }
    else
    {
        // bad luck: a xxxBegin event has been missed
    }
    currentGC.PauseDuration += suspensionDuration;

    // could be the end of a gen0/gen1 or of a non concurrent gen2 GC
    if (
        (currentGC.Generation < 2) ||
        (currentGC.Type == GCType.NonConcurrentGC)
        )
    {
        GcEvents?.Invoke(this, BuildGcArgs(currentGC));
        _gcInfo.GCInProgress = null;
        return;
    }

    // in case of background gen2, just need to sum the suspension time
    // --> its end will be detected during GcGlobalHistory event
}

Detect other collections end (and more details)

As shown in the events workflow figure, the GCRestartEEBegin/GCRestartEEEnd duo of events are used to detect the end of non-concurrent gen0/1/2 collections. It is more complicated to detect the end of a gen2 background or inner ephemeral gen0/1 collections: GCGlobalHeapHistory for the former and GCHeapStats for the latter. However, these two events payload does not contain the piece of information to know if we are in a middle of a background gen 2 or not. With this details in mind, the code of the different event handlers is quite straightforward.

The generations size are retrieved from the GCHeapStat event:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


// This event provides the size of each generation after the collection
// Note: last event for non background GC (will be GCGlobalHeapHistory for background gen 2)
private void OnGcHeapStats(EventWrittenEventArgs e)
{
    var currentGC = GetCurrentGC(_gcInfo);
    if (currentGC == null)
        return;

    currentGC.Heaps.Gen0Size = (long)GetFieldValue<ulong>(e, "GenerationSize0");
    currentGC.Heaps.Gen1Size = (long)GetFieldValue<ulong>(e, "GenerationSize1");
    currentGC.Heaps.Gen2Size = (long)GetFieldValue<ulong>(e, "GenerationSize2");
    currentGC.Heaps.LOHSize = (long)GetFieldValue<ulong>(e, "GenerationSize3");

    // this is the last event for non background collections  during a background gen2 collections
    if (
        (_gcInfo.CurrentBGC != null) &&
        (currentGC.Generation < 2)
       )
    {
        GcEvents?.Invoke(this, BuildGcArgs(currentGC));
        _gcInfo.GCInProgress = null;
    }
}

Remember this is the last event received for a gen0/gen1/foreground gen2 collection so I’m using it to clear the GCInProgress field: the next event will be for the current background gen2 if any (CurrentBGC field is not null) or a new collection.

As of today with Preview 5, the before/after generation sizes are not marshalled through event pipes (see the corresponding bug for more details) so the **GCPerHeapHistory **event does not bring any value.

The last GCGlobalHeapHistory event of background gen 2 collection is also used to detect compaction:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


// This event is used to figure out if a collection is compacting or not
// Note: last event for background GC (will be GCHeapStats for ephemeral (0/1) and non concurrent gen 2 collections)
private void OnGcGlobalHeapHistory(EventWrittenEventArgs e)
{
    var currentGC = GetCurrentGC(_gcInfo);

    // check unexpected event (we should have received a GCStart first)
    if (currentGC == null)
        return;
    var globalMask = GetFieldValue<GCGlobalMechanisms>(e, "GlobalMechanisms");
    currentGC.IsCompacting =
        (globalMask & GCGlobalMechanisms.Compaction) == GCGlobalMechanisms.Compaction;

    // this is the last event for gen 2 background collections
    if ((GetFieldValue<int>(e, "CondemnedGeneration") == 2) && (currentGC.Type == GCType.BackgroundGC))
    {
        // check unexpected generation mismatch
        var globalMask = (GCGlobalMechanisms)GetFieldValue<uint>(e, "GlobalMechanisms");
        currentGC.IsCompacting =
            (globalMask & GCGlobalMechanisms.Compaction) == GCGlobalMechanisms.Compaction;

        // this is the last event for gen 2 background collections
        if ((GetFieldValue<uint>(e, "CondemnedGeneration") == 2) && (currentGC.Type == GCType.BackgroundGC))
 {
            GcEvents?.Invoke(this, BuildGcArgs(currentGC));
            ClearCollections(_gcInfo);
        }
    }
}

In case of a background gen 2, this is the last event so there should not be any collection in progress:

1
2
3
4
5


private void ClearCollections(GCInfo info)
{
    info.CurrentBGC = null;
    info.GCInProgress = null;
}

The next received event will start a new garbage collection cycle of events.

This post concludes the series about CLR events and how to use them to better understand how the runtime is behaving under the workloads of your applications. The code available on Github has been updated to provide the EventListenerGcLog class that uses the code demonstrated in this post to generate GC logs with event pipes.

Let’s debug the Core CLR with WinDBG!

Thu, 04 Apr 2019 13:32:51 +0000

This post of the series shows how we debugged the Core CLR to figure out insane contention duration.

Part 1: Replace .NET performance counters by CLR event tracing.

Part 2: Grab ETW Session, Providers and Events.

Part 3: CLR Threading events with TraceEvent.

Part 4: Spying on .NET Garbage Collector with TraceEvent.

Part 5: Building your own Java GC logs in .NET.

Introduction

Long before migrating our .NET applications to Linux, our first step was to build a monitoring pipeline based on LTTng instead of ETW on Windows. To achieve this goal, the open source TraceEvent Nuget package needed to be updated in order to listen to LTTng live session (only a file based implementation was provided by Microsoft; mostly to allow Perfview to be able to open traces taken on Linux machines). This was a huge development task that led sometimes to weird results. Among the metrics we wanted to monitor, the contention duration gave insane value such as thousands of minutes… per minute:

As shown in a previous episode, this duration is computed by comparing the time between the two events ContentStart and ContentionStop. What could be the possible reasons to get such insane values?

A lot of small contentions are happening
A few very long contentions are happening

As a first step, it would be great to be able to debug the Core CLR and figure out what call stacks end up to triggering these contention events. Unfortunately for us, the .NET debugging ecosystem on Linux is far from being as rich as on Windows. So this episode is detailing the steps to compile and debug the Core CLR on Windows with WinDBG.

From the source to debugging the runtime

To better understand the implementation details in the CLR, we needed to find where the two events are emitted. In fact, during the CLR compilation, a lot of helpers are created based on the name of the event. In our case, FireEtwContentionStart_V1 and FireContentionStop are the two helpers in charge. Both are called in the AwareLock::EnterEpilogHelper function.

As a Windows developer, I would like to debug the CLR code and set a breakpoint in the EnterEpilogHelper with Visual Studio to see what are the call stacks that end up to contention. However, I did not find a way to do it with Visual Studio. I turned to WinDBG and things gets “easier”… in a certain way.

Here are the different steps you need to follow before setting a breakpoint on any Core CLR function in WinDBG:

Clone the Core CLR repository from https://github.com/dotnet/coreclr
Build it:
Get the Visual Studio, .NET Core SDK, CMake, Python, Powershell prerequisites from the documentation
Goto the root folder and type .\build -skiptests to build a DEBUG version of the Core CLR
Leave your desk and go to lunch (ok… maybe just take a coffee break)

When you go back, the result of the compilation should be available in the following folder:

…\coreclr\bin\Product\Windows_NT.x64.debug.

the next step is to use your custom Core CLR build in the application:

the application must be self-contained by adding win-x64 (or linux-x64 for Linux) in a PropertyGroup section of the .csproj.
publish the application by running dotnet publish or from within Visual Studio

Click the Configure link and select Debug configuration

after clicking Save and Publish, you should now have the result under the \bin\Debug\netcoreapp2.2\publish folder.
after clicking Save and Publish, you should now have the result under the \bin\Debug\netcoreapp2.2\publish folder.

It is now time to copy the following files from the Core CLR output to your application publication folder:

coreclr.dll (for the native part of the CLR) and System.Private.CoreLib.dll (if the CLR C# code has been modified)
in the PDB subfolder, coreclr.pdb and System.Private.CoreLib.pdb
note that you might also need the sos.dll and mscordaccore.dll files for any investigation in WinDBG.

If you wonder why the CoreFx repo is not rebuilt, the answer is simple: the contention related code is in the CoreCLR. Also, most of the managed “mscorlib” is defined in System.Private.CoreLib.dll that gets built with CoreCLR. The rest of the BCL is covered by CoreFX and not needed in this investigation.

From running to debugging in WinDBG

You should use corerun.exe instead of dotnet.exe to run an application with the debug version of the Core CLR you’ve just built.

Open up a command prompt in the coreclr\bin\Product\Windows_NT.x64.debug folder and type corerun** c:bin\Debug\netcoreapp2.2\publish folder of your application>**

You have to tell corerun where to find the CoreFx assemblies via the CORE_LIBRARY environment variable:

CORE_LIBRARIES=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.0

If you forget about it, don’t be surprised if the application stops with FileNoteFoundException for a missing assembly (usually System.Runtime)…

If, like me, your applications are running with server mode GC, you know that it is set in the application .csproj file to end up into the runtimeconfig.json file. Unfortunately, this is not taken into account by corerun (yet?) and you need to set it (and if you need concurrent version too) explicitly through the following environment variables:

COMPlus_gcServer=1 COMPlus_gcConcurrent=1

From there, (install WinDBG if not already done and) start the debugger: click the File menu and select Launch Executable (advanced) to setup a debugging session:

The Executable text field points to the corerun.exe file generated during the compilation of the Core CLR. The same folder is used as Start Directory and the Arguments text field contains the full path of the application to debug. You could also attach to a running process but sometimes you need to access Core CLR data structures before any C#-compiled managed code of your application starts executing (to see how the garbage collector initializes for example).

As soon as you click the Ok button, the application starts but is almost immediately stopped by WinDBG

Don’t be scared by the last lines of the output: even through you read the word exception, this int 3** **assembly instruction tells you that a breakpoint has been set for you by WinDBG, has been hit when the application reached it and the application is now paused just before calling its entry point.

As you can see from the list of loaded modules, even though CoreRun.exe is there, no managed assembly (especially your application) has been loaded yet; not even the Core CLR itself! This means that you have to tell WinDBG to keep on executing the application until a point you would be interested in. To achieve that task, you will first need a quick tour of WinDBG user interface even though this post is not there to replace the WinDBG online help nor provide a detailed walkthrough.

The debugging section of the Home tab is not too different from what you get in Visual Studio:

The icons are even easier to understand because their action is also displayed. If you want to see the current call stack, select the View tab and click the Stack icon:

Like in Visual Studio, you are able to pin each panel wherever you want into the IDE

The next step would be to set a breakpoint on the line of code you are interested in. But let’s be clear here: I’m talking about a line of code in a function exported by a native dll; not a line of C# code in a managed assembly. Remember that WinDBG is a native debugger and it debugs only native code. If you want to debug managed code with WinDBG, you need to use commands from the sos extension; but this is another story.

So let’s go back to the native world. Even though WinDBG does not have the notion of “solution” like Visual Studio provides, it is still possible to open a C++ file and set a breakpoint in it. Click File |Open Source File menu and go to your Core CLR github repo to select syncblk.cpp under the \src\vm folder. Look for AwareLock::EnterEpilogHelper with CTRL+F (yes: search is working in WinDBG) and go down to the call to the FireEtwContentionStart_V1 helper. Setting a breakpoint on this line is as simple as pressing **F9 **like in Visual Studio. Press the View tab and click the Breakpoint button to see the result:

Since the dll in which the breakpoint is set is not loaded yet, you can’t see the details of the breakpoint.

There is a way to tell WinDBG to continue the execution of the application until a dll get’s loaded. For coreclr.dll, type the following command:

sxeld:coreclr

and type F5 (or type g as a command or click the green triangle in the Home toolbar) to resume the execution of the application. The Breakpoints panel shows more details now:

Press F5 to resume the execution and the breakpoint should be triggered when the first contention happens.

From symbols to call stacks in WinDBG

Before digging into call stacks, I would like to show you one of the differences between native dlls and managed assemblies. As a .NET developer, you are used to Intellisense and strongly typed environment provided by the metadata stored in an assembly itself. For native dll, the story is different. Exported functions are visible with tools such as Dependency Walker or dumpbin /exports from the SDK. If the dll exports symbols built by the C++ compiler, their name gets mangled to describe their signature. To get human readable symbols, you need the associated .pdb file. It will also be required to map a function address to its name in call stacks.

WinDBG allows you to browse these symbols with the dt command. For example, if you want to know all members defined by the AwareLock class, use the following command:

dt CoreClr!AwareLock::*

Like what was shown in the previous Breakpoints screenshot, the prefix of a name is the dll in which the symbol is defined. Next, use ! as separator before the class name. Since Visual Studio is really slow to navigate the source code of the Core CLR or search in the thousands of include and C/C++ files, this is a very convenient way to navigate and learn its different parts. Don’t forget that the compilation could also inline functions (that won’t be visible in the symbols) and expand macros.

If you want to set a breakpoint on a function, use the bp command with the same syntax as dt. For example, the following command:

bp coreClr!AwareLock::EnterEpilogHelper

sets a breakpoint at the beginning of the function in which I already set a breakpoint.

This is the very basics of breakpoints in WinDBG. You are also able to define which actions to start when a breakpoint is hit. This is extremely powerful! For example, in the case of thread contention, you typically don’t want to stop the execution of the application because it will pause all threads and disturb the normal flow of execution that could lead to thread contention. Instead, you could ask WinDBG to dump the call stack leading to the function we are interested in and lets the execution resume with the following syntax:

bp coreClr!AwareLock::EnterEpilogHelper "!clrstack; g"

The commands to execute after the breakpoint is hit are defined between quotes. In this example, I’m using the clrstack command exported by the sos.dll extension (that must be previously loaded via .loadby sos coreclr) and once it is done, g resumes the execution.

What’s next?

Due to automatic suspension of all threads when the clrstack command gets executed (before g resumes), the interactions between threads are not the same as normal execution outside of a debugger. I have even used some code available in DEBUG to dump the callstacks outside of a debugger if the contention last more than a threshold. However, it was not possible to reproduce the problem on Windows.

In parallel on Linux, another colleague investigated another lead: some events may also be skipped by our LTTng implementation. Due to complicated event management, if a ContentionStop and ContentionStart are missed, a possible previous ContentionStart could be used by the next ContentionStop and the duration would be unrelated to the real contention that happened.

So there could be a simpler way to narrow down the issue: instead of relying on two events, why not simply compute the duration of the contention in the AwareLock::EnterEpilogHelper function and emit only one new event with the duration as payload? Well… this will be the topic of the next episode of this series.

References

Series of videos from the Defrag Tools show where Maoni Stephens explains how to debug the Garbage Collector for a better understanding of its arcana

Building your own Java-like GC logs in .NET

Tue, 12 Feb 2019 10:11:39 +0000

This post of the series focuses on logging each GC details in a file and how to leverage it during investigations.

Part 1: Replace .NET performance counters by CLR event tracing.

Part 2: Grab ETW Session, Providers and Events.

Part 3: Monitor Finalizers, contention and threads in your application.

Part 4: Spying on .NET Garbage Collector with TraceEvent.

Introduction

I’m working in a team where we investigate issues in production: both for Java and .NET applications. This is a good opportunity to learn what are the features provided by Java that are missing in .NET. One of the features heavily discussed with my colleague Jean-Philippe is called the GC Log. It is possible to start an application with parameters that tell the GC to save tons of details about each garbage collection in a file : the GC Log. Based on this file, it is possible to extract the reason of a collection, the times of the different phases including the suspension time. This is a great source of information during investigations… when you know how GC is working or by leveraging automatic report generation.

In addition, you can also build your own UI to more easily understand what is going on and get a more visual representation of the situation.

In the short video above you can see the heap evolution during several days. Then, as this is an interactive HTML page you can zoom in an interesting period to have a more detailed view of the evolution between GCs.

Also for the pause time graph, you can follow the behavior of the GC with different kinds of pauses and associated phases. In this example, we have minor GCs happening and then an “initial mark” is triggered, followed by “final remark” and “cleanup” pauses. After an extra minor GC, we have a series of mixed GCs that is the result of what was planned by the GC after the marking phase.

In the .NET world, there is no such thing as a GC Log. However, as shown in the previous post, it is possible to use Perfview to analyze traces corresponding to collected CLR events. The GCStats view shows high level details in the “All GC Events” section. In addition to this HTML rendering, you can get access to the data itself in different formats

The more complete one is the Raw Data XML file that you could parse to extract the details you need. This is very close to a .NET GC Log but it is complicated to build an automated process to get it from a production machine.

It would be great if you could tell a .NET application to generate such a GC Log like in Java instead of relying on manual steps with Perfview (and more scripts on Linux). This post will show you how to achieve this goal!

Defining the goal: basic GcLog implementation

In Java, you have to set on or off the GC log before the application starts and you can’t change it while it runs. Since I’m working with server applications, I would prefer to enable/disable the generation of a GC log file without having to stop and restart the application.

So I’ve defined the simple IGcLog interface:

1
2
3
4
5
6


public interface IGcLog
{
    void Start(string filename);

    void Stop();
}

In a dedicated administration handler (i.e. http endpoint of the application), the code could just use a class that implements this interface and call Start when the log is enabled and Stop when it is no more needed.

To make the implementation easier, I’ve written the following GcLogBase abstract class:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


public abstract class GcLogBase : IGcLog
{
    protected string Filename;
    private StreamWriter _fileWriter;

    public void Start(string filename)
    {
        if (string.IsNullOrEmpty(filename))
            throw new ArgumentNullException(nameof(filename));

        if (_fileWriter != null)
            throw new InvalidOperationException("Start can't be called twice: Stop must be called first.");

        _fileWriter = new StreamWriter(filename);
        Filename = filename;

        OnStart();
    }

    public void Stop()
    {
        if (string.IsNullOrEmpty(Filename))
            return;

        OnStop();
        Filename = null;
        _fileWriter.Flush();
        _fileWriter.Dispose();
        _fileWriter = null;
    }

    protected bool WriteLine(string line)
    {
        if (_fileWriter == null)
            return false;   // just in case the method is called AFTER Stop

        _fileWriter.WriteLine(line);

        return true;
    }

    protected abstract void OnStart();

    protected abstract void OnStop();
}

Its main goal is to hide the file management by providing the WriteLine method that child classes would call to save the details of a garbage collection into a single line of text. The write operations are flushed when Stopis called. This combination allows asynchronous writes with low performance impact: don’t be scared if you don’t see the file size change because the StreamWriter class is caching write operations.

The next step is to implement OnStart and OnStop in a derived class to enable/disable GC details retrieval.

How to get the GC details: the easy way?

As already discussed in the previous posts of the series, the CLR is emitting traces (via ETW on Windows and LTTng on Linux) that can be collected in C#. You have already seen how TraceEvent could help collecting and parsing GC traces from any application like what Perfview is doing. With TraceEvent, the TraceGC instance received when a garbage collection ends contains tons of information: it’s mapped to the GarbageCollectionArgs structure that you get while listening to the GarbageCollection event of my ClrEventsManager helper class. The only information to provide is the ID of the .NET process I’m interested in: that way, is it easy to filter the events for this process only.

1
2
3
4
5
6
7
8
9


EtwGcLog gcLog = EtwGcLog.GetProcessGcLog(pid);
var filename = GetUniqueFilename(pid);
gcLog.Start(filename);

// in a simple Console application, wait for the user to press ENTER.
// in a more realistic case, keep track of the EtwLog instance and 
// call Stop to end the processing of events when needed.

gcLog.Stop();

The GetUniqueFilename method builds a filename based on the process ID and the time of the day:

1
2
3
4
5
6
7
8


private static string GetUniqueFilename(int pid)
{
    var now = DateTime.Now;
    string filename = Path.Combine(Environment.CurrentDirectory, 
        $"{pid.ToString()}_{now.Year}{now.Month}{now.Day}_{now.Hour}{now.Minute}{now.Second}.csv"
        );
    return filename;
}

The GetProcessGcLog method is a factory-like helper to build an instance bound to the given process ID.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


public static EtwGcLog GetProcessGcLog(int pid)
{
    EtwGcLog gcLog = null;
    try
    {
        var process = Process.GetProcessById(pid);
        process.Dispose();

        gcLog = new EtwGcLog(pid);
    }
    catch (System.ArgumentException)
    {
        // there is no running process with the given pid
    }

    return gcLog;
}

The implementation of OnStart and OnStop overrides is straightforward based on the previous post:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


protected override void OnStart()
{
    string sessionName = $"GcLogEtwSession_{_pid.ToString()}_{Guid.NewGuid().ToString()}";
    Console.WriteLine($"Starting {sessionName}...\r\n");
    _userSession = new TraceEventSession(sessionName, TraceEventSessionOptions.Create);

    Task.Run(() =>
    {
        // only want to receive GC event
        ClrEventsManager manager = new ClrEventsManager(_userSession, _pid, EventFilter.GC);
        manager.GarbageCollection += OnGarbageCollection;

        // this is a blocking call until the session is disposed
        manager.ProcessEvents();
        Console.WriteLine("End of CLR event processing");
    });

    // add a header to the .csv file
    WriteLine(Header);
}

protected override void OnStop()
{
    // when the session is disposed, the call to ProcessEvents() returns
    _userSession.Dispose();
}

The created TraceEventSession is passed to the ClrEventManager with the process ID with a filter to receive only GarbageCollection event notifications. The OnGarbageCollection handler is super simple:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


private void OnGarbageCollection(object sender, GarbageCollectionArgs e)
{
    _line.Clear();
    _line.AppendFormat("{0},", e.StartRelativeMSec.ToString());
    _line.AppendFormat("{0},", e.Number.ToString());
    _line.AppendFormat("{0},", e.Generation.ToString());
    _line.AppendFormat("{0},", e.Type);
    _line.AppendFormat("{0},", e.Reason);
    _line.AppendFormat("{0},", e.IsCompacting.ToString());
    _line.AppendFormat("{0},", e.SuspensionDuration.ToString());
    _line.AppendFormat("{0},", e.PauseDuration.ToString());
    _line.AppendFormat("{0},", e.BGCFinalPauseDuration.ToString());
    _line.AppendFormat("{0},", e.Gen0Size.ToString());
    _line.AppendFormat("{0},", e.Gen1Size.ToString());
    _line.AppendFormat("{0},", e.Gen2Size.ToString());
    _line.AppendFormat("{0},", e.LOHSize.ToString());
    _line.AppendFormat("{0},", e.ObjSizeBefore[0].ToString());
    _line.AppendFormat("{0},", e.ObjSizeBefore[1].ToString());
    _line.AppendFormat("{0},", e.ObjSizeBefore[2].ToString());
    _line.AppendFormat("{0},", e.ObjSizeBefore[3].ToString());
    _line.AppendFormat("{0},", e.ObjSizeAfter[0].ToString());
    _line.AppendFormat("{0},", e.ObjSizeAfter[1].ToString());
    _line.AppendFormat("{0},", e.ObjSizeAfter[2].ToString());
    _line.AppendFormat("{0}", e.ObjSizeAfter[3].ToString());

    WriteLine(_line.ToString());
}

Each garbage collection appears as a textual line with the following columns separated by a comma:

The last twelve pieces of information require some explanation:

· **xxxBefore **: size of a generation before the collection; without free list

· **xxxAfter **: size of a generation after the collection; without free list

· **xxxSize **: size of a generation (including LOH) after the collection; including free list (i.e. fragmentation)

The computation of these sizes relies on inner fields of the TraceGC argument receives from TraceEvent. The xxxSize are grouped in the GenerationSize0/1/2/3 fields of the HeapStat property. It is a little bit more complicated for the Before/After sizes. The Garbage Collector keeps track of these numbers in the PerHeapHistories field: an array of GCPerHeapHistory elements; one per heap (i.e. one per core for server GC). The next level is provided by the GenData field storing an array of GCPerHeapHistoryGenData elements; one per generation with LOH as the last index 3. So, to compute the size of each generation, it is needed to iterate on each heap:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


private long[] GetGenerationSizes(TraceGC gc, bool before)
{
    var sizes = new long[4];
    if (gc.PerHeapHistories == null)
    {
        return sizes;
    }

    for (int heap = 0; heap < gc.PerHeapHistories.Count; heap++)
    {
        // LOH = 3
        for (int gen = 0; gen <= 3; gen++)
        {
            sizes[gen] += before ? 
                gc.PerHeapHistories[heap].GenData[gen].ObjSpaceBefore:
                gc.PerHeapHistories[heap].GenData[gen].ObjSizeAfter;
        }
    }

    return sizes;
}

The code of the GetGenerationSizes helper method does that sum the value of either ObjSpaceBefore or ObjSizeAfter.

As you have probably noticed from the implementation, it is possible that the PerHeapHistories field is not filled up and all Before/After values are zero. This happens for a background gen2 collection. Also note that for gen0 and gen1 collection the value for gen2 and LOH is also 0 (make sense that gen2 and LOH do not change during such a collection).

A little bit of UI

Now that a .csv file containing all garbage collections details is available, it is time to provide some UI on top of it such as the following for GC pauses:

Let’s start what you can get for Excel champions:

Generation ratio

Sizes of generations including Large Object Heap

Top 10 pauses (including suspension time comparison)

But you can get better interaction thanks to Jean-Philippe. My colleague adapted his script for JVM to my .NET GC log .csv format: it generates some nice zoomable HTML UI.

This short video above shows the heap evolution during ~20 minutes. Then, as this is an interactive HTML page you can focus on gen2 and LOH impact on memory consumption.

For the pause time graph on the same period, it is very easy to detect long pauses (even for gen0 collection) and zoom into smaller period to figure out the impact of different collections.

The code available on Github has been updated to make the EtwGcLog class available to you.

Spying on .NET Garbage Collector with TraceEvent

Sat, 15 Dec 2018 11:08:01 +0000

This post of the series focuses on CLR events related to garbage collection in .NET.

Part 1: Replace .NET performance counters by CLR event tracing.

Part 2: Grab ETW Session, Providers and Events.

Part 3: CLR Threading events with TraceEvent.

Introduction

The allocator and garbage collector components of the CLR may have a real impact on the performances of your application. The Book of the Runtime describes the allocator/collector design goals in the must read Garbage Collection Design page written by Maoni Stephens, lead developer of the GC. In addition, Microsoft provides large garbage collection documentation. And if you want more details about .NET garbage collector, take a look at Pro .NET Memory Management by Konrad Kokosa. In this post, I will focus on the events emitted by the CLR and how you could use them to better understand how your application is behaving, related to its memory consumption.

The impact on how your application behaves is mostly related to a couple of topics:

*How many times and how long your threads get suspended during a collection *Desktop applications and games provide fluent User Interfaces where glitches are less and less acceptable. In the opposite side of the spectrum, low latency server applications have short SLAs to answer each request. In both cases, applications cannot afford freezing for too long while the high priority GC threads are cleaning up the .NET heaps for background GCs or blocking non concurrent GCs.
*How much memory is dedicated to your process *With the rise of containers and their quotas, your application needs to trim down its memory consumption. For example, with server GC enabled, the amount of memory used by your application could grows big (depending on the number of cores) before a gen 0 collection kicks in (read this discussion about real world cases including StackOverflow web site and what are the possible solutions) The memory pressure on the system is also taken into account by the GC and could lead to more collections being triggered (read Maoni Stephen blog post about how Windows jobs are taken into account by the GC and how to leverage them if needed). It becomes more and more important to detect leaks and memory consumption spikes.

In the previous post, you saw how to get the type name of instances being finalized. The CLR provides many more events related to memory management. They definitively help understand the interactions between this crucial part of .NET and your own code. In this article, you will see how to replace the not always consistent performance counters such as generation sizes or collection counts. More importantly, you will get very useful metrics information like the type of GC (foreground or background) and your application threads suspension time.

Sequences of events during Garbage Collection phases

Ephemeral collections (of generation 0 and 1) are called “stop-the-world”: your application threads will be frozen during the whole collection. For generation 2 background collections, it is a little bit more complicated. As shown in the following figure (with Konrad Kokosa courtesy from his book)

The applications threads will be frozen during different phases:

Initial internal step at the beginning of the collection,
At the end of the marking phase to reconcile the changes (allocations, references updates) done while background collection threads are running (also if compaction is needed). Look for documentation about card table usage to get more details,
If a compaction occurs.

Please read Understanding different GC modes with Concurrency Visualizer to go deeper and blog posts from Matt Warren and Maoni Stephens about GC pauses.

What are the available garbage collections metrics?

The Perfview tool could help you analyze how many garbage collections occurred and for which reason. Select Run in the Collect menu and click the Run Command button.

You could also trigger a collection after the application is started with Collect | Collect. When you want to stop collecting information, click the Stop Collection. When the .etl file gets generated, go to the GCStats node

Look for your application to get statistics related to garbage collections. The first GC Rollup By Generation table gives you high level metrics such as the number of collections per generation and the mean pause time:

The next two sections list the collections with a pause time longer than 200ms before the section that lists all generation 2 collections:

The Suspend Msec columns gives you the time it took to suspend your application threads while Pause MSec counts the time during which your threads were actually suspended.

In addition to this, memory details such as the size of all generations after each collection are available:

However, my goal is to get these details to feed monitoring dashboards as the application runs. I can’t use Perfview but I can still rely on the same CLR events.

A solution for runtime please!

Since version 2 of TraceEvent, there is an easy way to get already computed metrics about GC as described by Maoni Stephens. It relies on the same code as Perfview for its GCStats window.

You only need to subscribe to two events; one when a GC starts and one when it ends:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


var source = userSession.Source;
source.NeedLoadedDotNetRuntimes();
source.AddCallbackOnProcessStart((TraceProcess proc) =>
{
    proc.AddCallbackOnDotNetRuntimeLoad((TraceLoadedDotNetRuntime runtime) =>
    {
        runtime.GCStart += (TraceProcess p, TraceGC gc) =>
        {
            // a GC is starting
        };
        runtime.GCEnd += (TraceProcess p, TraceGC gc) =>
        {
            // a GC ends
        };
    });
});

The TraceGC class provides too many details beyond the scope of this post but here are the main fields that should be used in GCEnd event handler to monitor your applications:

Note that the IsNotCompacting method currently returns invalid value.

Final words

I would like to mention one last event related to memory management. The GCAllocationTick CLR event (mapped by the ClrTraceEventParser.GCAllocationTick event) is emitted after ~100 KB has been allocated by your application. As you can infer from the Microsoft documentation, the field of the GCAllocationTickTraceData argument received by your handler provides the following properties:

As you can see, listening to this GCAllocationTick event gives you a sampling of the allocations made in your application. This is not as precise as what a .NET profiler (relying on expensive ObjectAllocated and ObjectAllocatedByClass ICorProfilerCallback hooks) would provide but it is much less intrusive. However, I would not recommend to systematically listen to this event in production, especially if your application is allocating GBs of memory per minute. Unlike what the documentation states, you need to set the verbosity to TraceEventLevel.Verbose (and not Informational) when you enable the CLR provider and this could impact your application performances due to the high number of emitted CLR events.

This event could be very helpful in case of unusual LOH allocations because you would get the type of the objects in the LOH almost each time (the 85.000 bytes threshold is close to the 100 KB trigger limit) or simply to have an hint on the most allocated types over time. Note that you won’t get the callstack leading to the allocations triggering the event. Instead, for memory leak or memory usage analysis, I would definitively recommend you to use Perfview. Vance Morrison has published a series of videos that detail .NET memory investigations, collecting the data and analyzing the data with Perfview. You will also find a lot of detailed memory-related investigations guidelines in Konrad Kokosa’s book.

You now have a complete view of the CLR events interesting to understand the different phases of a garbage collection and a few interactions (suspension) with the Execution Engine. Everything is in hands to replace the performance counters by CLR events: the metrics are more accurate and you get access to more information such as suspension time or contention time. The code presented during all episodes is available on Github with an easy to reuse ClrEventManager class that you could plug into your own applications or monitoring service!

In-process CLR event listeners with .NET Core 2.2

Thu, 06 Dec 2018 13:52:09 +0000

As the .NET Core 2.2 blog post introduced, it is now possible for a .NET Core application to listen to the events generated by the CLR that power it up. If you remember the Grab ETW Session, Providers and Events post, the CLR is emitting a lot of valuable events through ETW on Windows and LTTng on Linux. Thanks to TraceEvent nuget package, it is not that difficult to fetch these events at runtime on Windows, either in-process or out of process. However, it is much more complicated to achieve the same goal on Linux… With .NET Core 2.2, it is now super easy to listen to the events emitted by the CLR while your application is running: you simply need to implement a class that derives from System.Diagnostics.Tracing.EventListener and create an instance of it. Nothing more.

This class exists since .NET Framework 4.5 and .NET Core 1.0 but it could only be used to listen events pushed by managed code. Since .NET Core 2.2, it can also be used to listen to native events pushed by the CLR. The usage is simple, even a little bit magical.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48


sealed class GcFinalizersEventListener : EventListener
{
    // from https://docs.microsoft.com/en-us/dotnet/framework/performance/garbage-collection-etw-events
    private const int GC_KEYWORD =                 0x0000001;
    private const int TYPE_KEYWORD =               0x0080000;
    private const int GCHEAPANDTYPENAMES_KEYWORD = 0x1000000;

    protected override void OnEventSourceCreated(EventSource eventSource)
    {
        Console.WriteLine($"{eventSource.Guid} | {eventSource.Name}");

        // look for .NET Garbage Collection events
        if (eventSource.Name.Equals("Microsoft-Windows-DotNETRuntime"))
        {
            EnableEvents(
                eventSource, 
                EventLevel.Verbose, 
                (EventKeywords) (GC_KEYWORD | GCHEAPANDTYPENAMES_KEYWORD | TYPE_KEYWORD)
                );
        }
    }

    // from https://blogs.msdn.microsoft.com/dotnet/2018/12/04/announcing-net-core-2-2/
    // Called whenever an event is written.
    protected override void OnEventWritten(EventWrittenEventArgs eventData)
    {
        ...
    }
}

class Program
{
    static void Main(string[] args)
    {
        GcFinalizersEventListener listener = new GcFinalizersEventListener();

        Console.WriteLine("\nPress ENTER to trigger a few finalizers...");
        Console.ReadLine();
        for (int i = 0; i < 4; i++)
        {
            Thread t = new Thread(()=> {});
        }
        GC.Collect(2, GCCollectionMode.Forced, true, true);

        Console.WriteLine("\nPress ENTER to exit...");
        Console.ReadLine();
    }
}

First, you implement a class that derives from EventListener and override the following two methods:

1
2


   void OnEventSourceCreated(EventSource eventSource)
   void OnEventWritten(EventWrittenEventArgs eventData)

As soon as you new up an instance of your class, the OnEventSourceCreatedoverride is called for each event source defined in the application. An event source, as its name implies, produces events. You can define your own in managed code if you wish. For the sake of this post, I will focus on listening to the Microsoft-Windows-DotNETRuntime event source. The EventSourceinstance passed to the OnEventSourceCreatedmethod provides two interesting properties to let us identify the available sources. The following code :

1
2
3
4


protected override void OnEventSourceCreated(EventSource eventSource)
{
    Console.WriteLine($"{eventSource.Guid} | {eventSource.Name}");
}

generates the following output :

5e5bb766-bbfc-5662-0548-1d44fad9bb56 | Microsoft-Windows-DotNETRuntime
 2e5dba47-a3d2-4d16-8ee0-6671ffdcd7b5 | System.Threading.Tasks.TplEventSource
 8e9f5090-2d75-4d03-8a81-e5afbf85daf1 | System.Diagnostics.Eventing.FrameworkEventSource

You can imagine that the first one is the source we are interested in listening to its events!

By default, there is no connection between the sources and your listeners: you need to enable the source by calling the EnableEventsmethod in your OnEventSourceCreatedoverride. This EventListenermethod takes the following arguments:

EventSource eventSource: the event source you want to listen to
EventLevel level: minimum verbosity level for the received events
EventKeywords matchAnyKeyword: a keyword to filter on specific events

The Microsoft Docs provides the level and keywords for each events documented in the CLR. For the complete list, you take a look at ClrETWAll.man in CoreClr source code or in ClrTraceEventParser class of TraceEvent. In the sample code at the beginning of this post, I selected a group of keywords GC_KEYWORD | GCHEAPANDTYPENAMES_KEYWORD | TYPE_KEYWORD to receive only events related to the garbage collector and type information (read this previous post for more details).

Once the source has been paired to the listener, each time an event is emitted by the source with the right level and for the given keywords, the OnEventWrittenoverride will get called. The EventWrittenEventArgsinstance received as a parameter describes each event.

The Payload contains the value of the different properties stored in a ReadOnlyCollection and the corresponding property names are provided via the ReadOnlyCollection PayLoadNames. The following code shows how to extract all properties values:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


// from https://blogs.msdn.microsoft.com/dotnet/2018/12/04/announcing-net-core-2-2/
// Called whenever an event is written.
protected override void OnEventWritten(EventWrittenEventArgs eventData)
{
    // Write the contents of the event to the console.
    Console.WriteLine($"ThreadID = {eventData.OSThreadId} ID = {eventData.EventId} Name = {eventData.EventName}");
    for (int i = 0; i < eventData.Payload.Count; i++)
    {
        string payloadString = eventData.Payload[i] != null ? eventData.Payload[i].ToString() : string.Empty;
        Console.WriteLine($"    Name = \"{eventData.PayloadNames[i]}\" Value = \"{payloadString}\"");
    }
    Console.WriteLine("\n");
}

Here is the kind of output you get for common garbage collector and finalizer events:

ThreadID = 17456 ID = 200 Name = IncreaseMemoryPressure
Name = "BytesAllocated" Value = "1672"
Name = "ClrInstanceID" Value = "8"

ThreadID = 17456 ID = 9 Name = GCSuspendEEBegin_V1
Name = "Reason" Value = "1"
Name = "Count" Value = "0"
Name = "ClrInstanceID" Value = "8"

ThreadID = 17456 ID = 8 Name = GCSuspendEEEnd_V1
Name = "ClrInstanceID" Value = "8"

ThreadID = 17456 ID = 35 Name = GCTriggered
Name = "Reason" Value = "10"
Name = "ClrInstanceID" Value = "8"

ThreadID = 17456 ID = 1 Name = GCStart_V2
Name = "Count" Value = "1"
Name = "Depth" Value = "2"
Name = "Reason" Value = "10"
Name = "Type" Value = "0"
Name = "ClrInstanceID" Value = "8"
Name = "ClientSequenceNumber" Value = "0"

ThreadID = 18860 ID = 29 Name = FinalizeObject
Name = "TypeID" Value = "1210056592"
Name = "ObjectID" Value = "1371069040"
Name = "ClrInstanceID" Value = "8"

ThreadID = 18860 ID = 15 Name = BulkType
Name = "Count" Value = "1"
Name = "ClrInstanceID" Value = "8"

The fact that there is no strongly typed event argument per event is not as good as what TraceEvent provides. In addition, after a few tests, it seems that the .NET Core 2.2 implementation is not complete:

GC events are not all received when in Server Mode
Properties are missing for BulkType event necessary to figure out finalizer type names

However, with EventListener, Microsoft is giving us a very simple way to get valuable information, in-process, from the CLR while the application is running. A forthcoming blog post will show how to leverage this infrastructure to provide insights on how the garbage collection impacts an application.

Before leaving you building your own event listeners, you should know a couple of last details. Under the hood, the framework is creating a dedicated thread for you that will execute the two OnXXXmethods of your EventListener-derived class. It means that your code should not block or spend to much time processing the events if you want to keep on receiving events at a regular pace.

This thread will last as long as one of your listeners still exists. When I say “exist”, I mean until you decide to dispose them. This is the way for you to tell the sources that you are no more interested in receiving events. When all your listeners are disposed, then the processing thread will exit.