Debugging on Welcome to Christophe Nasarre's Blog

How to support .NET Framework PDB format and source line with ISymUnmanagedReader

Wed, 11 Feb 2026 09:16:11 +0000

In my previous posts, I explained how to use DIA and DbgHelp to map a method to its line in source code. I forgot to mention that it was correct for .NET Core but not for the “old” .NET Framework Windows PDB format. Instead of encoding the method token in the name, the symbol file contains the name of the methods. So, how to do the mapping for .NET Framework assemblies? You will find the answer (plus some tricks) in this article.

When I started to work on the support of the old Windows PDB format, I looked at what existed to parse the raw format and… I decided to try DbgHelp instead. With this first implementation, I realized that some token where missing and source code information was not retrieved for most of the methods.

So, I looked for another API to use and I found ISymUnmanagedReader. The usage philosophy is totally different from DIA or DbgHelp.

A little bit of magic

This interface is implemented in diasymreader.dll that comes with every .NET Framework installation. But you need to do COM magic to get it. After having called CoInitialize to setup COM, you ask for an instance of ISymUnmanagedBinder from CLSID_CorSymBinder_SxS:

1
2
3
4
5
6
7
8


CComPtr<ISymUnmanagedBinder> pBinder;
hr = CoCreateInstance(
    CLSID_CorSymBinder_SxS,
    NULL,
    CLSCTX_INPROC_SERVER,
    IID_ISymUnmanagedBinder,
    (void**)&pBinder
);

From the binder, you can get the ISymUnmanagedReader interface corresponding to the assembly you are interested in with GetReaderForFile. However, there are two tiny details to consider.

First, one parameter expects the path to the assembly, not to the .pdb file. That symbol file has to be stored in the same folder but note that the documentation states that you could have more flexible search with ISymUnmanagedBinder2::GetReaderForFile2 but I did not test it.

The second detail is the first parameter: an instance of IMetaDataImport for the same assembly. The steps to get it are… complicated.

Hosting the CLR

The idea is to host the .NET Framework and get the corresponding ICLRMetaHost interface:

1
2


CComPtr<ICLRMetaHost> pMetaHost;
HRESULT hr = CLRCreateInstance(CLSID_CLRMetaHost, IID_ICLRMetaHost, (void**)&pMetaHost);

Calling the CLRCreateInstance API allows you to get an instance of ICLRMetaHost from which you could enumerate installed version of .NET Framework. In my case, I know which version I want:

1
2
3


// Get the installed .NET Framework runtime (v4.0+)
CComPtr<ICLRRuntimeInfo> pRuntimeInfo;
hr = pMetaHost->GetRuntime(L"v4.0.30319", IID_ICLRRuntimeInfo, (void**)&pRuntimeInfo);

The ICLRRuntimeInfo interface allows you to get access to runtime services via GetInterface:

1
2


CComPtr<IMetaDataDispenser> pDispenser;
hr = pRuntimeInfo->GetInterface(CLSID_CorMetaDataDispenser, IID_IMetaDataDispenser, (void**)&pDispenser);

The service I’m interested in is the IMetadataDispenser interface that allows you to “open a scope” on the assembly you are interested in:

1
2
3
4
5
6


hr = pDispenser->OpenScope(
    wModulePath.c_str(),
    ofRead,
    IID_IMetaDataImport,
    (IUnknown**)&_pMetaDataImport
);

Note that the first parameter is the path to the assembly not to the .pdb file. The scope is abstracted by an IMetadataImport interface I have already described and that is needed to call GetReaderForFile: and get the ISymUnmanagedReader:

hr = pBinder->GetReaderForFile(_pMetaDataImport, wModulePath.c_str(), nullptr, &_pReader);

The road to get symbol details for a method

The ISymUnmanagedReader interface implements GetMethod to get details about a given method token via an ISymUnmanagedMethod interface. So, the next question is how to get these tokens. If you remember the previous article, these tokens are from the 06 MethodDef table in the assembly metadata; starting from 06000001 to the last one.

This means that you could write a simple loop starting from 1 up to a hardcoded maximum value, call TokenFromRid(index, mdtMethodDef) to get the corresponding token. However, since you are a professional developer, you would search for the exact number of tokens from IMetadataTables retrieved from IMetadataImport:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


ULONG cRows = 0;

// Get IMetaDataTables interface to query the MethodDef table
CComPtr<IMetaDataTables> pTables;
hr = _pMetaDataImport->QueryInterface(IID_IMetaDataTables, (void**)&pTables);
if (FAILED(hr) || pTables == nullptr)
{
    cRows = LAST_METHODDEF_TOKEN;
}
else
{
    // Get the number of rows in the MethodDef table (table index 0x06 = Method)
    hr = pTables->GetTableInfo(
        0x06,           // MethodDef table
        NULL,           // cbRow (not needed)
        &cRows,         // pcRows (number of methods)
        NULL,           // pcCols (not needed)
        NULL,           // piKey (not needed)
        NULL            // ppName (not needed)
    );

    if (FAILED(hr))
    {
        cRows = LAST_METHODDEF_TOKEN;
    }
}

Now that you have the number of rows (i.e. number of methods defined in the metadata), it is easy and safe to get method information from symbols:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


for (uint32_t i = 1; i <= cRows; i++)
{
    mdMethodDef token = TokenFromRid(i, mdtMethodDef);

    CComPtr<ISymUnmanagedMethod> pMethod;
    hr = _pReader->GetMethod(token, &pMethod);
    if (SUCCEEDED(hr) && pMethod != nullptr)
    {
        MethodInfo info;
        if (GetMethodInfoFromSymbol(pMethod, info))
        {
            _methods.push_back(info);
        }
    }
}

Note that GetMethod might fail (returning E_FAIL) for P/Invoked functions, abstract methods, or methods decorated with DebuggerHidden attribute.

Give me line and source code!

For the other methods with symbol information, you can get its token via the GetToken method. The ISymUnmanagedMethod interface allows low level access to line/column mapping that is beyond the scope of this article. At a high level, positions in source file are named sequence points. Call GetSequencePointCount to get… the number of sequence points for a given method.

The next step is to call GetSequencePoints with the number of points you want and the corresponding arrays of offsets, lines, columns, end lines, end columns and ISymUnmanagedDocument. In my case, I’m only interested in where the method starts so the first sequence point is good enough:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


// Get sequence points (source line information)
ULONG32 cPoints = 0;
hr = pMethod->GetSequencePointCount(&cPoints);
if (SUCCEEDED(hr) && (cPoints > 0))
{
    cPoints = 1; // We only need the first sequence point for start line
    std::vector<ULONG32> offsets(cPoints);
    std::vector<ULONG32> lines(cPoints);
    std::vector<ULONG32> columns(cPoints);
    std::vector<ULONG32> endLines(cPoints);
    std::vector<ULONG32> endColumns(cPoints);
    std::vector<ISymUnmanagedDocument*> documents(cPoints);

    ULONG32 actualCount = 0;
    hr = pMethod->GetSequencePoints(
        cPoints,
        &actualCount,
        &offsets[0],
        &documents[0],
        &lines[0],
        &columns[0],
        &endLines[0],
        &endColumns[0]
    );

    if (SUCCEEDED(hr) && (actualCount > 0))
    {

The source file is described by ISymUnmanagedDocument that provides its name when GetURL is called:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


// Get the first sequence point's document and line
        ISymUnmanagedDocument* pDoc = documents[0];
        if (pDoc != nullptr)
        {
            // Get document URL (file path)
            ULONG32 urlLen = 0;
            hr = pDoc->GetURL(0, &urlLen, NULL);
            if (SUCCEEDED(hr) && (urlLen > 0))
            {
                std::vector<WCHAR> url(urlLen);
                hr = pDoc->GetURL(urlLen, &urlLen, &url[0]);
                if (SUCCEEDED(hr))
                {
                    // Convert wide string to narrow string
                    int len = WideCharToMultiByte(CP_UTF8, 0, &url[0], urlLen, NULL, 0, NULL, NULL);
                    std::string narrowUrl(len, '\0');
                    WideCharToMultiByte(CP_UTF8, 0, &url[0], urlLen, &narrowUrl[0], len, NULL, NULL);
                    info.sourceFile = narrowUrl;
                }
            }

            // NOTE: 0xFEEFEE is a special value indicating hidden lines
            info.lineNumber = lines[0];
            pDoc->Release();
        }
    }
}

The final interesting trick is that the line number might have the special 0xFEEFEE value. It means that the line is hidden. I have seen it for methods generated by the C# compiler such as MoveNext for async state machines or anonymous methods:

The source code is available from my Github repository.

Happy coding!

References

But where is my method code? DbgHelp comes to the rescue

Fri, 16 Jan 2026 08:21:30 +0000

Introduction

In our Datatog continuous .NET profiler implementation, we collect the call stack of a thread when something interesting happens such as an exception is thrown for example. In addition to the method name we would like to figure out at what line in which source code file this method is implemented.

This information is usually stored in the program database (.pdb) file that is generated by the compiler when the assembly is generated from the source code. The type and the name of the method are stored in the metadata of the assembly itself but I already told this story before. The .NET compilers support two formats of .pdb: the Portable format for .NET Core and the Windows format for .NET Framework.

I’ve explained how to use the DIA API and it is now time to show how to leverage the DbgHelp API that is available on all Windows machines (even though it is recommended to always install the latest release via the Debugging Tools for Windows.

This time, my goal is to extract from a Windows .pdb file the source code and line information for a given managed method. You can find the corresponding source code of this DumpLine tool in my Github repository.

Starting with DbgHelp

There are two major ways to get access to symbols with DbgHelp: either from a running process (i.e. to map the currently loaded .dll to their associated .pdb files) or from a tool that would explicitly load a .dll or a .pdb file.

Before anything, you tell DbgHelp which options you want by calling SymSetOptions:

1
2
3
4
5
6
7
8


DWORD options = SymGetOptions();
    options |= SYMOPT_DEBUG;
    options |= SYMOPT_LOAD_LINES;           // Load line number information
    options |= SYMOPT_UNDNAME;              // Undecorate symbol names
    //options |= SYMOPT_DEFERRED_LOADS;       // Defer symbol loading
    options |= SYMOPT_EXACT_SYMBOLS;        // Require exact symbol match
    options |= SYMOPT_FAIL_CRITICAL_ERRORS; // Don't show error dialogs
    SymSetOptions(options);

This is where I ask that line number information should be collected.

The next step is to call SymInitialize to setup DbgHelp environment. The first parameter expects a process handle (returned by GetCurrentProcess in my case). You could pass a path where to find the .pdb files for your dlls as a second parameter. In my case, since I will provide a .pdb file path, I don’t need it, and NULL will be passed. It means that, if needed, DbgHelp will use the current folder and the path set in _NT_SYMBOL_PATH and _NT_ALTERNATE_SYMBOL_PATH environment variables.

The last boolean parameter tells DbgHelp if you want that SymLoadModule64 to be called for each and every loaded .dll in the given process. Definitively not what I want so I’m passing FALSE.

1
2
3
4
5


_hProcess = GetCurrentProcess();
    if (!SymInitialize(_hProcess, NULL, FALSE))
    {
        _hProcess = NULL;
    }

At that point, I’m ready to load a .pdb file.

Loading a .pdb file

The API is straightforward: just call SymLoadModuleEx :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


_baseAddress = SymLoadModuleEx(
        _hProcess,
        NULL,
        pdbFilePath.c_str(),
        NULL,
        0x10000000, // arbitrary base address
        0,
        NULL,
        0
    );

    if (_baseAddress == 0)
    {
        return false;
    }

The important parameters are the process handle (same as for SymInitialize) and the path of the .pdb file. I’ve lost some time trying to understand why my code was not working due to a weird behavior of this function. You know that it succeeds when the returned address is not 0. Well… This is not 100% correct. If the path you provide does not exist, you won’t get 0 but the base address that you also provide. Even worth, when you call the functions I’ll detail later on, no error will happen but nothing will work as expected. So, I simply check that the file exists:

1
2
3
4
5


// BUG? : dbghelp does not fail if the .pdb file does not exist...
    if (GetFileAttributesA(pdbFilePath.c_str()) == INVALID_FILE_ATTRIBUTES)
    {
        return false;
    }

Note that it is possible to unload the symbols of a given loaded module by calling SymUnloadModule with the same process handle and its base address: this will reduce the memory consumption if you don’t need the symbols anymore.

In case of deferred load symbols, it is needed to call SymGetModuleInfo64 before trying to access the symbols:

1
2
3
4
5
6


IMAGEHLP_MODULE64 moduleInfo = { 0 };
    moduleInfo.SizeOfStruct = sizeof(IMAGEHLP_MODULE64);
    if (!SymGetModuleInfo64(_hProcess, _baseAddress, &moduleInfo))
    {
        return false;
    }

In addition, this will fill up an IMAGEHLP_MODULE64 structure with possibly interesting details:

The .pdb signature and age could be useful to build urls to communicate with symbol servers; but this is another story:

1
2
3
4
5
6
7
8
9


_age = moduleInfo.PdbAge;
    GUID guid = moduleInfo.PdbSig70;
    char strGUID[80];
    sprintf_s(strGUID, 80, "%08x%04x%04x%02x%02x%02x%02x%02x%02x%02x%02x",
        guid.Data1, guid.Data2, guid.Data3,
        guid.Data4[0], guid.Data4[1], guid.Data4[2], guid.Data4[3],
        guid.Data4[4], guid.Data4[5], guid.Data4[6], guid.Data4[7]
        );
    _guid = strGUID;

You also know if line numbers are available or not thanks to the LineNumbers field.

Note that if you asked for deferred symbols option, you won’t get any interesting details:

Only the module name and path are provided but nothing else.

Enumerating the methods

It is now time to iterate on the symbols in the loaded .pdb thanks to SymEnumSymbols:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


if (!SymEnumSymbols(
            _hProcess,
            _baseAddress,
            "*!*",  // Mask (all symbols)
            EnumMethodSymbolsCallback,
            this    // User context to store the methods in _methods instance field
    ))
    {
        return false;
    }

In addition to the obvious parameters, this function expects a callback function that will be called for each symbol in the module specified by the process handle and the base address. Note that you can pass any context as the last parameter. In my case, the instance of my DbgHelpParser class is passed to be able to store the methods in a dedicated _methods field:

1
2
3
4
5
6
7
8
9


struct MethodInfo
{
    std::string name;
    uint64_t address;
    uint32_t size;
    std::string sourceFile;
    uint32_t lineNumber;
};
std::vector<MethodInfo> _methods;

The “*!*” mask tells DbgHelp to look for symbols in all modules. This might sound counter intuitive, but the syntax is similar to what you find in WinDBG or Visual Studio: !. This could be useful if you load more than one .pdb.

The job of the callback function is to detect the symbols you are interested in from the SYMBOL_INFO structure passed for each matching symbol:

1
2
3
4
5
6
7
8
9


BOOL CALLBACK DbgHelpParser::EnumMethodSymbolsCallback(PSYMBOL_INFO pSymInfo, ULONG SymbolSize, PVOID UserContext)
{
    DbgHelpParser* parser = reinterpret_cast<DbgHelpParser*>(UserContext);

    if (
        (pSymInfo->Tag == SymTagFunction) &&
        ((pSymInfo->Flags & (SYMFLAG_CLR_TOKEN | SYMFLAG_METADATA)) == (SYMFLAG_CLR_TOKEN | SYMFLAG_METADATA))
        )
    {

The Tag field contains a value from SymTagEnum but, for a managed .pdb file, you will only get SymTagFunction. Also, the Flags field should contain SYMFLAG_CLR_TOKEN and SYMFLAG_METADATA because we are only interested in managed methods.

Next, you get the name, address and size from other fields before looking for the source file and line details by calling SymGetLineFromAddr64:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


MethodInfo info;
        info.name = pSymInfo->Name;
        info.address = pSymInfo->Address;
        info.size = pSymInfo->Size;

        // Try to get source file and line information
        IMAGEHLP_LINE64 line = { 0 };
        line.SizeOfStruct = sizeof(IMAGEHLP_LINE64);
        DWORD displacement = 0;

        if (SymGetLineFromAddr64(parser->_hProcess, pSymInfo->Address, &displacement, &line))
        {
            info.sourceFile = line.FileName ? line.FileName : "";
            info.lineNumber = line.LineNumber;
        }
        else {
            info.sourceFile = "";
            info.lineNumber = 0;
        }

        parser->_methods.push_back(info);
    }

    return TRUE; // Continue enumeration
}

The callback returns TRUE to continue the enumeration. You could return FALSE if you would look for specific symbols and wanted to speed up the processing.

The managed side of the story

When the tool is run on a managed assembly, you get the following kind of output:

The name of the method does not match at all with the names in my test assembly! Why do all methods have this generic Method.# format?

Well… the number corresponds to the RID of the corresponding method in the metadata of the assembly. Let’s have a look at what the MethodDef metadata table of this assembly looks like in ILSpy:

The RID column corresponds to the number in the name in the tool output. So, the Method.#3 is the get_Records property getter in PDBFormatReaderTest.cs:28. And this is exactly what I can see in the test source code:

You can also check that the lines look correct compared to what is listed by the tool:

FilePath getter at line 19
FilePath setter at line 20
Records getter at line 28
Records setter at line 29

Feel free to look at the source code from my Github repository.

DbgHelp provides many more services to look into symbols but that’s all for today!

How to dump function symbols from a .pdb file

Mon, 08 Dec 2025 10:12:20 +0000

During this R&D week at Datadog, I wanted to implement a tool accepting a .pdb file and generate a .sym file listing functions symbols with their address, size, name with signature and if they are public or private. This post dig into the implementation details of using Microsoft Debug Interface Access (DIA) COM API to achieve these objectives. If you want to see what my vibe coding experience in Cursor was, read this other post instead.

One self-contained tool please!

I would like the tool to be self-contained but since DIA is based on a COM server, it would require registering msdia40.dll on the machine. Not a good idea. In case the dll is in the same folder as the tool, one could “emulate” the magic done by CoCreateInstance to get an instance of IDiaDataSource (more on this interface soon) by:

Call LoadLibrary to load the dll in memory
Call GetProcAddress to get the DllGetClassObject implementation
Call this function to get the IClassFactory implementation
Call its CreateInstance method to get an object implementing IDiaDataSource

Here is the corresponding code (without error checking for readability)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


// Create DIA data source without registration using DLL loading
HRESULT PdbSymbolExtractor::NoRegCoCreate(const std::wstring& dllPath, REFCLSID rclsid, REFIID riid, void** ppv) {
    HMODULE hDll = LoadLibraryW(dllPath.c_str());

    typedef HRESULT(__stdcall* DllGetClassObjectFunc)(REFCLSID, REFIID, LPVOID*);
    DllGetClassObjectFunc pDllGetClassObject = (DllGetClassObjectFunc)GetProcAddress(hDll, "DllGetClassObject");

    CComPtr<IClassFactory> pClassFactory;
    HRESULT hr = pDllGetClassObject(rclsid, IID_IClassFactory, (void**)&pClassFactory);

    hr = pClassFactory->CreateInstance(NULL, riid, ppv);

    // Note: We intentionally don't call FreeLibrary here because the DLL needs to stay loaded
    // The COM object references will keep it alive
    return S_OK;

This function is called with the path of msdia40.dll and the UUID of the expected IDiaDataSource:

1
2


// Create DIA data source without registration
hr = NoRegCoCreate(dllPath, CLSID_DiaSource, __uuidof(IDiaDataSource), (void**)&_pDiaDataSource);

But still, I don’t want to have two binaries!

The trick is to embed the msdia40.dll inside the tool as a Windows resource. In the .rc file, add an RCDATA entry that points to the dll:

1

IDR_MSDIA_DLL      RCDATA      "x64\\Release\\msdia140.dll"

You should see it in the Resource View in Visual Studio:

Here is the code that extracts it as a file on disk is straightforward (error checking has been removed for readability):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


// Extract embedded msdia140.dll from resources
bool PdbSymbolExtractor::ExtractEmbeddedDll(const std::wstring& outputPath) {
    // Find the resource
    HMODULE hModule = GetModuleHandle(NULL);
    HRSRC hResource = FindResource(hModule, MAKEINTRESOURCE(IDR_MSDIA_DLL), RT_RCDATA);

    // Load the resource
    HGLOBAL hLoadedResource = LoadResource(hModule, hResource);

    // Lock the resource to get a pointer to the data
    LPVOID pResourceData = LockResource(hLoadedResource);

    // Get the size of the resource
    DWORD resourceSize = SizeofResource(hModule, hResource);

    // Write the DLL to disk
    std::ofstream outFile(outputPath, std::ios::binary);
    outFile.write(static_cast<const char*>(pResourceData), resourceSize);
    outFile.close();

    return true;
}

Where are my function symbols?

After calling NoRegCoCreate(), _pDiaDataSource stores a reference to the entry point into the DIA APIs. Here are the steps to follow before being able to list the symbols:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


HRESULT PdbSymbolExtractor::ExtractSymbolsFromPdb(const std::wstring& pdbPath, std::vector<FunctionSymbol>& symbols) 
{
    // Load the PDB file
    HRESULT hr = _pDiaDataSource->loadDataFromPdb(pdbPath.c_str());

    // Open a session
    CComPtr<IDiaSession> pSession;
    hr = _pDiaDataSource->openSession(&pSession);

    // Get the global scope
    CComPtr<IDiaSymbol> pGlobal;
    hr = pSession->get_globalScope(&pGlobal);

Now, you have the global scope of the symbols, you can ask for an enumerator for the type of symbols you are interested in; SymTagFunction in my case:

1
2
3
4
5
6
7


// Enumerate all function symbols
    CComPtr<IDiaEnumSymbols> pEnumSymbols;
    hr = pGlobal->findChildren(SymTagFunction, NULL, nsNone, &pEnumSymbols);

    LONG count = 0;
    pEnumSymbols->get_Count(&count);
    std::wcout << L"Found " << count << L" function symbols" << std::endl;

The pEnumSymbols iterator allows you to loop on each SymTagFunction symbol and get its name:

1
2
3
4
5
6
7
8
9


while (SUCCEEDED(pEnumSymbols->Next(1, &pSymbol, &celt)) && celt == 1) {
        FunctionSymbol func;

        // Get function name
        BSTR bstrName;
        if (pSymbol->get_name(&bstrName) == S_OK) {
            func.name = bstrName;
            SysFreeString(bstrName);
        }

Note that each symbol details are stored in a FunctionSymbol instance:

1
2
3
4
5
6
7
8


struct FunctionSymbol {
    std::wstring name;
    ...
    std::wstring signature; // Function signature (parameters only, no return type)
    DWORD rva;              // Relative Virtual Address
    ULONGLONG length;
    bool isPublic;
};

with the rest of the code in the while() loop:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


// Get function signature (parameters only, no return type)
    func.signature = ExtractFunctionSignature(pSymbol);

    // Get relative virtual address
    DWORD rva;
    if (pSymbol->get_relativeVirtualAddress(&rva) == S_OK) {
        func.rva = rva;
    }

    // Get function length
    ULONGLONG length;
    if (pSymbol->get_length(&length) == S_OK) {
        func.length = length;
    }

    // Determine if function is public or private
    // Check access level - default to private
    func.isPublic = false;
    DWORD access;
    if (pSymbol->get_access(&access) == S_OK) {
        func.isPublic = (access == CV_public);
    }

    pSymbol.Release();

I did not have the time to do more trial for private/public state, but I should have tried by enumerating SymTagPublicSymbol or SymTagExport that could be considered as public.

Better with a signature

The final step is to figure out the signature of each function. This is where the genericity of DIA could be confusing because so many things are represented by IDiaSymbol: a symbol, a function, the type of a function, or the type of a parameter…

So, the type of the function is retrieved as an IDiaSymbol by calling getType() on the function symbol. From that IDiaSymbol, findChildren() lets you iterate on the parameters:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


// Extract function signature (parameters only, no return type)
std::wstring PdbSymbolExtractor::ExtractFunctionSignature(IDiaSymbol* pSymbol) {
    if (!pSymbol) {
        return L"()";
    }

    // Get function type
    CComPtr<IDiaSymbol> pFunctionType;
    if (pSymbol->get_type(&pFunctionType) != S_OK || !pFunctionType) {
        return L"()";
    }

    // Enumerate function arguments
    CComPtr<IDiaEnumSymbols> pEnumArgs;
    if (FAILED(pFunctionType->findChildren(SymTagFunctionArgType, NULL, nsNone, &pEnumArgs))) {
        return L"()";
    }

    LONG argCount = 0;
    pEnumArgs->get_Count(&argCount);

    if (argCount == 0) {
        return L"()";
    }

Now, the same Next() method is called on the enumerator to iterate on each parameter:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


// Build signature string
    std::wstring signature = L"(";
    CComPtr<IDiaSymbol> pArg;
    ULONG argCelt = 0;
    bool first = true;

    while (SUCCEEDED(pEnumArgs->Next(1, &pArg, &argCelt)) && argCelt == 1) {
        if (!first) {
            signature += L", ";
        }
        first = false;

        // Get the argument type
        CComPtr<IDiaSymbol> pArgType;
        if (pArg->get_type(&pArgType) == S_OK && pArgType) {
            signature += GetTypeName(pArgType);
        } else {
            signature += L"?";
        }

        pArg.Release();
    }

    signature += L")";
    return signature;
}

The final step is to get the name of the type from the IDiaSymbol returned by get_type(). If it is a custom type, call get_name() like any other symbol. Otherwise, for basic types, call get_baseType() and get_length() as shown by the code below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55


std::wstring PdbSymbolExtractor::GetTypeName(IDiaSymbol* pType) {
    if (!pType) {
        return L"?";
    }

    // Try to get type name directly
    BSTR bstrTypeName;
    if (pType->get_name(&bstrTypeName) == S_OK && bstrTypeName && wcslen(bstrTypeName) > 0) {
        std::wstring typeName = bstrTypeName;
        SysFreeString(bstrTypeName);
        return typeName;
    }

    // For basic types or unnamed types, try getting basic type info
    DWORD baseType = 0;
    ULONGLONG length = 0;

    if (pType->get_baseType(&baseType) == S_OK) {
        pType->get_length(&length);

        // Map basic types to names
        switch (baseType) {
            case btVoid: return L"void";
            case btChar: return L"char";
            case btWChar: return L"wchar_t";
            case btBool: return L"bool";

            case btInt:
            case btLong:
                if (length == 1) return L"char";
                else if (length == 2) return L"short";
                else if (length == 4) return L"int";
                else if (length == 8) return L"__int64";
                else return L"int" + std::to_wstring(length * 8);

            case btUInt:
            case btULong:
                if (length == 1) return L"unsigned char";
                else if (length == 2) return L"unsigned short";
                else if (length == 4) return L"unsigned int";
                else if (length == 8) return L"unsigned __int64";
                else return L"uint" + std::to_wstring(length * 8);

            case btFloat:
                if (length == 4) return L"float";
                else if (length == 8) return L"double";
                else return L"float" + std::to_wstring(length * 8);

            default:
                return L"?";
        }
    }

    return L"?";
}

This is a “simple” implementation that does not take pointers, addresses, arrays, and more into account. For a more complete solution, I would recommend looking at the PrintType() implementation in the DIA2Dump code sample that is installed with Visual Studio.

I hope this will get your foot in the door of symbol parsing and make you want to dig further into DIA.

References

Corresponding source code is available in my github repository.
Archived Microsoft documentation/implementation of .pdb format including a symbol dumper code.
DIA2Dump Visual Studio code sample.

Measuring the impact of locks and waits on latency in your .NET apps

Mon, 13 Jan 2025 15:31:03 +0000

Introduction

In an old post, I detailed how to use ContentionStart and ContentionStop events to measure the lock contentions duration for a .NET application. In a .NET 9 pull request, a former Criteo’s colleague Grégoire Verdier has added new events to be notified when wait time similar to lock contention is happening for Mutex, Semaphore, Manual/AutoResetEvent. Read his post for more details about what he was trying to investigate.

With asynchronous and multi-threaded algorithms, it is essential to detect unexpected wait/locks in our applications. This post shows you how to leverage these events to measure the duration of these waits and get the call stack when the wait started:

New WaitHandleWait events

These new events are emitted by the Microsoft-Windows-DotNETRuntime CLR provider when you enable the WaitHandle (= 0x40000000000) keyword with Verbose verbosity. Each time WaitOne is called on a waitable object and this object is already owned, a WaitHandleWaitStart event is emitted. When the object is released, a WaitHandleWaitStop event is emitted.

For example, the following code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32


static Mutex mutex = new Mutex();

static void Main()
{
    var owningThread = new Thread(OwningThread);
    owningThread.Start();
    var mutexThread = new Thread(MutexThread);
    mutexThread.Start();

    owningThread.Join();
    mutexThread.Join();
}

static void OwningThread()
{
    Console.WriteLine($"    [{GetCurrentThreadId(), 8}] Start to hold resources");
    Console.WriteLine("___________________________________________");
    mutex.WaitOne();

    Thread.Sleep(3000);  // the wait should last ~3 seconds

    Console.WriteLine("    Release resources");
    mutex.ReleaseMutex();
}

static void MutexThread()
{
    Console.WriteLine($"    [{GetCurrentThreadId(), 8}] waiting for Mutex...");
    mutex.WaitOne();  // events are emitted in the implementation when a contention happens
    mutex.ReleaseMutex();
    Console.WriteLine("    <-- Mutex");
}

generates a Start and Stop events pair:

1
2


125980 | 00000000-0000-0000-0000-000000000000 > event 301 __ [ 1| Start] WaitHandleWait/Start
125980 | 00000000-0000-0000-0000-000000000000 > event 302 __ [ 2|  Stop] WaitHandleWait/Stop

There is no associated activity ID so you rely on the fact that the same waiter thread (125980 in the previous example) is emitting for both events.

Listening to the new Wait events

As usual, you should rely on the TraceEvent nuget to start an EventPipe session with an already running .NET application. The last version already contains the definition of the keyword:

1

keywords |= ClrTraceEventParser.Keywords.WaitHandle; // .NET 9 WaitHandle kind of contention

and the C# events for Start and Stop:

1
2


source.Clr.WaitHandleWaitStart += OnWaitHandleWaitStart;
source.Clr.WaitHandleWaitStop += OnWaitHandleWaitStop;

The handler’s implementation is straightforward. The start of the wait is recorded for the current thread:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


private void OnWaitHandleWaitStart(WaitHandleWaitStartTraceData data)
{
    // get the contention info for the current thread
    ContentionInfo info = _contentionStore.GetContentionInfo(data.ProcessID, data.ThreadID);
    if (info == null)
        return;

    // keep track of the wait start
    info.ContentionStartRelativeMSec = data.TimeStampRelativeMSec;
}

When the wait ends, the duration is computed based on the recorded wait start because it is not provided in the payload like for ContentionStop:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


private void OnWaitHandleWaitStop(WaitHandleWaitStopTraceData data)
{
    ContentionInfo info = _contentionStore.GetContentionInfo(data.ProcessID, data.ThreadID);
    if (info == null)
        return;
    // unlucky case when we start to listen just after the WaitHandleStart event
    if (info.ContentionStartRelativeMSec == 0)
    {
        return;
    }

    // Too bad the duration is not provided in the payload like in ContentionStop...
    var contentionDurationMSec = data.TimeStampRelativeMSec - info.ContentionStartRelativeMSec;
    info.ContentionStartRelativeMSec = 0;
    var duration = TimeSpan.FromMilliseconds(contentionDurationMSec);
    Console.WriteLine($"{e.ThreadId,7} | {e.Duration.TotalMilliseconds} ms");
}

This is nice but it would be more useful if we could get the call stack of long waits.

Call stacks with EventPipe

In a previous post, I explained that it is possible to get the call stack when an event is emitted thanks to the ClrStackWalk event that follows the event you are interested in. Unfortunately, this is not more the case for .NET 5+ that is using EventPipe instead of ETW.

As Olivier Coanet presents in his post, you can get the call stack as an array of addresses from the hidden event record that is mapped by the TraceEvent parameter passed to each event handlers. This EVENT_RECORD structure contains a ExtendedData field that is an array of EVENT_HEADER_EXTENDED_DATA_ITEM:

1
2
3
4
5
6
7
8


public struct EVENT_HEADER_EXTENDED_DATA_ITEM
{
    public ushort Reserved1;
    public ushort ExtType;
    public ushort Reserved2;
    public ushort DataSize;
    public ulong DataPtr;
}

If the ExtType value is EVENT_HEADER_EXT_TYPE_STACK_TRACE64 (=6) then DataPtr points to a EVENT_EXTENDED_ITEM_STACK_TRACE64 structure:

1
2
3
4
5


public struct EVENT_EXTENDED_ITEM_STACK_TRACE64
{
    public ulong MatchId;
    public unsafe fixed ulong Address[1];
}

that contains an array of 64-bit addresses. The size of this array is given by DataSize — sizeof(ulong).

For 32-bit applications, you will get EVENT_HEADER_EXT_TYPE_STACK_TRACE32 (=5) as ExtType value and DataPtr will point to EVENT_EXTENDED_ITEM_STACK_TRACE32:

1
2
3
4
5


public struct EVENT_EXTENDED_ITEM_STACK_TRACE32
{
    public ulong MatchId;
    public unsafe fixed uint Address[1];
}

that stores an array of 32-bit addresses.

Knowing that makes writing the code to get the call stacks as an array of 64-bit addresses (same with 32-bit applications for simplicity sake) pretty straightforward:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50


public static EventPipeUnresolvedStack ReadStackUsingUnsafeAccessor(TraceEvent traceEvent)
{
    return GetFromEventRecord(traceEvent.eventRecord);
}

private static EventPipeUnresolvedStack GetFromEventRecord(TraceEventNativeMethods.EVENT_RECORD* eventRecord)
{
    if (eventRecord == null)
        return null;

    var extendedDataCount = eventRecord->ExtendedDataCount;

    for (var dataIndex = 0; dataIndex < extendedDataCount; dataIndex++)
    {
        var extendedData = eventRecord->ExtendedData[dataIndex];
        if (extendedData.ExtType == TraceEventNativeMethods.EVENT_HEADER_EXT_TYPE_STACK_TRACE64)
        {
            var stackRecord = (TraceEventNativeMethods.EVENT_EXTENDED_ITEM_STACK_TRACE64*)extendedData.DataPtr;
            var addresses = &stackRecord->Address[0];
            var addressCount = (extendedData.DataSize - sizeof(UInt64)) / sizeof(UInt64);
            if (addressCount == 0)
                return null;

            var callStackAddresses = new ulong[addressCount];
            for (var index = 0; index < addressCount; index++)
            {
                callStackAddresses[index] = addresses[index];
            }
            return new EventPipeUnresolvedStack(callStackAddresses);
        }
        else if (extendedData.ExtType == TraceEventNativeMethods.EVENT_HEADER_EXT_TYPE_STACK_TRACE32)
        {
            var stackRecord = (TraceEventNativeMethods.EVENT_EXTENDED_ITEM_STACK_TRACE32*)extendedData.DataPtr;
            var addresses = &stackRecord->Address[0];
            var addressCount = (extendedData.DataSize - sizeof(UInt32)) / sizeof(UInt32);
            if (addressCount == 0)
                return null;

            var callStackAddresses = new ulong[addressCount];  // store the 32 addresses as 64 bit addresses
            for (var index = 0; index < addressCount; index++)
            {
                callStackAddresses[index] = addresses[index];
            }

            return new EventPipeUnresolvedStack(callStackAddresses);
        }
    }

    return null;
}

Note that the last version of TraceEvent nuget provides a public access to the eventRecord field so it is no more needed to use the UnsafeAccessor attribute used by Olivier.

Symbolize the call stack addresses

Address is good but the corresponding method name is better. I won’t repeat what I’ve already detailed in an older post that shows how to get the name of a native and managed name from an instruction pointer address. Instead, I want to pinpoint a big limitation of this solution to listen to CLR provider MethodLoadVerbose/MethodDCStartVerboseV2 events. If the methods you are interested in are jitted BEFORE your tool attaches to the application, you will never get these events.

You could get the same mapping address span/method name via the other “Microsoft-Windows-DotNETRuntimeRundown” provider and its MethodDCEndVerbose event that contains the expected MethodStartAddress, MethodSize and MethodName in its payload. But I need this information before the end of the application…

Looking at the documentation, it seems that the rundown provider accepts the StartRundownKeyword value to emit the DCStart events when the provider is enabled! Since .NET 9, it is possible to pass the keywords you want (before, the default value did not contain StartRundownKeyword) when creating the EventPipe session

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


//                V-- this is the default rundown keyword
rundownKeywords = 0x80020139 | (long)ClrTraceEventParser.Keywords.StartEnumeration;
var config = new EventPipeSessionConfiguration(GetProviders(), 256, rundownKeywords, true);
using (var session = client.StartEventPipeSession(config))
{
    var source = new EventPipeEventSource(session.EventStream);
    RegisterListeners(source);

    // this is a blocking call
    source.Process();
}

Note that you should not add the rundown provider to the list passed as parameter.

Unfortunately, there is currently an issue in the runtime since September 2020 that pinpoints this exact problem. I even tried to create and close a session to get the DCStop events before recreating a new one, but I failed.

The next episode will talk about how it is possible to start a .NET application and get the events since its startup… with the problems that are happening.

Monitor HTTP redirects to reduce unexpected latency

Fri, 13 Dec 2024 14:21:41 +0000

In the previous post, I detailed how I used the undocumented events from the BCL to create the dotnet-http CLI tool to monitor your outgoing HTTP requests. After testing with older versions of .NET, I realized that the code needed to be updated and I’m sharing my findings in this post.

The main point is that url redirections could have a major impact on requests latency:

Always test supported versions…

When I wrote the initial version of dotnet-http, I only tested it with .NET 8 and .NET 9 with limited formats of urls. Unfortunately, things went bad when I tried to monitor applications running on .NET 5 and .NET 6: no events are emitted by these versions of the BCL.

So, the next step was to test .NET 7 and the result was simple: crash! After investigating, I realized that some events I looked at in .NET 8 source code did not have the same payload in .NET 7; even no payload at all:

Even though there is no version field in the events payload, it is easy to check its size such as the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


private void OnConnectionEstablished(
     DateTime timestamp,
     int threadId,
     Guid activityId,
     Guid relatedActivityId,
     byte[] eventData
     )
 {
     ...
     EventSourcePayload payload = new EventSourcePayload(eventData);
     var versionMajor = payload.GetByte();
     var versionMinor = payload.GetByte();

     // in .NET 7, nothing else is available
     Int64 connectionId = 0;
     var scheme = "";
     var host = "";
     UInt32 port = 0;
     var path = "";

     if (eventData.Length > 2)
     {
         connectionId = payload.GetInt64();
         scheme = payload.GetString();
         host = payload.GetString();
         port = payload.GetUInt32();
         path = payload.GetString();
     }
     ...

Another difference is that one even is not even emitted in .NET 7:

Explain what a redirection is please!

Because I based the code on this Redirect event, I needed to find another way to support .NET 7 even though I would not have a redirected url to display. But first, let’s see what I’m talking about in terms of HTTP communication.

When you try to get the content / status code behind a url, the code is following different phases with the related events:

**Start **RequestStart
**DNS resolution **ResolutionStart ResolutionStop/Fail
**Socket connection **ConnectStart ConnectStop
**Security hand check (HTTPS only) **HandshakeStart HandshakeStop/Failed
**Request/response **RequestHeadersStart RequestHeadersStop ResponseHeadersStart ResponseHeadersStop Redirect (.NET 8+) ResponseContentStart ResponseContentStop
**Request stop **RequestStop/Failed

Based on the received url, a server can decide to answer that another url should be used instead. For example, if you call github with http:// instead of https://, such a redirection will happen. Without invasive tools such as Wireshark, these redirections are impossible to detect and could cause unnecessary delay.

From the client perspective, this can be detected in the ResponseHeadersStop event payload that provides a status code. If its value is 301, then it is a redirection. The other effect of a redirection is that the BCL code will change the flow of events because it needs to start over with the new url from step 2. to step 6. As you can see, instead of paying the cost of just one request, two are actually emitted and processed.

Impact on the implementation

In addition to the payload size checks addition, my initial implementation was not properly handling the redirection because the values (timestamps and durations) where overridden by the events related to the second redirected url.

The new implementation is splitting the request details into two classes. A base class that contains common fields to both parts of the request in case of redirection:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


private class HttpRequestInfoBase
   {
       public HttpRequestInfoBase(DateTime timestamp, string scheme, string host, uint port, string path)
       {
           StartTime = timestamp;
           if (scheme == string.Empty)
           {
               Url = string.Empty;
           }
           else
           {
               if (port != 0)
               {
                   Url = $"{scheme}://{host}:{port}{path}";
               }
               else
               {
                   Url = $"{scheme}://{host}:{path}";
               }
           }
       }

       public string Url { get; set; }
       public DateTime StartTime { get; set; }
       public DateTime ReqRespStartTime { get; set; }
       public double ReqRespDuration { get; set; }

       // DNS
       public double DnsWait { get; set; }
       public DateTime DnsStartTime { get; set; }
       public double DnsDuration { get; set; }

       // HTTPS
       public double HandshakeWait { get; set; }
       public DateTime HandshakeStartTime { get; set; }
       public double HandshakeDuration { get; set; }

       // socket connection
       public DateTime SocketConnectionStartTime { get; set; }
       public double SocketWait { get; set; }
       public double SocketDuration { get; set; }

       public DateTime QueueuingEndTime { get; set; }
       public double QueueingDuration { get; set; }
   }

The second one inherits from the base class and contains addition details; including the details of the redirected url if any:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


private class HttpRequestInfo : HttpRequestInfoBase
   {
       public HttpRequestInfo(DateTime timestamp, string scheme, string host, uint port, string path)
           :
           base(timestamp, scheme, host, port, path)
       {
       }

       public HttpRequestInfoBase Redirect { get; set; }

       public UInt32 StatusCode { get; set; }

       // HTTPS
       public string HandshakeErrorMessage { get; set; }

       public string Error { get; set; }
   }

A new instance of HttpRequestInfoBase is created when a 301 status code is received in HttpResponseHeaderStop handler:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


private void OnHttpResponseHeaderStop(object sender, HttpRequestStatusEventArgs e)
    {
        // used to detect redirection in .NET 8+
        if (e.StatusCode != 301)
        {
            return;
        }

        // create a new request info for the redirected request
        // because .NET 7 does not emit a Redirect event, we need to create a new request info here
        // --> it means that the redirect url will be empty in .NET 7
        var root = GetRoot(e.ActivityId);
        if (_requests.TryGetValue(root, out HttpRequestInfo info))
        {
            info.Redirect = new HttpRequestInfoBase(e.Timestamp, "", "", 0, "");

            // if you really want to have the duration of both original request + redirected request,
            // then do the following:
            //    info.ReqRespDuration = (e.Timestamp - info.ReqRespStartTime).TotalMilliseconds;
            // However, I prefer to show the duration of the redirected request only to more easily
            // compute the cost of the initial redirected request = total duration - other durations
        }
    }

For .NET 8+, the redirected url is provided to the Redirect handler and stored in the Url field of the instance created in the previous handler:

1
2
3
4
5
6
7
8
9


private void OnHttpRedirect(object sender, HttpRedirectEventArgs e)
    {
        // since this is an Info event, the activityID is the root
        var root = ActivityHelpers.ActivityPathString(e.ActivityId);
        if (_requests.TryGetValue(root, out HttpRequestInfo info))
        {
            info.Redirect.Url = e.RedirectUrl;
        }
    }

In each handler of events that could be received for both initial and redirected requests,

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


private void OnHttpResponseContentStop(object sender, EventPipeBaseArgs e)
   {
       var root = GetRoot(e.ActivityId);
       if (!_requests.TryGetValue(root, out HttpRequestInfo info))
       {
           return;
       }

       if (info.Redirect == null)
       {
           info.ReqRespDuration = (e.Timestamp - info.ReqRespStartTime).TotalMilliseconds;
       }
       else
       {
           info.Redirect.ReqRespDuration = (e.Timestamp - info.Redirect.ReqRespStartTime).TotalMilliseconds;
       }
   }

The wait time and durations are now computed as the events are received and aggregated at the end of the request by adding the value of both parts (initial and redirected if any:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


double dnsDuration = info.DnsDuration + ((info.Redirect != null) ? info.Redirect.DnsDuration : 0);
    if (dnsDuration > 0)
    {
        double dnsWait = info.DnsWait + ((info.Redirect != null) ? info.Redirect.DnsWait : 0);
        Console.Write($"{dnsWait,9:F3} | {dnsDuration,9:F3} | ");
    }
    else
    {
        Console.Write($"          |           | ");
    }

As a conclusion, you should try to monitor these redirections by using my dotnet-http CLI tool. Feel free to download it or install it with the following command line: dotnet tool install -g dotnet-http or update to the latest version with: dotnet tool update -g dotnet-http

You could also integrate some event listening code into your framework that simply handles the ResponseHeadersStop/Redirect events.

Unexpected usage of EventSource or how to test statistical results in CLR pull request

Fri, 13 Sep 2024 14:25:30 +0000

Testing the statistical results

In parallel of the performance impact, it is important to validate the expected statistical distribution of the sampled allocations. Basically, I need to execute the same run of allocations multiple times in a row. Each run allocates the same number of instances of different types. For example, it is interesting to know if sampling instances of types with sizes proportional to a base value gives good results. Same question for totally different sized types or with Finalizers.

I would like to pass the number of runs to execute and a given scenario to a C# runner program and listen to the emitted events in another C# listener.

I’m facing 3 issues here:

How many instances are allocated to validate the upscaling algorithm (sampled vs real count)
What are the types I want to focus on because I don’t want to hard code them in the listener application.
When does each run start?

It would be great if I could send the answer to these questions via events so the listener would know at runtime. Well… This is exactly what a class inherited from EventSource allows you to do!

In the runner application, I’ve defined the AllocationsRunEventSource that is decorated with the EventSource attribute to set its name that will be used as a provider name like Microsoft-Windows-DotNETRuntime for the .NET runtime provider.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


[EventSource(Name = "Allocations-Run")]
public class AllocationsRunEventSource : EventSource
{
    public static readonly AllocationsRunEventSource Log = new AllocationsRunEventSource();

The four implemented methods are defining which event ID for which verbosity will be emitted with which payload:
    [Event(600, Level = EventLevel.Informational)]
    public void StartRun(int iterationsCount, int allocationCount, string listOfTypes)
    {
        WriteEvent(eventId: 600, iterationsCount, allocationCount, listOfTypes);
    }

    [Event(601, Level = EventLevel.Informational)]
    public void StopRun()
    {
        WriteEvent(eventId: 601);
    }

    [Event(602, Level = EventLevel.Informational)]
    public void StartIteration(int iteration)
    {
        WriteEvent(eventId: 602, iteration);
    }

    [Event(603, Level = EventLevel.Informational)]
    public void StopIteration(int iteration)
    {
        WriteEvent(eventId: 603, iteration);
    }
}

To make the payload serialization and parsing easy, the list of types that will be allocated is passed as a string with the following format allocatedTypes = “Object24;Object48;Object72;Object32;Object64;Object96”.

The code of the runner calls these methods as expected at different moment of the execution:

1
2
3
4
5
6
7
8


AllocationsRunEventSource.Log.StartRun(iterations, allocationsCount, allocatedTypes);
 for (int i = 0; i < iterations; i++)
 {
     AllocationsRunEventSource.Log.StartIteration(i);
     allocationsRun.Allocate(allocationsCount);
     AllocationsRunEventSource.Log.StopIteration(i);
 }
 AllocationsRunEventSource.Log.StopRun();

Instead of recording the events with dotnet-trace, this time I’m using TraceEvent and Microsoft.Diagnostics.NETCore.Client to code a listener application. The code is very similar to what was presented for my dotnet-fullgc CLI tool except that I’m enabling the AllocationsRun provider corresponding to the event source of the runner in addition to the .NET runtime one:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


public static void PrintEventsLive(int processId)
{
    var providers = new List<EventPipeProvider>()
    {
        new EventPipeProvider(
                "Microsoft-Windows-DotNETRuntime",
                EventLevel.Verbose, // verbose is required for AllocationTick
                (long)0x80000000001 // new AllocationSamplingKeyword + GCKeyword
                ),
        new EventPipeProvider(
                "Allocations-Run",
                EventLevel.Informational
                ),
    };

The custom events from that provider are received via the source.Dynamic.All C# event:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


var client = new DiagnosticsClient(processId);
    using (var session = client.StartEventPipeSession(providers, false))
    {
        Console.WriteLine();

        Task streamTask = Task.Run(() =>
        {
            var source = new EventPipeEventSource(session.EventStream);
            _source = source;
            ClrTraceEventParser clrParser = new ClrTraceEventParser(source);
            clrParser.GCAllocationTick += OnAllocationTick;
            source.Dynamic.All += OnEvents;
            ...

Because TraceEvent is not already aware of the new AllocationSampled event emitted by the PR code, it will also be received via the same OnEvent handler:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


private static void OnEvents(TraceEvent eventData)
{
    if (eventData.ID == (TraceEventID)303)
    {
        // AllocationSampled parsing 
        ...
        return;
    }

    if (eventData.ID == (TraceEventID)600)  // Start run
    {
        // keep track of the expected types and the number of allocated instances 
        ...
        return;
    }

    if (eventData.ID == (TraceEventID)601)  // Stop run
    {
        // show the results of the run
        ...
        return;
    }

    if (eventData.ID == (TraceEventID)602)  // Start an iteration in a run
    {
        // reset for a new iteration
        ...
        return;
    }

    if (eventData.ID == (TraceEventID)603)  // Stop an iteration in a run
    {
        // Show iteration results
        ...
        return;
    }
}

The parsing of the payload of the run related events is done the same way as for AllocationSampled by a dedicated xxxData class:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


class AllocationsRunData
{
    const int EndOfStringCharLength = 2;
    private TraceEvent _payload;

    public AllocationsRunData(TraceEvent payload)
    {
        _payload = payload;

        ComputeFields();
    }

    public int Iterations;
    public int Count;
    public string AllocatedTypes;

    private void ComputeFields()
    {
        int offsetBeforeString = 4 + 4;

        Span<byte> data = _payload.EventData().AsSpan();
        Iterations = BitConverter.ToInt32(data.Slice(0, 4));
        Count = BitConverter.ToInt32(data.Slice(4, 4));
        AllocatedTypes = Encoding.Unicode.GetString(data.Slice(offsetBeforeString, _payload.EventDataLength - offsetBeforeString - EndOfStringCharLength));
    }
}

By keeping track of this data, it is possible to show each iteration results:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


> starts 100 iterations allocating 1000000 instances
0|
Tag  SCount  TCount          SSize          TSize   UnitSize     UpscaledSize  UpscaledCount  Name
--------------------------------------------------------------------------------------------------
 ST     247     384           5928           9216         24         24702711        1029279  Object24
 ST     322     106          10304           3392         32         32205122        1006410  Object32
 ST     435     509          20880          24432         48         43510266         906463  Object48
 ST     587     776          37568          49664         64         58718825         917481  Object64
 ST     747     481          53784          34632         72         74726662        1037870  Object72
 ST     958     916          91968          87936         96         95845392         998389  Object96

that integrates AllocationTick numbers too.

At the end of the run, error distribution per type is also computed over the iterations:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


Object72
-------------------------
   1  -10.5 %
   2   -8.9 %
   3   -8.4 %
   4   -7.8 %
   5   -6.4 %
        ...
  49    0.2 %
  50    0.2 %
  51    0.2 %
        ...
  96    6.8 %
  97    6.8 %
  98    7.7 %
  99    8.6 %
 100   10.0 %

You could use the same mechanisms (a custom EventSource to emit additional information and an EventPipe listener to aggregate the data) for your own usage. This is a different way to use EventSource rather than emitting events for monitoring like what is done by the BCL.

Testing the standalone GC

In addition to the usual .NET GC, it is needed to validate that the changes are also working for the standalone GC. Long story short, it is possible to replace the existing .NET garbage collector by your own implementation. For the test, I needed to check that the standalone GC clrgcexp.dll generated by the .NET compilation generates the expected AllocationSampled events when the corresponding keyword with informational verbosity is enabled.

Debugging NativeAOT scenarios

The final step was to implement the feature for the NativeAOT scenario. When you build your C# application for NativeAOT, a lot happens behind the scenes, based on compilers known by Visual Studio corresponding to the official released version of the .NET runtime. In my case, I needed to use the brand-new code of my local branch and debug some simple C# applications. The steps to reach that goal are not that simple.

First, you follow the steps given by the documentation to build AOT CLR in debug and libs in release:

1
2


build clr.aot+lib -rc debug -lc release
build -c release

Then, open src\coreclr\tools\aot\ilc.sln

the repro project contains a program.cs file where you write the C# code you want to test and debug.

When you build a NativeAOT application, you need to select if you want the runtime that emits events or not. This is done by setting the EventSourceSupport to true in the .csproj:

Add the following in the .csproj to get events:

1

true

However, with the repro project, you need to change ILCompiler.csproj in a different way:

1

 Include="--feature:System.Diagnostics.Tracing.EventSource.IsSupported=true" />

Also, change the reproNative.vcxproj file to bind to eventpipe-enabled.lib instead of eventpipe-disabled.lib for the platform/configuration you want to debug:

1
2
3
4
5
6


 Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
    ...
    
      ...
      ...$(ArtifactsRoot)bin\coreclr\windows.x64.Debug\aotsdk\eventpipe-enabled.lib;...
    

Then, build repro in Debug x64

Next, change the target for the ILCompiler project:

Build and run it to generate the .obj file corresponding to the repro project

Finally, open src\coreclr\tools\aot\ILCompiler\reproNative\reproNative.vcxproj that will allow you to debug the program.cs you’ve just built!

Tips and tricks from validating a Pull Request in .NET CLR

Tue, 13 Aug 2024 09:23:28 +0000

Introduction

During the implementation of our .NET allocation profiler, we realized that the current sampling mechanism based on a fixed threshold did not provide a good enough statistical distribution. With the help of Noah Falk from the CLR Diagnostics team, I started to implement a randomized sampling based on a Bernoulli distribution model for .NET.

With this kind of changes, you need to ensure that you don’t break any existing code, the impact on performance is limited and the mathematical results map the expected mathematical distribution.

The rest of this blog series details the different tests I wrote and the corresponding tips and tricks that could be reused when you write C# code.

Testing the basics

From a high-level view, the code change does something simple: each time an allocation context is needed to fulfill an allocation, the code checks if it should be sampled. In that case, a new AllocationSampled event is emitted with the same information as the existing AllocationTick event plus an additional field. So, the first level of testing is to validate that the events are emitted when the keyword and verbosity are enabled for the .NET runtime provider.

The runtime has already some tests in place to validate that some events are emitted under the** \src\tests\tracing\eventpipe** folder. Here is the code of my XUnit test that mimics the existing ones such as simpleruntimeeventvalidation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


[Fact]
public static int TestEntryPoint()
{
    // check that AllocationSampled events are generated and size + type name are correct
    var ret = IpcTraceTest.RunAndValidateEventCounts(
        new Dictionary<string, ExpectedEventCount>() { { "Microsoft-Windows-DotNETRuntime", -1 } },
        _eventGeneratingActionForAllocations,
        // AllocationSamplingKeyword (0x80000000000): 0b1000_0000_0000_0000_0000_0000_0000_0000_0000_0000_0000
        new List<EventPipeProvider>() { new EventPipeProvider("Microsoft-Windows-DotNETRuntime", EventLevel.Informational, 0x80000000000) },
        1024, _DoesTraceContainEnoughAllocationSampledEvents, enableRundownProvider: false);
    if (ret != 100)
        return ret;

    return 100;
}

The IpcTraceTest.RunAndValidateEventCounts helper method accepts:

The list of providers to enable with which keyword and verbosity level.
How many events are expected (using -1 in my case because I can’t predict how many random events will be generated
A callback with the code that will generate events (allocating a lot of instances of a custom type in my case)
A callback that looks at emitted events

The last callback code relies on TraceEvent to listen to emitted events:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


private static Func<EventPipeEventSource, Func<int>> _DoesTraceContainEnoughAllocationSampledEvents = (source) =>
{
    int AllocationSampledEvents = 0;
    int Object128Count = 0;
    source.Dynamic.All += (eventData) =>
    {
        if (eventData.ID == (TraceEventID)303)  // AllocationSampled is not defined in TraceEvent yet
        {
            AllocationSampledEvents++;

            AllocationSampledData payload = new AllocationSampledData(eventData, source.PointerSize);
            // uncomment to see the allocation events payload
            // Logger.logger.Log($"{payload.HeapIndex} - {payload.AllocationKind} | ({payload.ObjectSize}) {payload.TypeName}  = 0x{payload.Address}");
            if (payload.TypeName == "Tracing.Tests.SimpleRuntimeEventValidation.Object128")
            {
                Object128Count++;
            }
        }
    };
    return () => {
        Logger.logger.Log("AllocationSampled counts validation");
        Logger.logger.Log("Nb events: " + AllocationSampledEvents);
        Logger.logger.Log("Nb object128: " + Object128Count);
        return (AllocationSampledEvents >= MinExpectedEvents) && (Object128Count != 0) ? 100 : -1;
    };
};

In my case, I’m adding a new event that is emitted when a new keyword is enabled. It means that TraceEvent does not know yet its ID (hence the 303 hardcoded value) or how to unpack the new event payload. This is why I created the AllocationSampleData type to expose the payload as public fields:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40


class AllocationSampledData
{
    const int EndOfStringCharLength = 2;
    private TraceEvent _payload;
    private int _pointerSize;
    public AllocationSampledData(TraceEvent payload, int pointerSize)
    {
        _payload = payload;
        _pointerSize = pointerSize;
        TypeName = "?";

        ComputeFields();
    }

    public GCAllocationKind AllocationKind;
    public int ClrInstanceID;
    public UInt64 TypeID;
    public string TypeName;
    public int HeapIndex;
    public UInt64 Address;
    public long ObjectSize;
    public long SampledByteOffset;

    ...
}
And the extraction of each field from the payload is done in the ComputeFields method:
// The payload of AllocationSampled is not defined in TraceEvent yet
//
//  
//  
//  
//  
//  
//  
//  
//  
//
    private void ComputeFields()
    {
        int offsetBeforeString = 4 + 2 + _pointerSize;

This offsetBeforeString value is computed based on the size of UInt32 (=4 bytes), UInt16 (=2 bytes) and a Pointer (depends on 32 bit=4 or 64 bit=8) fields before the string. As Span wraps the binary payload provided by TraceEvent:

1

Span<byte> data = _payload.EventData().AsSpan();

Since I know the size of each field from the payload definition in ClrEtwAll.man, the numeric fields are extracted thanks to the BitConverter methods:

1
2


AllocationKind = (GCAllocationKind)BitConverter.ToInt32(data.Slice(0, 4));
        ClrInstanceID = BitConverter.ToInt16(data.Slice(4, 2));

Things start to be more complicated when you need to get the value of an address. Its size is 4 bytes in 32 bit and 8 bytes in 64 bit:

1
2
3
4
5
6
7
8


if (_pointerSize == 4)
        {
            TypeID = BitConverter.ToUInt32(data.Slice(6, _pointerSize));
        }
        else
        {
            TypeID = BitConverter.ToUInt64(data.Slice(6, _pointerSize));
        }

The bitness of the monitored application is given by the EventPipeSource’s PointerSize property that is passed to the AllocationSampledData constructor.

For the string case, you need to know that it is stored as UTF16 (so each character requires 2 bytes) with the trailing \0 and its length is the total size of the payload minus the size of the other fields. That way, you can slice the Span to properly read the characters:

1
2


//   \0 should not be included for GetString to work
        TypeName = Encoding.Unicode.GetString(data.Slice(offsetBeforeString, _payload.EventDataLength - offsetBeforeString - EndOfStringCharLength - 4 - _pointerSize - 8));

The rest of the fields are extracted with BitConverter helpers taking into account the size of the string:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


HeapIndex = BitConverter.ToInt32(data.Slice(offsetBeforeString + TypeName.Length * 2 + EndOfStringCharLength, 4));
        if (_pointerSize == 4)
        {
            Address = BitConverter.ToUInt32(data.Slice(offsetBeforeString + TypeName.Length * 2 + EndOfStringCharLength + 4, _pointerSize));
        }
        else
        {
            Address = BitConverter.ToUInt64(data.Slice(offsetBeforeString + TypeName.Length * 2 + EndOfStringCharLength + 4, _pointerSize));
        }        ObjectSize = BitConverter.ToInt64(data.Slice(offsetBeforeString + TypeName.Length * 2 + EndOfStringCharLength + 4 + 8, 8));
    }

The name of the sampled allocated type from the parsed payload is used to ensure that the expected allocations are indeed emitted when the keyword/verbosity are enabled for the .NET provider.

Testing the performance impact

The next step was to validate the impact of the changes on the GC performance. The baseline was the .NET 9 branch before the changes and in Release. The GCPerfSim library from the performance repository was used to allocate 500 GB of mixed size objects on 4 threads with a 50MB live object size. From the output, the seconds_taken line provides the duration to allocate these objects.

To ensure that you run with the rebuilt branch, you need to use the following commands:

1
2


build.cmd clr+libs -c release
src\tests\build.cmd generatelayoutonly Release

The next step is to use \artifacts\tests\coreclr\windows.x64.Release\Tests\Core_Root\corerun.exe instead of the usual dotnet.exe like the following:

1

\artifacts\tests\coreclr\windows.x64.Release\Tests\Core_Root\corerun \artifacts\bin\GCPerfSim\Release\net7.0\GCPerfSim.dll -tc 4 -tagb 500 -tlgb 0.05 -lohar 0 -sohsi 0 -lohsi 0 -pohsi 0 -sohpi 0 -lohpi 0 -sohfi 0 -lohfi 0 -pohfi 0 -allocType reference -testKind time

I run this scenario 10 times to compute the median and the average. I’m doing the same for the PR branch. So far so good. Now, how to do the same but to measure the impact of the random sampling? Remember that the code only triggers if the .NET provider is enabled with a certain keyword and verbosity. It means that you have to use a tool such as dotnet-trace to start an event pipe session but you would need the process id. I could have changed the code of GCPerfSim to show the process id but I would still need to wait for the session to have been created before starting the seconds_taken computation. Not really easy to script a 10x runs that way…

Don’t worry! dotnet-trace supports the — show-child-io true arguments that makes it start the session as the process starts and — providers allows you to enable a provider the way you want. Here is an example of the command line used for the performance runs:

1

dotnet-trace collect --show-child-io true --providers Microsoft-Windows-DotNETRuntime:0x80000000000:4 -- corerun \artifacts\bin\GCPerfSim\Release\net7.0\GCPerfSim.dll …

These dotnet-trace features are very handy for any scripting scenario unrelated to testing the CLR. For example, you could use Perfview to later on analyze how an application behaves thanks to the emitted events stored in the generated .nettrace file!

The next episode will describe unexpected usage of EventSource and debugging NativeAOT scenario.

View your GCs statistics live with dotnet-gcstats!

Fri, 01 Mar 2024 18:30:13 +0000

Introduction

While working on the second edition of Pro .NET Memory Management, it was needed to get statistics about each garbage collection to explain the condemned generation and other decisions taken by the GC. This post explains the different internal data structures used by the GC and how to get their value for each collection. Some require debugging the CLR and others are emitted via events. For the latter, I will show how I wrote the new dotnet-gcstats CLI tool to collect them and a personal Perfview GCStats displaying live data, garbage collection after garbage collection.

High level view of GC Internals

With regions, the GC keeps track of managed memory allocated by your application in instances of the gc_heap class. In Workstation mode, only 1 instance exists and in Server mode, by default, 1 instance is created per core. Each gc_heap keeps track of its 5 generations (gen0, gen1, gen2, Large Object Heap and Pinned Object Heap) in an array of 5 generation instances. Each generation references its dedicated regions wrapped by instances of heap_segment. These regions are reserved from a giant part of the process address space and committed as needed.

During a garbage collection, the GC code relies on global fields per gc_heap:

1
2
3
4


static gc_mechanisms settings;
gc_history_global gc_data_global;  // for non background GC including foreground GC during a background
gc_history_global bgc_data_global; // for background GC only
static dynamic_data dynamic_data_table[total_generation_count = 5];

The settings field contains a few interesting fields:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


class gc_mechanisms
{
public:
    gc_index; // starts from 1 for the first GC
    int condemned_generation;  // generation to collect
    BOOL compaction;  // true when compaction instead of sweep
    BOOL loh_compaction;  // true when LOH needs compaction
    uint32_t concurrent;  // 1 = concurrent/background GC 
    gc_reason reason;  // trigger reason
    gc_pause_mode pause_mode;  // see GCSettings.LatencyMode
    ...
};

including the trigger reason that will tell if your code called GC.Collect (i.e. induced), or if it was due to a LOH or SOH allocation for example. If compaction is true, a compacting GC will happen (instead of a sweeping one).

The gc/bgc_data_global contains almost the same information:

1
2
3
4
5
6
7
8
9


struct gc_history_global
{
    uint32_t num_heaps;  // number of gc_heap instances
    int condemned_generation;
    gc_reason reason;
    int pause_mode;
    uint32_t mem_pressure; 
    uint32_t global_mechanisms_p;
};

Most of the fields are available from different events:

GCStart: gc_index in Count, condemned_generation in Depth, reason in Reason
GCGlobalHeapHistory: pause_mode in PauseMode and some others in GlobalMechanisms

Which generation to collect = condemned generation

The computation of the condemned_generation is complicated and relies on many factors including metrics stored for each “generation” (gen0, gen1, gen2, LOH and POH) in an array of dynamic_data called dynamic_data_table. The dynamic_data class contains a few fields used by the GC to take decisions such as when a collection should be triggered and which generation to condemn:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


class dynamic_data
{
public:
    ptrdiff_t new_allocation;     // remaining budget = budget - allocated
    size_t    desired_allocation; // budget to trigger a GC

    // # of bytes taken by survived objects after mark.
    size_t    survived_size;

    // # of bytes taken by survived pinned plugs after mark.
    size_t    pinned_survived_size;

    // total object size after a GC, ie, doesn't include fragmentation
    size_t    current_size;
    size_t    promoted_size;
    size_t    fragmentation;
};

Most of these fields are found in the payload of GCPerHeapHistory or GCHeapStat events. However, the most interesting one, new_allocation is not available. Why is it interesting? Because it would give you which generation had its budget exceeded. It is initialized with the generation budget at the end of a GC and then, each time an allocation context gets created, its size is deducted from it. When it reaches 0, it means that the budget is exceeded, and a collection should happen.

Since I needed to debug the CLR to better understand all these algorithms, I added a breakpoint at the beginning of gc_heap::garbage_collect with the following action:

#{settings.gc_index}[{gc_trigger_reason}]{"\n",s8b} new_allocation(0) = {dynamic_data_table[0].new_allocation}{"\n",s8b} desired_allocation(0) = {dynamic_data_table[0].desired_allocation}{"\n",s8b} begin_data_size(0) = {dynamic_data_table[0].begin_data_size}{"\n",s8b} promoted_size(0) = {dynamic_data_table[0].promoted_size}{"\n",s8b}-{"\n",s8b} new_allocation(1) = 
...
{dynamic_data_table[4].new_allocation}{"\n",s8b} desired_allocation(4) = {dynamic_data_table[4].desired_allocation}{"\n",s8b} begin_data_size(4) = {dynamic_data_table[4].begin_data_size}{"\n",s8b} promoted_size(4) = {dynamic_data_table[4].promoted_size}{"\n",s8b}__________{"\n",s8b}

And now, each time a GC happens, I get the corresponding log in my Output pane in Visual Studio:

#2[reason_alloc_soh (0)]
 new_allocation(0) = -22728
 desired_allocation(0) = 134217728
 begin_data_size(0) = 8391376
 promoted_size(0) = 8383432
-
 new_allocation(1) = -5910416
 desired_allocation(1) = 2473016
 begin_data_size(1) = 375528
 promoted_size(1) = 353288
-
 new_allocation(2) = -91144
 desired_allocation(2) = 262144
 begin_data_size(2) = 0
 promoted_size(2) = 0
-
 new_allocation(3) = 28000088
 desired_allocation(3) = 28000088
 begin_data_size(3) = 8000024
 promoted_size(3) = 8000024
-
 new_allocation(4) = 3145728
 desired_allocation(4) = 3145728
 begin_data_size(4) = 32712
 promoted_size(4) = 32712

As you can see, gen0, gen1 and gen2 have all their budget exceeded (i.e. their new_allocation is negative) and it explains why a simple gen0 collection (from allocation in SOH = gen0) becomes a gen2 collection. If you wonder how gen1 and gen2 budgets are exceeded as your application is only allocating in gen0, you need to understand that when a GC copy surviving objects from one younger generation to the older, they are counted as allocations in the older and subtracted from its new_allocation metric.

The GC is encoding the different steps leading to the final condemned generation in a 32 bit value stored in a gen_to_condemn_tuning field that allows you to get:

initial condemned generation,
final generation to condemn,
which generation’s budget is exceeded.

The value of the last one corresponds to the highest generation for which its new_allocation was negative.

This information is available in the CondemnReasons0 field of the GCPerHeapHistory event, and you need some arithmetic to get the generation you want:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


private const int gen_initial = 0;          // indicates the initial gen to condemn.
private const int gen_final_per_heap = 1;   // indicates the final gen to condemn per heap.
private const int gen_alloc_budget = 2;     // indicates which gen's budget is exceeded.

private const int InitialGenMask = 0x0 + 0x1 + 0x2;

static int GetGen(int val, int reason)
{
    int gen = (val >> 2 * reason) & InitialGenMask;
    return gen;
}

Building your own tool

Even though I could dig into the different matrices available in the Perfview’s GCStats view or its export to Excel, I decided to write dotnet-gcstats. This CLI tool listens to the CLR events emitted by a .NET application thanks to Microsoft.Diagnostics.NETCore.Client (connect to the application EventPipe) and TraceEvent (receive and analyze the CLR events).

The code is amazingly simple:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


var providers = new List<EventPipeProvider>()
{
    new EventPipeProvider("Microsoft-Windows-DotNETRuntime",
        EventLevel.Informational, (long)ClrTraceEventParser.Keywords.GC),
};
var client = new DiagnosticsClient(processId);

using (var session = client.StartEventPipeSession(providers, false))
{
    Console.WriteLine();

    Task streamTask = Task.Run(() =>
    {
        var source = new EventPipeEventSource(session.EventStream);

        ClrTraceEventParser clrParser = new ClrTraceEventParser(source);
        clrParser.GCPerHeapHistory += OnGCPerHeapHistory;
        clrParser.GCStart += OnGCStart;
        clrParser.GCGlobalHeapHistory += OnGCGlobalHeapHistory;

        try
        {
            source.Process();
        }
        catch (Exception e)
        {
            ShowError($"Error encountered while processing events: {e.Message}");
        }
    });

Each event handler is responsible for extracting and translating the interesting fields of its event payload with a few color enhancements:

GCStart: collection count and reason (highlight induced collections).
GCGlobalHeapHistory: condemned generation, pause mode and memory pressure.
GCPerHeapHistory: starting -> final condemned generation and for each heap, budget, begin size, begin obj size, final size, promoted size and fragmentation.

The final step was to transform a simple console application into a .NET CLI tool that everyone will be able to install with dotnet tool install -g dotnet-gcstats and use with dotnet gcstats . I followed the documentation by adding the following to the project file:

1
2
3
4
5
6



    true
    dotnet-gcstats
    ./nupkg
    true
  

In addition, I provided a few additional details:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15



    dotnet-gcstats
    1.0.0
    </span>dotnet-gcstats<span class="nt">
    christophe Nasarre
    chrisnas
    https://github.com/chrisnas
    git
    https://github.com/chrisnas/GCStats
    LICENSE
    Global CLI tool to display live statistics during .NET garbage collections
    Initial release
    Copyright Christophe Nasarre 2024-$([System.DateTime]::UtcNow.ToString(yyyy))
    .NET TraceEvent CLR GC
  

Once built, I simply uploaded the generated package to nuget.org et voila!

Now, you should be able to better understand why some collections are triggered:

And if it is not enough, wait for reading the second edition of Pro .NET Memory Management ;^)

Be Aligned! Or how to investigate a stack corruption

Mon, 11 Dec 2023 11:01:45 +0000

Introduction

During the Datadog R&D week, my goal is to mimic the generation of a .gcdump from our .NET profiler. I’ve already written most of the code for a previous post and after changing the required plumbing, it is time to test the workflow.

Unfortunately, I’m facing the dreaded stack corruption dialog:

The rest of the post explains the different steps I’m following to investigate this issue.

Trying to understand the problem

This stack check is done by the debug version of the C Runtime library by basically adding some special bytes on the stack before calling a function and checking these bytes are not tampered when returning from the call.

So, the next step is to debug the application to get more details and at least where in the code the problem happened:

The failed check occurs at the end of a function that looks like the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


bool GcDumpProvider::Get(IGcDumpProvider::gcdump_t& gcDump)
{
    // trigger the GC and get the dump
    GcDump gcd(::GetCurrentProcessId());
    gcd.TriggerDump();

    auto const& dump = gcd.GetGcDumpState();
    auto& types = dump._types;
    for (auto& type : types)
    {
        auto& typeInfo = type.second;

        uint64_t instancesCount = typeInfo._instances.size();
        uint64_t instancesSize = 0;
        for (size_t i = 0; i < instancesCount; i++)
        {
            instancesSize += typeInfo._instances[i]._size;
        }

        gcDump.push_back({typeInfo._name, instancesCount, instancesSize});
    }
    return true;
}

There is much more code behind this; especially in the TriggerDump() function. I have already tested this code many times when I dug into the .gcdump generation process without facing this stack corruption. I’m spending a few hours digging back into the code because:

I’m not running inside the profiled process and not outside like in the blog post
I’m introducing a “slight” change because I need to exit the communication with the CLR when the GC ends.
I need to mention that Visual Studio is refusing to debug (Step Over or Step Into) and only Run to Cursor was possible due to mixed mode (managed and native) debugging. So, I created a simple native console application with my updated code for easier and faster debugging.

After a couple of hours, it is time to go back to the Get() implementation because I do not find anything obviously wrong.

Make it simpler and simpler and simpler again

In that type of situation, I recommend the “remove code and debug” strategy (from “divide and conquer” attributed to Julius Cesar). From the simplified console application, the code now looks like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


bool GetGcDump(int pid, IGcDumpProvider::gcdump_t& gcDump)
{
    GcDump gcd(pid);

    // no more gcd.TriggerDump()

    // no more for (auto& type : types)

    return true;
}

No more complicated call nor iteration on the vector of results. Guess what? Same stack corruption.

It is time to go one level deeper: what does this GcDump class look like?

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


class GcDump
{
public:
    GcDump(int pid);
    ~GcDump();

...

private:
    int _pid;
    DiagnosticsClient* _pClient;
    EventPipeSession* _pSession;
    HANDLE _hListenerThread;
    GcDumpState _gcDumpState;
};

The constructor is setting the fields value to zero/nullptr and the destructor is cleaning up these fields if necessary. Since TriggerDump() is no more called, these fields never change.

I’m commenting out all fields until only _gcDumpState remains and it continues to crash. When is it commented out, it is not more crashing.

Use your debugger Luke!

Let’s turn to the GcDumpState class that is even simpler:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


class GcDumpState
{
public:
    GcDumpState();
    ~GcDumpState();

public:
    // fields removed for brevity

private:
    bool _isStarted;
    bool _hasEnded;
    uint32_t _collectionIndex;
};

The code of the destructor only sends a trace to the console (removing it completely does not fix the issue) and here is the constructor code:

1
2
3
4
5
6


GcDumpState::GcDumpState()
{
    _isStarted = false;
    _hasEnded = false;
    _collectionIndex = 0;
}

Again, same strategy: remove one field after the other. This time, the code stops crashing if the two Boolean fields are removed or if the 32 bits index is removed. If the index field is not set, no more corruption!

How could this assignation corrupt the stack? It is time to use the Visual Studio debugger to better understand what is going on.

First, set a breakpoint on the assignment line and click Debug | Disassembly to see the corresponding assembly code:

The two lines of assembly code are easy to understand:

1
2


mov rax,qword ptr [this]  
mov dword ptr [rax+4],0

The this pointer is stored in the rax register
The 32 bits (mov dword) memory starting 4 bytes after the beginning of the object pointed to by this, is set to 0

I enter “this” in a Memory panel (Debug | Windows | Memory xx)

And Visual Studio gives me the corresponding address and the content of the memory there:

Pressing F10 twice to Step Over the two assembly instructions and this is confirmed:

Instead of storing the 32 bits 0 value just after the two bytes corresponding to the bool fields, it is stored 2 bytes away. It looks like a padding is added on my behalf.

I change the build settings for the GcDumpState.cpp file to enable all warnings:

The compilation confirms what has been seen in the memory:

1

GcDumpState.h(41,14): warning C4820: 'GcDumpState': '2' bytes padding added after data member 'GcDumpState::_hasEnded'

What’s next?

My understanding is that the compiler is:

adding a 2 bytes padding to align the 32 bits index field
generating the constructor code based on that padding
but the stack corruption checking code does not take it into account

The solution is to either add a uint16_t field after the 2 bool fields as an explicit padding or use #pragma pack(1) to decorate the class definition.

However, this looks really weird to me. We should have faced this issue a long time ago because we were never cautious about alignment in all the classes and structures that we allocate in our code. To validate the assumption, I’m writing a small reproduction code outside of all the .gcdump complexity. And guess what? I’m not able to reproduce the stack corruption. Another mystery of the C++ compilation optimizations probably…

This is the end of my debugging Friday at Datadog :^)

How to dig into the CLR

Sun, 12 Nov 2023 09:12:28 +0000

Introduction

When I started to work on the second edition of Pro .NET Memory Management : For Better Code, Performance, and Scalability by Konrad Kokosa, I already spent some time in the CLR code for a couple of pull requests related to the garbage collector. However, updating the book to cover 5 new versions of .NET requires looking at new APIs but also digging deep inside the CLR (and especially the GC) hundreds of thousand lines of code!

The first step is to install Visual Studio 2022 Preview that allows you to compile and run projects targeting .NET 8. Then, goto https://github.com/dotnet/runtime and git clone the tag of the .NET 8 preview version you have installed.

That way, you will be able to directly run the same version that you will debug.

And now, what are the next steps?

The goal of this post is to share with you the tips and tricks I used to navigate into the CLR implementation so you could better understand how things are working.

From C# to C++

As a .NET developer, I’m used to the APIs provided by the Base Class Library built on top of the CLR. Let’s take as an example the following code that is using the GC.AllocateArray method that allows you to allocate a pinned in memory array and available since .NET 5.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


using System;

internal class Program
{
    static void Main(string[] args)
    {
        byte[] pinned = GC.AllocateArray<byte>(90000, true);
        Console.WriteLine($"generation = {GC.GetGeneration(pinned)}");
    }
}

When you Ctrl+click the method name (or use F12), thanks to Source Link integration, you go to its implementation where you can even set breakpoint:

If you don’t use Visual Studio, you could open the generated assembly into a decompiler such as ILSpy or DnSpy. The latter even allows you to set breakpoints and debug the disassembly IL without any source.

In both cases, only the managed implementation will be available: you soon end up to an “internal call” corresponding to a native function implemented by the CLR. The managed methods are decorated with the MethodImplOptions.InternalCall attribute.

For the garbage collector code, you can look into the GC.CoreCLR.cs file where these methods are defined. You can note some methods decorated with the DllImport attribute to bind to native functions exported by a “QCall” library. There is an optimized path in P/Invoke done by the JIT to transform these calls not like a usual LoadLibrary/GetProcAddress as you could expect. Instead, they will be routed to the exported methods by coredll.dll and defined in the s_QCall array in qcallentrypoints.cpp. But where to look further for the native implementation?

Instead of searching among the thousands of files, focus on comutilnative.h that defines the signature of most exported functions. The implementation of the exported native functions is found in comutilnative.cpp. This is where you should start your journey in the native implementation of the CLR. For the list of all functions called by the libraries in the runtime, look at the ecalllist.h file (around gGCInterfaceFuncs and gGCSettingsFuncs specifically for the GC).

Note that you might also find some implementations under the classlibnative folder like in the system.cpp file for GCSettings.IsServerGC.

CLR Source code debugging

It is nice to know that the implementation of most CLR exported native functions used by the BCL is in comutilnative.cpp. For the GC, the functions are either statics from the GCInterface class or static functions prefixed by GCInterface_; I don’t know why all are not part of GCInterface…

When you look at the GC-related methods implementation, a lot are calling methods from the instance returned by GCHeapUtilities::GetGCHeap() that corresponds to the static g_pGCHeap global variable. It is interesting to follow the threads of calls like that, but I have to admit that, after a few hops, I’m starting to get lost. So, I’m drawing boxes for types on a piece of paper and arrays from their fields to other types as boxes.

However, with a code base that big, I definitively prefer to set breakpoints and write a small C# application to call the methods I’m interested in and see what data structures are used in the different layers of implementation. Don’t be scared: WinDBG is not required to achieve this goal. As this page explains, you need to type the following commands in a shell at the root of the repo:

.\build.cmd -s clr -c Debug .\build.cmd clr.nativeprereqs -a x64 -c debug .\build.cmd -msbuild

The last command generates a CoreCLR.sln solution file in artifacts\obj\coreclr\windows.x64.Debug\ide) that you can open in Visual Studio 2022 Preview.

In VS, right-click the INSTALL project, select Properties and setup the Debugging properties

Here are the details of each property:

It could be interesting to set some environment variables such as DOTNET_gcServer to 1 for a GC Server configuration instead of workstation. In that case, click the choice in the combo-box:

And update the textbox at the top:

The final step is to set this project as the startup project:

You are now able to set the breakpoint you want in the native code of the CLR and type F5/Debug in Visual Studio to step into the code!

And what about the assembly code?

Some specific data structures, such as the NonGC Heap, are used by the JIT compiler when generating the assembly code from the IL compiled from your C# code. It means that you need to look at that JITted code to fully understand what is going on.

A first way to get it is to use https://sharplab.io/, type your C# code and select x64 for Core of x86/x64 for Framework:

But as you can see from this screenshot, it is using the .NET 7 compiler. What if you would like to see the .NET 8 compilation result just in case something changed?

The solution I’m using is to generate a memory dump with procdump -ma of a test application. Before opening the dump in WinDBG, there is something you should be aware of: with the tiered compilation, you will need to call a method several times before the final optimized assembly code gets JITed. Or… decorate the method you are interested in with the [MethodImpl(MethodImplOptions.AggressiveOptimization)] attribute to instruct the JIT to directly generate the most optimized tier.

Once the dump loaded in WinDBG, the first step is to get the MethodTable pointer corresponding to the method you are interested in. For that, use the name2ee SOS command:

Click the link corresponding to MethodDesc to run the dumpmd SOS command:

The last step is to click the link corresponding to CodeAddr to run the U command and see the JITted assembly code:

If you compare this code to get the “Hello, World!” string, with the one shown by sharplab,

1
2
3
4


Program.Hello()
    L0000: mov rcx, 0x257f7cbc368
    L000a: mov rcx, [rcx]
    L000d: jmp qword ptr [0x7ff9c9bd7f48]

you might notice a tiny difference: there is one less indirection in .NET 8! But this is another story that will be told in the second edition of the “Pro .NET Memory Management: For Better Code, Performance, and Scalability” book ;^)

Crap: the application is randomly crashing!

Mon, 02 Oct 2023 09:02:26 +0000

Introduction

When you have a call with a customer who explains to you that his application is crashing when your profiler is enabled, it is never a great experience. This post is listing which steps were followed to investigate such an issue I faced last week; from the basics up to the final in analysing memory dumps in WinDbg.

Get as many setup details as possible

The situation was the following:

A web application was running fine with our Datadog .NET profiler on some non-production servers with less traffic.
The same application was crashing on production servers with more traffic.

We were lucky to be able to remote access both machines. A lot of time was spent to check the setup that is based on environment variables. Basically, for our profiler to be loaded by a .NET application, a few Microsoft related environment variables need to be set. Then, you enable the Datadog profiler by setting DD_PROFILING_ENABLED to 1 in order to get the profiling details available in our UI. Since the web application is running in IIS, things get more complicated because some environments variables must be set in… the Registry.

So, we checked the environment variables set at the machine level with the set command in a prompt and those for IIS with the Registry Editor. However, we got some inconsistencies, and we needed a way to validate what were the environment variables really seen by the web application! The Process Explorer tool from Sysinternals was downloaded and launched. After finding the process ID of the running w3wp.exe corresponding to the web application, a simple right-click to get the Properties and selecting the Environment Tab gave us the truth:

(This screenshot shows the results for one of our test applications on my development machine).

Getting a memory dump

Once the setup was checked on both machines without any too weird issues, the next step was to figure out why the application was randomly crashing. Even if the machines received different traffic loads, since applications running without our profiler enabled were not crashing, the chances were high that our C++ code was at the source of the problem. But the crashes were random… And you can’t install Visual Studio on a production server and attach to the process hoping that it will crash and start a debugging session there!

Windows Error Reporting is generating mini dumps when applications are crashing but they are usually not enough to start an investigation. Again, the other Sysinternals tools procdump was installed as a global crash handler with procdump -i c:\dumps -ma. The next time the application crashed, a memory dump was be generated in the c:\dumps folder. Don’t forget to create it manually if it does not exist.

From addresses to source code

To play with a memory dump, WinDbg is my preferred toy. I opened the memory dump and, in the case of a crash, the stack panel automatically displayed the call stack of the faulted thread:

The last frame triggering the issue (i.e., before KiUserExceptionDispatch) is Datadog_Profiler_Native!DllCanUnloadNow+0x2954b. Knowing that WinDbg transforms . in file names into _ leads to Datadog.Profiler.Native.dll which is the file where our profiler is implemented. However, WinDbg was not able to find the name of the function and only looked at the exported public symbols. With the lm command, you can see how WinDbg gets the symbols for this dll:

With DllCanUnloadNow, you could tell that we are dealing with some COM stuff but it did not really help me for the investigation: I needed to know which function was running which part of its code. Hopefully, for each release of the .NET profiler in Github, in addition to the .msi installer, the symbols and the source code are also provided.

Both files were unzipped in the folder where the dumps were copied. Then, I changed the Debugging Settings in WinDbg to point to these folders:

Let’s start with the symbols to let WinDbg match an instruction pointer to a function name. I asked WinDbg to provide details about the symbol resolution with !sym noisy. Then I forced the symbols for my module to gets reloaded with .reload /f “Datadog.Profiler.Native.dll”. In the flow of errors, I find out where the .pdb file should be stored so that WinDbg would find it:

So the problem is triggered somewhere in our Windows64BitStackFramesCollector::CollectStackSampleImplementation function. By simply double-clicking this frame, WinDbg automagically found the corresponding source file and pinpointed the culprit line:

A bit of WinDbg magic

To follow me a bit further, you need to understand what this code is doing: it is walking the stack of a thread to find the instruction pointers of each called function. This line 260 is dereferencing the address contained in context.Rsp. I looked at Locals panel to get its value:

The !address command gave me in which module this code was executed from:

It looked like a valid page with executable code…

I wanted to see why our stack walking code would break here. What if I asked WinDbg to show me this stack? To do that, I first needed to know which thread our code was trying to stack walk. I knew that Windows64BitStackFramesCollector was keeping track of the currently walked thread in a ManagedThreadInfo instance pointed to by its _pCurrentCollectionThreadInfo field:

This instance stores the thread ID in its _osThreadId field: now let’s ask WinDbg to switch to this thread.

The ~ command lists all threads:

A quick CTRL+F with “27600” stopped on the thread #72. Threads have a lot of identifiers in WinDbg and the first one allowed me to switch with ~72s.

The Stack panel was almost empty:

To be sure, I used the kp command… that told me that WinDbg was not really happy neither:

I was kind of stuck but my colleague Kevin Gosse mentioned that I could use r rip to see what would be the next instruction to be executed by this thread:

Then, the ln command (close to the !address command I used just before) allowed me to click the Browse Module link and see that, again, some code from Sentinel One was ready to execute.

This agent is part of an anti-virus (and more) solution that seems to highjack the stack of threads and our code was not dealing properly with this kind of situation. The fix was to protect our dereferencing code against access violation and stop walking the stack in that case.

Another debugging day at Datadog :^)

Raiders of the lost root: looking for memory leaks in .NET

Mon, 08 May 2023 09:05:15 +0000

Introduction

It’s been almost 12 years since I wrote LeakShell to help me automate the search of memory leaks in .NET. The idea was simple: compare 2 memory dumps of a leaking .NET application to show the types with increasing instances count.

Today, you could use Visual Studio Memory Usage tool to do the same but with a much better user interface! The additional killer feature is the ability to see the references chain that explains why a “leaky” object stays in memory.

My previous series about building your own .NET memory profiler in C# is based on CLR events and does not allow to get the references chain. This post explains how you could write your own memory profiler based on.NET profiler APIs in C++. Refer to this post for an introduction of how to implement ICorProfilerCallback to be loaded by the CLR in a .NET process.

How to detect memory leaks

From a high level view, detecting a memory leak means being able to know which objects stay alive garbage collection after garbage collection:

implement ICorProfilerCallback::ObjectAllocated to keep track of ALL objects in the heap,
use ICorProfilerCallback::MovedReferences2 to fixup the addresses when the live objects are moved during compaction garbage collections,
since** **ICorProfilerCallback::ObjectReferences is called for each surviving object, clean your list of live objects.

The first drawback of this solution is that the CLR has to disable concurrent GC to call these functions with probable impact on performances. However, if you can’t find a leak in production that leads to out of memory crashes, running one instance in this mode is perfectly acceptable. The second drawback is the complexity of keeping track of objects through compacting GCs.

This is why I implemented in .NET 7 a new set of functions in ICorProfilerInfo13 to mimic what you can do in C# with a weak reference:

CreateHandle : create a weak handle to wrap an object,
GetObjectIDFromHandle: get the address of the wrapped object or null if the object is no more in the heap,
DestroyHandle: clean up the weak handle.

Creating such a weak handle for allocated objects get rid of the address fixup complexity. However, you should not create a handle for ALL allocated objects because it will slow down the garbage collections. So the next step is to listen to the AllocationTick CLR event and create a weak handle for each sampled allocation. Even though the statistical distribution of such 100 KB threshold-based sampling is not perfect, leaking objects should appear.

After each garbage collection detected in ICorProfilerCallback2::GarbageCollectionFinished or via specific GC events, you could clean up this list of allocated objects by removing those for which GetObjectIDFromHandle returns null.

Feel free to look at the corresponding implementation in Datadog .NET profiler code.

Rebuild references chain up to a root

Even though it is possible to get the call stack that led to allocating a leaking object thanks to the AllocationTick event, it would be better to know why it stays in memory. So the next step is to rebuild the references chain up to the root.

As explained in a previous post, it is possible, for a given object, to get the list of its fields and build a graph of dependencies from a parent to its children. However, you are interested in the opposite and it would require to get these parent/children references for ALL objects in the heap. And this is not possible with the sampled AllocationTick event…

This is where ICorProfilerCallback::ObjectReferences shines:

1
2
3
4
5
6


HRESULT ObjectReferences(
   ObjectID objectId,
   ClassID  classId,
   ULONG    cObjectRefs,
   ObjectID objectRefIds[] 
);

This method is called during a garbage collection for all objects (i.e. objectId first parameter) still alive and lists its fields referencing objects in the heap (i.e. objectRefIds last parameter).

You could store each object as an ObjectNode:

1
2
3
4
5
6
7
8
9


struct ObjectNode
{
public:
    ObjectNode(ObjectID objectId);

public:
    ObjectID instance;
    std::vector<ObjectNode*> rootRefs;
};

in a vector that represents the heap.

For each fields, look for its corresponding node in the vector and add the node of its parent (given by objectId) to its rootRefs vector of parents. That way, you are building a back reference graph:

The small blue arrows show the parent/children reference given by ObjectReferences and the large purple ones are kept to build a reverse references graph you are interested in.

You know when all live objects in the heap have been listed when ICorProfilerCallback2::GarbageCollectionFinished is called. It is now time to get the build the references chain for all sampled objects still alive (thanks to GetObjectIDFromHandle returning non null address).

It is important to understand that it is a graph and not a tree because cycles exist in .NET.

This is common in situations where objects need to keep a reference to their “parents”. It means that these cycles should be detected when looking for the list of references of a given live object to avoid infinite recursion:

bool DumpNode(ObjectNode* node, std::vector& referenceStack)

The traversing DumpNode method takes a node (i.e. an object of the heap) and a stack where the parents will be added as we dig into the graph.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59


bool DumpNode(ObjectNode* node, std::vector<ObjectID>& referenceStack)
{
    // end of recursion: the node is a root
    if (node->rootRefs.size() == 0)
    {
        //  dump the root
        std::cout << std::endl;
        std::cout << std::hex << node->instance << std::dec;
        COR_PRF_GC_ROOT_KIND kind;
        COR_PRF_GC_ROOT_FLAGS flags;
        if (FindRoot(_roots, node->instance, kind, flags))
        {
            std::cout << " | ";
            DumpKind(kind);
            std::cout << " - ";
            DumpFlags(flags);
        }
        else
        {
            std::cout << " | ?";
        }
        std::cout << " = ";

        DumpObjectType(node->instance, _pCorProfilerInfo, _pFrameStore);
        std::cout << std::endl;

        // dump the references from the root
        for (int16_t i = referenceStack.size()-1; i >= 0; i--)
        {
            ObjectID reference = referenceStack[i];
            std::cout << " --> ";
            std::cout << std::hex << reference << std::dec;
            std::cout << " = ";
            DumpObjectType(reference, _pCorProfilerInfo, _pFrameStore);
            std::cout << std::endl;
        }

        return true;
    }

    // detect cycles
    if (Find(referenceStack, node->instance))
    {
        return false;
    }

    // go up into the reference chain
    referenceStack.push_back(node->instance);
    for (auto& parentNode : node->rootRefs)
    {
        if (DumpNode(parentNode, referenceStack))
        {
            return true;
        }
    }
    referenceStack.pop_back();

    return false;
}

If a parent node is already in the stack, a cycle is detected and that path is not used. Once a root is reached, the stack is dumped as shown in the following output:

OnGarbageCollectionFinished: 3859 objects in the heap.

OnRootReferences2: 90/109 roots.
            stack:40
        finalizer:1
           handle:49
            other:0
------------------

21266c00020 | H - 0 = Object[]
 --> 21268c092c8 = NativeRuntimeEventSource
 --> 21268c3fbe8 = EventSource.EventMetadata[]
 --> 21268c18010 = ParameterInfo[]
 --> 21268c17ef0 = RuntimeParameterInfo
 --> 21268c0ed58 = RuntimeMethodInfo
 --> 21268c0e708 = RuntimeType.RuntimeTypeCache
 --> 21268c0e840 = RuntimeType.RuntimeTypeCache.MemberInfoCache
 --> 21268c14978 = RuntimeMethodInfo[]
 --> 21268c10d70 = RuntimeMethodInfo
 --> 21268c29720 = Signature
 --> 21268c29770 = RuntimeType[]
=====================================

As shown in the output, it is possible to provide details about the kind of root is keeping the references chain alive thanks to ICorProfilerCallback::RootReferences2:

1
2
3
4
5
6
7


HRESULT RootReferences2(  
   ULONG  cRootRefs,  
   ObjectID rootRefIds[],  
   COR_PRF_GC_ROOT_KIND rootKinds[],  
   COR_PRF_GC_ROOT_FLAGS rootFlags[],  
   UINT_PTR rootIds[]
);

This function is called with three synchronized arrays cRootRefs long that contain for each root:

the address (**rootRefsIds **objectID),
the kind (rootKind for stack, finalizer, handle and other)
and flags (rootFlags for pinned, weak reference interior or ref counted).

These are stored in a vector of ObjectRoot:

Goodies: how to get arrays type name

I did not mention how to get the type name of either an ObjectID or a ClassID because it is explained in a previous post. However, I forgot to explain how to deal with the different kinds of arrays: single dimension (ex: byte[]), multidimensional (ex: byte[,]) or jagged (ex: byte[][]).

When you call ICorProfilerInfo::GetClassInfo on a ClassID corresponding to an array,

1
2
3


ModuleID moduleId;
mdTypeDef typeDefToken;
hr = _pCorProfilerInfo->GetClassIDInfo(classId, &moduleId, &typeDefToken);

it won’t fail but the module id and the metadata token will both be set to 0.

Instead, you have to call ICorProfilerInfo::IsArrayClass to get the rank and the item class ID of the array. This is then done recursively on the item class ID until it fails:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


std::string arrayBuilder;

CorElementType baseElementType;
ClassID itemClassId;
ULONG rank = 0;
if (_pCorProfilerInfo->IsArrayClass(classId, &baseElementType, &itemClassId, &rank) == S_OK)
{
    classId = itemClassId;
    isArray = true;
    AppendArrayRank(arrayBuilder, rank);

    // in case of matrices, it is needed to look for the last "good" item class ID
    // because all others might be array of array of ...
    for (size_t i = 0; i < rank; i++)
    {
        HRESULT hr = _pCorProfilerInfo->IsArrayClass(classId, &baseElementType, &itemClassId, &rank);
        if ((hr == S_FALSE) || FAILED(hr))
        {
            itemClassId = classId;

            break;
        }

        AppendArrayRank(arrayBuilder, rank);
        classId = itemClassId;
    }
}

Notice that the way to concatenate the possible [] / [,] / [][] could is the opposite of how the array type is defined in C#:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


void AppendArrayRank(std::string& arrayBuilder, ULONG rank)
{
    if (rank == 1)
    {
        arrayBuilder = "[]" + arrayBuilder;
    }
    else
    {
        std::stringstream builder;
        builder << "[";
        for (size_t i = 0; i < rank - 1; i++)
        {
            builder << ",";
        }
        builder << "]";

        arrayBuilder = builder.str() + arrayBuilder;
    }
}

For example, a byte[][,] is defined as an rank 2 array of array of byte.

References

Automate the search of memory leaks with LeakShell
Building your own .NET memory profiler in C#
Introduction to .NET Profiling with ICorProfilerCallback
Pull request in .NET 7 for ICorProfilerInfo13 to create weak handles
Datadog .NET Live Heap Profiler implementation

Reading “object” in memory — starting with stacks

Sun, 15 Jan 2023 16:38:45 +0000

The previous episodes started the parsing of the “nettrace” format used when contacting the .NET Diagnostics IPC server and initiate the protocol to receive CLR events. It is now time to see how to get the payload of each “object” type, especially how stacks are stored.

We have seen that the stream starts with a TraceObject that describes the rest of the stream followed by a sequence of “object”:

The remaining of each “object” is a 32 bit block size followed by the payload.

Well… not only. One thing I missed when I started to work on the nettrace format is the fact that all “object” payloads must be 4-bytes aligned on the beginning of the stream!

This is why I’m keeping track of the current position in the EventPipeSession class:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


private:
    IIpcEndpoint* _pEndpoint;
    bool _stopRequested;

    // parsers
    MetadataParser _metadataParser;
    EventParser _eventParser;
    StackParser _stackParser;
    SequencePointParser _sequencePointParser;

    // Keep track of the position since the beginning of the "file"
    // i.e. starting at 0 from the first character of the NettraceHeader
    //      Nettrace
    uint64_t _position;
    ...
};

So each ParseXXXBlock function checks the minimum reader version in the header before reading the “object” payload as a memory block. The idea is being able to support backward compatibility:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


bool EventPipeSession::ParseMetadataBlock(ObjectHeader& header)
{
    if (header.MinReaderVersion != 2) return false;

    uint32_t blockSize = 0;

    // read the block and send it to the corresponding parser
    uint64_t blockOriginInFile = 0;
    if (!ExtractBlock("Metadata", blockSize, blockOriginInFile))
        return false;

    return _metadataParser.Parse(_pBlock, blockSize, blockOriginInFile);
}

The ExtractBlock function reads the size of the payload (and skips the padding if any) with ReadBlockSize:

1
2
3
4
5
6
7
8
9


bool EventPipeSession::ExtractBlock(const char* blockName, uint32_t& blockSize, uint64_t& blockOriginInFile)
{
    // get the block size
    if (!ReadBlockSize(blockName, blockSize))
        return false;

    // skip the block + final EndOfObject tag
    blockSize++;
    ...

The block name is only used for error messages if needed.

The next step is to read the payload in a memory block using these two EventPipeSession fields:

1
2
3
4
5


...
    // buffer used to read each block that will be then parsed
    uint8_t* _pBlock;
    uint32_t _blockSize;
    ...

In the session constructor, _blockSize is set to 4 KB and _pBlock points to an allocated memory buffer of that size.

The rest of ExtractBlock deals with payload size: if the current payload to parse is larger than _blockSize, then these fields are updated up to a maximum of 100 KB (i.e. max block size sent by the CLR).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


// check if it is needed to resize the block buffer
    if (_blockSize < blockSize)
    {
        // don't expect blocks larger than 100KB
        if (blockSize > MAX_BLOCK_SIZE)
            return false;

        delete [] _pBlock;
        _pBlock = new uint8_t[blockSize];
        ::ZeroMemory(_pBlock, blockSize);
        _blockSize = blockSize;
    }

    // keep track of the current position in file for padding
    blockOriginInFile = _position;
    if (!Read(_pBlock, blockSize))
    {
        Error = ::GetLastError();
        std::cout << "Error while extracting " << blockName << " block: 0x" << std::hex << Error << std::dec << "\n";
        return false;
    }
    std::cout << "\n" << blockName << " block (" << blockSize << " bytes)\n";
    DumpBuffer(_pBlock, blockSize);

    return true;
}

For debugging sake, I’m displaying each “object” payload

thanks to the DumpBuffer helper.

To ease the memory access to the memory block content, my BlockParser will be used as a base class for each dedicated parsers:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


class BlockParser
{
public:
    BlockParser();
    bool Parse(uint8_t* pBlock, uint32_t bytesCount, uint64_t blockOriginInFile);
    void SetPointerSize(uint8_t pointerSize);

public:
    uint8_t PointerSize;

protected:
    virtual bool OnParse() = 0;

    // Access helpers
    bool Read(LPVOID buffer, DWORD bufferSize);
    bool ReadByte(uint8_t& byte);
    bool ReadWord(uint16_t& word);
    bool ReadDWord(uint32_t& dword);
    bool ReadLong(uint64_t& ulong);
    bool ReadDouble(double& d);
    bool ReadVarUInt32(uint32_t& val, DWORD& size);
    bool ReadVarUInt64(uint64_t& val, DWORD& size);
    bool ReadWString(std::wstring& wstring, DWORD& bytesRead);
    bool SkipBytes(uint32_t byteCount);

// shared fields
protected:
    bool _is64Bit;
    uint32_t _blockSize;
    uint32_t _pos;

private:
    uint8_t* _pBlock;
    uint64_t _blockOriginInFile;
};

The Parse function accepts the memory buffer containing an “object” payload, its size and its position since the beginning of the stream. The derived class will have to implement the OnParse function using the ReadXXX helpers.

The two ReadVarUintXXX functions are different from the other direct read helpers because they deal with some simple compression mechanisms used by the serialization of 32-bit and 64-bit numbers.

In the different types of “object” payloads, the strings are serialized as UTF16 strings ending with a “\0” wide character. Here is the implementation of the helper function used to read a std::wstring from a memory block:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


bool BlockParser::ReadWString(std::wstring& wstring, DWORD& bytesRead)
{
    uint16_t character;
    bytesRead = 0;  // in case of empty string
    while (true)
    {
        if (!ReadWord(character))
        {
            return false;
        }

        // protect against invalid UNICODE character (due to missing fields in ExceptionThrown event)
        if (character > 256)
        {
            // rewind the character
            _pos = _pos - sizeof(character);

            // this is only covering a missing string
            return (bytesRead == 0);
        }

        bytesRead += sizeof(character);

        // Note that an empty string contains only that \0 character
        if (character == 0) // \0 final character of the string
            return true;

        wstring.push_back(character);
    }
}

Note the check for character content in the loop: this is due to a serialization issue I will discuss later when the event “object” block will be detailed.

The Stack “object”

If you remember my previous post about retrieving call stacks for CLR events with TraceEvent, you might be wondering why there is a specific stack object since a ClrStackWalk event should contain the frames if the Stack keyword is enabled for the .NET provider. In fact, the current TraceEvent implementation is not using the stack object sent by the CLR (maybe to have the same code between ETW and EventPipe).

One stack “object” received in a nettrace stream contains one or more stacks. Each stack is identified by an id (more about this soon) and contains a list of instruction pointer addresses.

In the previous screenshot, the id of the first stack is 1 and the second is 2. In the next stack “object”, the FirstId field will be 3 and so on. This avoids storing the id in each call stack and saves space.

Note that even if this does not seem to make any sense, it might happen that the addresses list is empty.

These call stacks are stored in EventPipeSession as a per id cache:

1
2
3
4


// per stackID stack
    // only one will be used depending on the bitness of the monitored application
    std::unordered_map<uint32_t, EventCacheStack32> _stacks32;
    std::unordered_map<uint32_t, EventCacheStack64> _stacks64;

The frames are stored as addresses in a vector:

1
2
3
4
5
6


class EventCacheStack32
{
public:
    uint32_t Id;
    std::vector<uint32_t> Frames;
};

The CLR is sending one stack per unique callstack (i.e. at least one frame is different). As you will soon see, each event “object” contains a stack id corresponding to the chain of code from which it is sent.

The next episode will detail the Metadata and Event blocks to end the series.

Resources

Episode 1 — Digging into the CLR Diagnostics IPC Protocol in C#
Episode 2 — .NET Diagnostic IPC protocol: the C++ way
[Episode 3 ](/posts/2022-10-23_clr-events-go-for/ CLR events: go for the nettrace file format!
[Episode 4 ](/posts/2022-11-27_parsing-the-nettrace-stream/ Parsing the “nettrace” steam
Source code for the C++ implementation of CLR events listener
Diagnostics IPC protocol documentation

Troubleshooting CPU and exceptions issues with Datadog toolbox

Thu, 09 Jun 2022 16:09:27 +0000

Introduction

With the new 2.10 release of the Datadog .NET Tracer and Continuous Profiler available, it is time to update some investigation workflows I already introduced. New features have been added to help you diagnose performance issues in your .NET applications:

Linux support!
Code Hotspots: allow you to automatically navigate from lengthy spans and requests to profiles
CPU profiling: pinpoint high CPU consuming methods
Exceptions profiling: identify exceptions distributions
Profile sequence: easily profile an application startup

The goal of this post is to show you how all these features make your investigations easier. I would recommend reading the previous post; especially for the environment setup that I won’t repeat here.

It’s Linux showtime!

The .NET Continuous Profiler is now available for Linux. The only limitation is the presence of glibc 2.18+ in the distribution; for example, CentOS 7 is not supported. Beyond that, we provide features parity between Linux and Windows.

In terms of installation, download the .NET Tracer package that supports your operating system and architecture. Go to the documentation for the additional configuration steps.

From spans to profiles

When analysing lengthy requests, you usually start from looking at the corresponding spans in the APM Traces part of the UI. It is now possible to view the corresponding profiles by clicking the “View Profile” button in the “Code Hotspots” tab:

Before digging into the profiling information, you are already able to see that more than half of the time is spent in Buffer._Memmove that is called by Buggybits ProductsController.Index method:

From the profile view, it is also possible to come back to the traces:

Let’s see now what new features are available at the profiling side.

CPU profiling

The most demanded feature was the ability to analyse CPU consumption (a.k.a. CPU profiling). The idea is to be able to identify code that really consumes CPU usage and optimize it. This is particularly important in the context of cloud-based computing where what you pay is related to the consumed CPU.

In term of implementation, unlike Wall Time profiling, we look at the time spent by a thread on a CPU core and not the elapsed time since the last time we checked (every ~10ms). We also collect the call stack of a thread only if it is currently running on a core. Why? Because we want to only record call stacks corresponding to code paths that are consuming CPU. For example, ThreadPool threads are usually waiting (not interesting call stack) for a work item to process (interesting call stack).

In the last blog post, the code responsible for lengthy requests is doing too many string concatenations (look for += in the following code):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


public IActionResult Index()
{
    var sw = new Stopwatch();
    sw.Start();
    var products = dataLayer.GetAllProducts();
    var productsTable = "";    foreach (var product in products)
    {
        productsTable += $"";    }

    productsTable += "Product NameDescriptionPrice
{product.ProductName}{product.Description}{product.Price}
";
    sw.Stop();

    ViewData["ElapsedTimeInMs"] = sw.ElapsedMilliseconds;
    ViewData["ProductsTable"] = productsTable;
    return View();
}

The Wall Time view was very explicit about ProductsController.Index() calling String.Concat() culprit:

A simple solution is to use a StringBuilder to optimize the concatenations:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


public IActionResult Builder()
{
    var sw = new Stopwatch();
    sw.Start();
    var products = dataLayer.GetAllProducts();
    var productsTable = new StringBuilder(1000 * 80);
    productsTable.Append("");    foreach (var product in products)
    {
        productsTable.Append($"");    }

    productsTable.Append("Product NameDescriptionPrice
{product.ProductName}{product.Description}{product.Price}
");
    sw.Stop();

    ViewData["ElapsedTimeInMs"] = sw.ElapsedMilliseconds;
    ViewData["ProductsTable"] = productsTable;
    return View("Index");
}

The Wall Time view of the profile with the fix does not provide anything useful…

If you want to go deeper, you need to look at the CPU consumption:

The ProductController.Builder method is handling the request (like Index() in the String.Concat case) and calls DataLayer.GetAllProducts() where most of the CPU-related work is done.

Would it be interesting to continue optimizing the code? Notice that GetAllProducts() is “only” consuming 94ms and the other Number.* and String.Concat method around 350ms. So a gain might be neglectable compared to the total ~3 seconds CPU usage

Remember that you should not optimize for the sake of “optimizing”: you should have metrics that tell you when to start (too lengthy request processing) and when to stop.

Exceptions profiling

In the .NET world, exceptions are at the center of errors handling. It is now possible to get a sampled view of the exceptions that happened during an application lifetime; by type:

and by message:

At the implementation level, the new exceptions profiler is notified by the CLR when an exception is thrown. Since in special cases (such as network issue or invalid parsed data for example), an application could trigger thousands of exceptions in a very short period, it is needed to sample them. Otherwise, the impact on performances would be severe; especially if the call stack needs to be rebuilt for each exception.

First, at least one exception per type is kept, ensuring that weird specific exceptions are not lost in the flow. Second, exceptions are sampled over time based on a fixed number of exceptions per profile and the rate of appearance. For knowing the exact number of exceptions, feel free to leverage the Runtime Metrics package as explained in the previous blog post.

Profiling the application bootstrap

In some situations, you are interested in analysing an application bootstrap. In Datadog APM Profile Search UI, it means finding the first profile of the given service execution. However, even with the date and time column, it is not obvious to find the right one:

To help you find the initial profile of a service execution, a new “profile_seq” tag has been added to the HTTP request used to upload the profiles. It contains the count of generated profiles for a given execution of a service, starting from 0.

So now, in the Options of the profile list, add a “profile_seq” column:

The first profile is then easily spottable with a 0 value:

In the future, a more visual hint might be added to identify it without the need to add the column.

Major implementation refactoring

Finally, our implementation has benefited from a large code refactoring. As the previous post explained, the generation of .pprof files and their upload was done in C#. This has been replaced by using a rust library shared amongst different profiler libraries (native, Ruby, .NET).

It means that you should not anymore see these frames in the application call stacks:

It does not mean that one third of the processing has been removed! Just that no more C# code is running with performance gain. First, the managed implementation was allocating objects managed by the garbage collector; adding pressure that might trigger more collections. Second, with the native rust implementation, there is no need to duplicate data between the collecting native part of the continuous profiler and the managed code used to serialize it.

In addition, several optimizations have been done in the symbol’s resolution (i.e., type and method names) part of the code that also reduce memory consumption and CPU usage.

Happy profiling!

References

Datadog Tracer & Continuous Profiler .msi Installer and Linux tar.gz
Datadog Continuous Profiler documentation
Datadog Tracer documentation
Datadog Runtime metrics documentation
Tess Ferrandez repository for BuggyBits labs

Troubleshooting .NET performance issues with Datadog toolbox

Fri, 28 Jan 2022 12:55:40 +0000

Introduction

The beta of Datadog .NET Continuous Profiler is available!

This is a great opportunity to show how to use the different tools provided by Datadog to troubleshoot .NET applications facing performance issues. Tess Ferrandez updated her famous BuggyBits application to .NET Core. Among the different available scenarios, let’s see how to investigate the Lab 4 — High CPU Hang with Datadog. It will be completely different from Tess way: no need to analyze memory dump anymore.

Setup the environment

First, you need to download and run the .msi from our Tracer repository: it will install both the Tracer and the Profiler. The former allows you, among other things, to see how long it takes to process ASP.NET Core requests. The latter is in Beta today and provides wall time duration of your threads (more on this later). Look at the corresponding documentations for the details of enabling tracing and profiling once installed.

Next, ensure that .NET Runtime Metrics are installed for your organization:

Add DD_RUNTIME_METRICS_ENABLED=true environment variable for the application/service you want to monitor. Once enabled for your application, this package allows you to see the evolution of important metrics including some that you won’t find anywhere else such as GC pause time, thread contention time or count of exceptions per type.

Ensure that DogstatsD is setup for the Agent

Looking at the symptoms

In my example, the buggybits application is running under the datadog.demos.buggybits service name. This is how I can filter the related traces in the APM/Traces part of the Datadog portal:

In this screenshot, the Products/Index requests duration is around 6 seconds; which is way too long!

When clicking such a request, the details panel provides the exact URL in the Tags tab:

The Metrics tab shows CPU usage and other few metrics around the trace time:

So, in addition to having slow requests, the CPU usage seems to increase.

It is now time to go to the .NET runtime metrics dashboard and look at what is going on in more details. The first graph that shows up is the number of gen2 collections:

It means that during the 10 minutes test where very few requests are processed, almost 800 gen2 GC are happening every 10s (all runtime metrics are computed every 10 seconds).

The load test corresponding to these requests lasted 10 minutes between 4pm and 6+pm. Each time the requests were processed:

the CPU usage increased
the number of gen2 collections increased
the duration of pauses due to garbage collections increased
the threads contention time increased

In addition to being slow, it seems that the code that processes products/index HTTP requests has also an impact on the CPU (i.e. on the overall application and machine performances).

It would be great if we could see the callstacks corresponding to this processing. This is exactly what the .NET Wall time Continuous Profiler is all about: looking at the duration of methods through a flamegraph representation.

Here comes the profiler

Today, there is no direct way to jump from a trace to the profile containing the callstacks while the corresponding request was processed. We are currently working on this new feature called Code Hotspots.

However, it is easy to use the service name to filter the profiles and select the period of time from APM/Profile Search:

When you click a one-minute profile (the callstacks are gathered and sent every minute), a panel appears with the Performance tab selected. It shows a framegraph on the left and a list on the right.

Getting used to flamegraph

When you look at the wall time flamegraph, you see everything that happened during a single minute:

The previous screenshot highlights the groups of callstacks corresponding to the different threads of execution (from left to right):

the Main() entry point of the application
the code in the CLR responsible for sending counters
the code in the Profiler in charge of generating and sending (the very thin spike) the profile every minute
the Tracer code
the code that listens to the CLR events to generate the runtime metrics
…and the application code that processes the requests!

In the flamegraph, the width of each frame on a row represents the relative time during which the frame was found on a callstack. For example, in our tests, we have 4 threads simply calling Thread.Sleep; one for 10 seconds, one for 20 seconds, one for 30 seconds and a last one for 40 seconds. This is the expected result in a flamegraph (i.e. the widths are consistent with the 1/2/3/4 ratio):

This also applies to CPU-bound threads. For example, if 3 threads are computing the sum of numbers in a tight loop, this is the expected result (i.e. all OnCPUxxx have the same width)

These explanations should stop the fear that started to crawl inside your head about the “visible cost” of the Datadog Tracer and Profiler based on the previous screenshot. The large width of the Datadog threads frames is all about wall time, not CPU time: we are mostly sleeping or waiting but we don’t stop :^)

Investigate the performance issue

The next step is to focus on the stack frames corresponding to the request processing to better understand what is going on.

Basically, you would like to either remove a branch or keep only a branch. You simply have to move the mouse over a frame (i.e. ThreadPoolWorkQueue in the previous screenshot) and click the three dots that just appeared. Next, select Show From to keep only that branch in the flamegraph:

Now, scroll-down into the flamegraph and the flow of execution corresponding to processing the Products/Index request becomes more visible:

It seems that the Index() method of the ProductsController is spending most of its time calling String.Concat().

Let’s have a look at the source code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


// GET: Products
public IActionResult Index()
{
    var products = dataLayer.GetAllProducts();
    var productsTable = "";    foreach (var product in products)
    {
        productsTable += $"";    }
    productsTable += "Product NameDescriptionPrice
{product.ProductName}{product.Description}{product.Price}
";

    ViewData["ProductsTable"] = productsTable;
    return View();
}

But still no sign of String.Concat()… Well, this is because the C# compiler is hiding it from you with the += syntaxic sugar. Let’s have a look at the decompiled code as shown by IlSpy (without the string.Concat transformation):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


public Microsoft.AspNetCore.Mvc.IActionResult Index()
{
    var products = dataLayer.GetAllProducts();
    string productsTable = "";    var enumerator = products.GetEnumerator();
    try
    {
        while (enumerator.MoveNext())
        {
            BuggyBits.Models.Product product = enumerator.Current;
            string[] array = new string[8];
            array[0] = productsTable;
            array[1] = "";            productsTable = string.Concat(array);
        }
    }
    finally
    {
        ((System.IDisposable)enumerator).Dispose();
    }
    productsTable = string.Concat(productsTable, "Product NameDescriptionPrice
";
            array[2] = product.ProductName;
            array[3] = "";
            array[4] = product.Description;
            array[5] = "";
            array[6] = product.Price;
            array[7] = "
");
    base.ViewData["ProductsTable"] = productsTable;
    return View();
}

So now we can see the call to string.Concat() at the end of the while loop iteration.

Behind the scene, since a string object is immutable, string.Concat() will create a new string each time it is called and the previous string referenced by productsTable is no more rooted and will put more pressure on the GC. If I’m telling you that datalayer.GetAllProducts() returns 10.000 products, it means that string.Concat gets called 10.000 times.

As the string grows, it will reach the 85000 bytes limit and start to be allocated in the LOH, adding more pressure on GC that will trigger gen2 collections; hence the high number of gen2 collections seen in the runtime metrics dashboard.

Note that if the native frames were visible in the flamegraph (by the way, let me know if this is a feature that would make sense to add), you would see the methods of the CLR responsible for the GC.

Look at Tess Ferrandez post for a possible solution to this expensive code pattern (i.e. calling string.Contact in a large tight loop)

Different types of filters

Before leaving, I would like to quicky talk about the list shown on the right hand-side of the UI.

It allows you to see a wall time summary per method name, namespace (i.e. sum of methods from types in the same namespace, thread ID (i.e. internal Datadog unique ID), thread name or AppDomain Name.

First, the only methods listed here are “leaf” methods: they appear at the top of at least one callstack. If you would like to visually see some specific frames, you should use the filter box:

All other frames are faded out.

Second, the list is sorted with the largest wall time at the top: this could be an easy way to spot method “expensive” in term of CPU (i.e. will frequently be running so appear at the top of the stack). You simply need to skip the wait and sleep related methods like shown in the previous screenshot: String.Concat and Buffer._Memmove (used by string.Concat) were just in front of your eyes!

When you select an element of the list, the flamegraph is updated accordingly: only the callstacks containing this element will be visible (it could speed up the filtering process)

References

Datadog Tracer & Continuous Profiler .msi Installer
Datadog Continuous Profiler documentation
Datadog Tracer documentation
Datadog Runtime metrics documentation
Tess Ferrandez repository for BuggyBits labs

Start a journey into the .NET Profiling APIs

Sat, 07 Aug 2021 12:21:39 +0000

Introduction

When I want to dig into a new API, I implement a real world scenario. This is exactly what I did for the .NET native profiling API. I want to know how to get parameters and return value of any method call during any .NET application life. The expected result would be something like :

Enter PublicClass::ClassParamReturnClass
this = 0x6f97e190 (8)
ClassType obj = 0x6f97e488 (8)
| int32 k__BackingField = 84
| int32 intField = 42
| String stringField = 43
ClassType obj = 0x0000023A475BBAD8

Leave PublicClass::ClassParamReturnClass
| int32 k__BackingField = 170
| int32 intField = 85
| String stringField = 86
returns 0x0000023A475BBBB0

when the following method is executed:

1
2
3
4


public ClassType ClassParamReturnClass(ClassType obj)
{
   return new ClassType(obj.IntProperty + 1);
}

You will have to write native C/C++ code to leverage the .NET Profiling API as John Robbins explained in his 2003 Debugging Applications book. Even though Microsoft is providing a code sample for that, it does just show how to get notified when a function is called or exited but nothing about its type, its name, what are its parameters value and its return value. This series will detail both the .NET profiling API and the metadata API (i.e. the native reflection API).

Let’s start with the basics of .NET Profiling. As shown in the following figure from the Microsoft Profiling documentation, with the right environment configuration, the CLR will load a COM-like object implementing ICorProfilerCallback interface to notify almost everything happening in a .NET application from startup to shutdown.

Since .NET was launched, more and more notifications have been added by versioning the interface up to ICorProfilerCallback9. Welcome to the usual COM world! I recommend that you watch Pavel Yosifovich session about writing a CLR Profiler in an hour to get an overview. I will use his sample solution as a starting point.

For performance sake, you tell the runtime which events you are interested in; i.e. which ICorProfilerCallback functions will be called and you simply have to return S_OK from all other functions. This setup is done in your implementation of ICorProfilerCallback::Initialize. This first function called by the CLR provides a parameter from which you need to QueryInterface a version of ICorProfilerInfo interface (the current one is ICorProfilerInfo10). This interface provides functions to query information about parameters passed to your ICorProfilerCallback functions (such as AppDomain, assembly, type, function, thread and so on).

The first ICorProfilerInfo function you will use is SetEventMask to filter the ICorProfilerCallback functions that will be called by the runtime. It accepts a flag combination of values from the COR_PRF_MONITOR enumeration.

Assembly code is needed for Enter/Leave/TailCall

To get notified when a managed method is called or exits, you should pass:

COR_PRF_MONITOR_ENTERLEAVE | 
COR_PRF_ENABLE_FUNCTION_ARGS | 
COR_PRF_ENABLE_FUNCTION_RETVAL | 
COR_PRF_ENABLE_FRAME_INFO

to ICorProfilerInfo::SetEventMask. The first flag tells the runtime to call static callbacks (i.e. not exposed as functions of ICorProfilerCallback) when a managed method gets executed or returns. The other three flags ensure that these callbacks will receive enough information to extract method arguments and return value.

Unlike the other notifications that end up calling your ICorProfilerCallback functions, you need to register three special callbacks to the runtime via ICorProfilerInfo3::SetEnterLeaveFunctionHooks3WithInfo. For performance reasons, the .NET team asks you to write the prolog and epilog (i.e. saving/restoring CPU registers on/from the stack) yourself in assembly code instead of relying on well-defined calling conventions supported by the C/C++ compiler.

This is why you have to call ICorProfilerInfo3:: SetEnterLeaveFunctionHooks3WithInfo and pass pointers to these “naked” functions. The Microsoft ELTProfiler sample implements the stubs both for x86 (as inlined assembly embedded in a C++ file) and x64 (defined in .asm file). In x64, you need to update your project file to add the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


 Label = "ExtensionSettings">
    Project = "$(VCTargetsPath)\BuildCustomizations\masm.props" / >


 Include = "../DotNext.Profiler.Shared/asm/windows/nakedcallbacks.asm"
	Condition = "'$(Platform)' == 'x64'" / >

 Label = "ExtensionTargets">
     Project = "$(VCTargetsPath)\BuildCustomizations\masm.targets" / >

The nakedcallbacks.asm file contains the assembly code to call stub functions wrapped by the expected prolog and epilog written in assembly code. Here is the signature of the functions from where you will be able to start working in C++:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


PROFILER_STUB EnterStub(FunctionIDOrClientID functionId, COR_PRF_ELT_INFO eltInfo)
{
   ...
}

PROFILER_STUB LeaveStub(FunctionID functionId, COR_PRF_ELT_INFO eltInfo)
{
   ...
}

PROFILER_STUB TailcallStub(FunctionID functionId, COR_PRF_ELT_INFO eltInfo)
{
   ...
}

A function is identified by a FunctionID and this is where you start your adventure (I will come back later to the COR_PRF_ELT_INFO parameter). Note that for the EnterStub function, you need to get the FunctionID from the FunctionIDOrClientID.functionId field.

Once your hook callbacks have been registered via ICorProfilerInfo3:: SetEnterLeaveFunctionHooks3WithInfo, it is still possible to decide whether or not a managed method call should trigger them. For that, it is needed to register a “mapper” function that is called once per managed method with one of these functions:

HRESULT ICorProfilerInfo::SetFunctionIDMapper([in] FunctionIDMapper* pFunc); with UINT_PTR __stdcall Mapper(FunctionID functionId, BOOL* pHookFunction)
HRESULT ICorProfilerInfo3::SetFunctionIDMapper2([in] FunctionIDMapper2* pFunc, [in] void* clientData); with UINT_PTR __stdcall Mapper2(FunctionID functionId, void* clientData, BOOL* pHookFunction)

The latter allows you to pass some “client data” to the mapper function such as a helper class to manipulate the received FunctionID or your profiler state.

If pHookFunction is set to TRUE, your enter/leave functions will be called and the returned UINT_PTR will be passed as the FunctionID parameter. This allows you to handle function name or signature computation at one single place outside of the real profiling work done each time a method is called. If pHookFunction is set to FALSE, the enter/leave functions will never be called for that FunctionID. The mapper callback is called only once per FunctionID: this could be a good way to avoid performance impact if you just want to profile a small subset of methods.

How to debug your profiler

Before going any further, it is needed to know how to debug your C++ profiler code with Visual Studio. The first step is to write a simple .NET application to execute the method calls you want to intercept. The second natural step would be to setup the environment variables needed to inject your profiler:

and also check the Enable native code debugging option:

If you start a debug session, the Visual Studio debugger with use both managed and native debugging APIs. Unfortunately, the managed debugging API does not allow breakpoints sets in ICorProfilerCallback functions.

Instead, you need to Debug | Start Without Debugging (CTRL+F5), not Debug | Start Debugging (F5) the C# test application with the same environment variables and attach the debugger via Debug | Attach to Process. Select the process and click Select button in the Attach to section:

To make attachment simple, add a Console.ReadLine in a console application before starting the method calls to test.

The next post will show you how to extract information from a FunctionID.

References

Writing a .NET Core cross platform profiler in an hour video and the corresponding source code from Pavel Yosifovich
Microsoft Profiling documentation
Microsoft Enter/Leave code sample

How to ease async callstacks analysis in Perfview

Tue, 02 Mar 2021 09:35:37 +0000

Introduction

In the previous post, I described why you might get weird reversed callstacks in Visual Studio when analyzing or debugging async/await code. And if you are using Perfview to profile the same application, you should also get the same reverse continuation flow:

The rest of the post describes how to easily profile with Perfview and more interestingly, how to leverage grouping/folding features to get much more readable asynchronous callstacks.

Perfview 101

Here are the different steps to get the previous tree-like representation of a profiling session results.

First, start a data collection by clicking the Start Collection button from the Collect | Collect dialog box and check Kernel Base, CPU Samples, and .NET boxes:

Stop the collection when the application ends and double-click the CPU Stacks node :

After selecting the application in the Select Process Window

click the CallTree tab:

Before entering the dreaded yellow/white CPU Stacks window, let’s spend some time detailing its vast toolbar in the following figure:

The most powerful elements are the *Pats combo-boxes. Each of them supports a “simple” matching pattern syntax for different purposes (don’t worry, you will see how to use them in many more examples later):

GroupPats: merge sibling matching frames into one.
FoldPats: matching frames are folded into parent frame.
IncPats: non matching frames are removed (used for process filtering for example).
ExcPats: matching frames are excluded.

Let’s see what we get with all combo-box set as empty for the CallTree tab:

For server applications, we are usually not interested in making any difference between threads so it would be nice to group all threads under a single AllThreads node. This is exactly what the first choice of the GroupPats combo-box provides:

The effect is simple: all lines at the same level containing “Thread” in the text are merged into a new line with “AllThreads” as new text

You can now get the same kind of tree based representation as in Visual Studio: the difference is that you need to open each node by clicking a checkbox (or right-click + Expand All to see the whole tree)

Most columns meaning are quite self-explicit except maybe the When column that provides the CPU usage graph over time in a “textual” way:

The CallTree representation obviously displays the frames sorted by Inc% column.

Going further with Perfview

When expanding the calltree, you usually get lost in the async/await implementation details. The following screenshot shows the signal/noise ratio for my simple test application where we don’t really care about the blue lines!

This is where the different Perfview combo-boxes are coming to the rescue. Some frames and their children are clearly not interesting such as the last two coreclr!ThePreStub. In that case, select the frame and copy the text from the status bar (yes: this is possible and so handy!)

and paste it into the ExcPats combo-box

to make the corresponding frames disappear.

Unfortunately, you can’t do the same for the other Task-related frames because these ExcPats matching frames are completely removed with their children where your async method calls appear.

Folding patterns are your friends

This time, the FoldPats combo-box will be your friend: each frame that maps one of its ; separated substring will disappear and its occurrence count will be added to its parent frame. Since all these Task-related frame do not appear a lot, the impact in the Inc/Exc columns of the parent frames should be minimal. After I used the following substrings:

Tasks.Task+DelayPromise.CompleteTimedOut(;Tasks.Task.FinishContinuations(;Tasks.Task.RunOrQueueCompletionAction;Tasks.Task.RunContinuations(;Tasks.Task+DelayPromise+;Tasks.AwaitTaskContinuation.RunOrScheduleAction;Runtime.CompilerServices.AsyncTaskMethodBuilder`;CompilerServices.AsyncTaskMethodBuilder.Start(;CompilerServices.AsyncMethodBuilderCore.Start(;ExecutionContext.RunInternal(;CompilerServices.AsyncTaskMethodBuilder.SetResult(;Tasks.VoidTaskResult].TrySetResult;.TrySetResult(System.Threading.Tasks.;Tasks.Task.TrySetResult(

the callstack was much more readable:

Morph the frames

The final step is to transform the frames text into something more meaningful thanks to the GroupPats combo-box. At the beginning of this post, I picked the predefined [fold threads] Thread -> AllThreads grouping pattern. The starting text between [] is used as a title by Perfview to allow the user to more easily figure out what its role is. The rest of the string defines how parts of each frame should match and be grouped. The corresponding contextual help has already been shown earlier when the toolbar was detailed.

Here, I don’t want to group all sibling frames into a group but rather morph the text into something more readable. The part before -> or => is used as matching pattern to be replaced by the part after the sign. It is also possible to “extract” elements between {} from the matching pattern to be used to build the replacement string. Each matched element is identified as $1, $2,… based on its position in the pattern.

In my example, I would like to apply the following transformation:

To write the pattern, you should focus on the separators (!, +< and >d__ in this case):

{%}!{%}+<{%}>d__*

And the items to extract are what is left in between; identified as {%}.

The building of the replacement string is simply counting the matching item position (starting at 1):

($1) $2 ~~~async back to~~~ $3()

And here is the corresponding final result:

Don’t lose your xxxPats!

It is interesting to note that you could define your own patterns preset via the Preset menu item

As you can see here, I have defined my own Criteo Arbitrage preset. If you want to reuse the content of GroupPats, FoldPats, and Fold% combo-boxes, click the Save As Preset (or even Set As Startup Preset to get them when you start Perfview) and pick a name

Feel free to use the Manage Presets dialog for easier preset manipulation:

I hope that, now, you better understand the value of Perfview to analyze complicated callstacks.

Read more from Christophe on our Medium blog!

Consul Streaming: What’s behind it? *Let’s look at new hidden feature for Consul large or very dynamic clusters of Consul 1.9: Streaming.*medium.com

Like what you are reading? Join us and make an impact!

Careers at Criteo | Criteo jobs *Find opportunities everywhere. Choose your next challenge. Find the job opportunities at Criteo in Product, research &…*careers.criteo.com

Build your own .NET CPU profiler in C#

Tue, 08 Dec 2020 10:27:37 +0000

The last series was describing how to get details about your .NET application allocation patterns in C#.

It is now time to do the same but for the CPU consumption of your .NET applications.

Thanks you Mr Windows Kernel!

Under Windows, the kernel ETW provider allows you to get notified every milli-second with the call stack of all threads running on a core. Without any surprise, it is easy with TraceEvent to listen to these events. As explained in an old posts, you simply need to create a session, enable providers and listen to the right event.

For sampled CPU profiling, I’m using the TraceLogEventSource to wrap the event source and automatically get the stack frames symbol resolution:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


string sessionName = "Cpu_Profiling_Session+" + Guid.NewGuid().ToString();
_session = new TraceEventSession(sessionName, TraceEventSessionOptions.Create);
if (!EnableProviders(_session))
{
    _session.Dispose();
    _session = null;
    return false;
}

_profilingTask = Task.Factory.StartNew(() =>
{
    using (TraceLogEventSource source = TraceLog.CreateFromTraceEventSession(_session))
    {
        // CPU sampling kernel events
        source.Kernel.PerfInfoSample += (SampledProfileTraceData data) =>
        {
            ...
        };

        // this call exits when the session is stopped
        source.Process();
    }
});

You need to enable three providers:

Kernel: get the profiling event every milli-second and be notified when a dll gets loaded by a process to let TraceEvent manage the symbols
Clr: get JIT events describing managed method details
ClrRundown: get already JITted methods details

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


protected bool EnableProviders(TraceEventSession session)
{
    session.BufferSizeMB = 256;

    // Note: it could fail if the user does not have the required privileges
    var success = session.EnableKernelProvider(
        KernelTraceEventParser.Keywords.ImageLoad |
        KernelTraceEventParser.Keywords.Process |
        KernelTraceEventParser.Keywords.Profile,
        stackCapture: KernelTraceEventParser.Keywords.Profile
        );
    if (!success) return false;

    // this call always returns false  :^(
    session.EnableProvider(
        ClrTraceEventParser.ProviderGuid,
        TraceEventLevel.Verbose,
        (ulong)(
        // events related to JITed methods
        ClrTraceEventParser.Keywords.Jit |                       // Turning on JIT events is necessary to resolve JIT compiled code 
        ClrTraceEventParser.Keywords.JittedMethodILToNativeMap | // This is needed if you want line number information in the stacks
        ClrTraceEventParser.Keywords.Loader                      // You must include loader events as well to resolve JIT compiled code.
        )
    );

    // this provider will send events of already JITed methods
    session.EnableProvider(
        ClrRundownTraceEventParser.ProviderGuid,
        TraceEventLevel.Verbose,
        (ulong)(
        ClrTraceEventParser.Keywords.Jit |              // We need JIT events to be rundown to resolve method names
        ClrTraceEventParser.Keywords.JittedMethodILToNativeMap | // This is needed if you want line number information in the stacks
        ClrTraceEventParser.Keywords.Loader |           // As well as the module load events.  
        ClrTraceEventParser.Keywords.StartEnumeration   // This indicates to do the rundown now (at enable time)
        ));

    return true;
}

The code to handle the event is really simple:

1
2
3
4
5
6
7
8
9


source.Kernel.PerfInfoSample += (SampledProfileTraceData data) =>
{
    if (data.ProcessID != Pid) return;

    var callstack = data.CallStack();
    if (callstack == null) return;

    MergeCallStack(callstack, Reader);
};

I’m only interested in profiling a given process (hence the check on process id) and events with a call stack. The callstack is returned by the extension method CallStack() (see the previous post for more details). The main processing is done by the MergeCallStack() method. But before looking at the only complicated part, it is time to discuss a useful tip.

Tip: use ETLx Luke!

Like the previous posts about memory profiling, my goal is to demonstrate how to monitor applications as they run. However when you monitor an application CPU consumption, you would like to avoid any noisy neighbor that could highjack some cores. So minimizing the work of your profiling code is always a good idea. In addition, it could also be valuable to record the events and analyze them later. Microsoft Perfview is the open source tool that I’m using the most to dig into CPU consumption. So the solution is to simply record the events and generate an .etlx file for Perfview.

The first code change is small: the session is created with a filename.

1
2


string sessionName = "Cpu_Profiling_Session+" + Guid.NewGuid().ToString();
_session = new TraceEventSession(sessionName, _filename);

I’m using a naming convention that contains the process ID I want to monitor so it will be easy to remember when I will analyze the recording in Perfview:

1

profiler = new EtlCpuSampleProfiler($"trace-{parameters.pid}.etl");

The second step to generate the .etlx file is a one liner:

1

var traceLog = TraceLog.OpenOrConvert(_filename, new TraceLogOptions() { ConversionLog = SymbolMessages });

The ConversionLog TraceLogOptions property is expecting a TextWriter to log all possible messages related to symbols resolution.

The parsing of kernel profiling samples is done on the TraceLog in a more manual way by selecting the events based on TaskGuid corresponding to the kernel profiling task and the OpCode:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


// parse profiling kernel events
// from https://github.com/microsoft/perfview/blob/master/src/TraceEvent/Samples/41_TraceLogMonitor.cs#L150
// from https://docs.microsoft.com/en-us/windows/win32/etw/perfinfo
// from https://github.com/microsoft/perfview/blob/master/src/TraceEvent/Parsers/KernelTraceEventParser.cs#L3128
// and https://github.com/microsoft/perfview/blob/master/src/TraceEvent/Parsers/KernelTraceEventParser.cs#L2298
//
Guid perfInfoTaskGuid = new Guid(0xce1dbfb4, 0x137e, 0x4da6, 0x87, 0xb0, 0x3f, 0x59, 0xaa, 0x10, 0x2c, 0xbc);
int profileOpcode = 46;
foreach (var data in traceLog.Events)
{
    if (data.ProcessID != Pid) continue;
    if (data.TaskGuid != perfInfoTaskGuid) continue;
    if ((uint)data.Opcode != profileOpcode) continue;

    var callstack = data.CallStack();
    if (callstack == null) continue;

    MergeCallStack(callstack, Reader);
}

How to “merge” call stacks

In both live and file based implementations, I end up merging call stacks by calling the MergeCallStack() method. Instead of jumping directly into the C# code, I prefer to describe what I’m expecting from “merging“ call stacks.

If you think about what frames (i.e. method call) would appear at the beginning all these threads call stacks, it seems obvious that they should start with the same code: either the main thread startup, timer/thread pool initialization or custom thread bootstrap. In case of server applications, the same request processing calls would lead to specific handlers or controllers code. Each time a common group of frames appears in different call stacks, it would be more readable to see them as different branches starting from the same trunk like in Visual Studio Parallel Stack panel.

In order to build a “visual” representation, I have to count the number of time each frame appears at the same place in the recorded call stacks. My data structure looks like a tree where each node contains the current frame, the sampling count (as node or as leaf) and a list of different child frames corresponding to the different execution branches:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


public class MergedSymbolicStacks
{
    private int _countAsNode;
    private int _countAsLeaf;

    public ulong Frame { get; private set; }
    public string Symbol { get; private set; }
    public int CountAsNode => _countAsNode;
    public int CountAsLeaf => _countAsLeaf;

    public List<MergedSymbolicStacks> Stacks { get; set; }

Each frame contains both the address and the method signature that have been extracted from the callstack retrieved from the events:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39


protected void MergeCallStack(TraceCallStack callStack, SymbolReader reader)
{
    var currentFrame = callStack.Depth;
    var frames = new SymbolicFrame[callStack.Depth];

    // the first element of callstack is the last frame: we need to iterate on each frame
    // up to the first one before adding them into the MergedSymbolicStack
    while (callStack != null)
    {
        var codeAddress = callStack.CodeAddress;
        if (codeAddress.Method == null)
        {
            var moduleFile = codeAddress.ModuleFile;
            if (moduleFile != null)
            {
                // TODO: this seems to trigger extremely slow retrieval of symbols 
                //       through HTTP requests: see how to delay it AFTER the user
                //       stops the profiling
                if (!_missingSymbols.TryGetValue(moduleFile, out var _))
                {
                    codeAddress.CodeAddresses.LookupSymbolsForModule(reader, moduleFile);
                    if (codeAddress.Method == null)
                    {
                        _missingSymbols[moduleFile] = true;
                    }
                }
            }
        }
        frames[--currentFrame] = new SymbolicFrame(
            codeAddress.Address,
            codeAddress.FullMethodName
            );

        callStack = callStack.Caller;
    }

    _stackCount++;
    _stacks.AddStack(frames);
}

The MergedSymbolicStack.AddStack() method is doing the real merging. The idea of merging call stacks is to start from the bottom and if the frame has already been seen (at this position), increment its sampling count. If not, remember it before incrementing the count. Look at the next frame and do the same match/remember + increment up to the top of the stack.

Here is an animation of what it would look like on a piece of paper (like the one I wrote down before starting to write the C# implementation :^)

Here is the corresponding C# code to merge a stack (i.e. an array of frames)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


public void AddStack(SymbolicFrame[] frames, int index = 0)
{
    _countAsNode++;

    var firstFrame = frames[index];

    // search if the frame to add has already been seen
    var callstack = Stacks.FirstOrDefault(s => string.CompareOrdinal(s.Symbol, firstFrame.Symbol) == 0);

    // if not, we are starting a new branch
    if (callstack == null)
    {
        callstack = new MergedSymbolicStacks(frames[index].Address, frames[index].Symbol);
        Stacks.Add(callstack);
    }

    // it was the last frame of the stack
    if (index == frames.Length - 1)
    {
        callstack._countAsLeaf++;
        return;
    }

    callstack.AddStack(frames, index + 1);
}

Last but not least, the constructors of the class reflect how to (1) create the root instance and (2) each node in the tree:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


public MergedSymbolicStacks() : this(0, string.Empty)
{
    // this will be the root of all stacks
}

private MergedSymbolicStacks(ulong frame, string symbol)
{
    Frame = frame;
    Symbol = symbol;
    _countAsNode = 0;
    _countAsLeaf = 0;
    Stacks = new List<MergedSymbolicStacks>();
}

The code to render the merged stack

is not that complicated because everything is already in the tree of frames.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32


private static void RenderStack(MergedSymbolicStacks stack, IRenderer visitor, bool isRoot, int increment)
{
    var alignment = new string(' ', Padding * increment);
    var padding = new string(' ', Padding);
    var currentFrame = stack.Frame;

    // special root case
    if (isRoot)
        visitor.WriteCount($"{Environment.NewLine}{alignment}{stack.CountAsNode, Padding} ");
    else
        visitor.WriteCount($"{Environment.NewLine}{alignment}{stack.CountAsLeaf + stack.CountAsNode, Padding} ");

    visitor.WriteMethod(stack.Symbol);

    var childrenCount = stack.Stacks.Count;
    if (childrenCount == 0)
    {
        visitor.WriteFrameSeparator("");
        return;
    }
    foreach (var nextStackFrame in stack.Stacks.OrderByDescending(s => s.CountAsNode + s.CountAsLeaf))
    {
        // increment when more than 1 children
        var childIncrement = (childrenCount == 1) ? increment : increment + 1;
        RenderStack(nextStackFrame, visitor, false, childIncrement);
        if (increment != childIncrement)
        {
            visitor.WriteFrameSeparator($"{Environment.NewLine}{alignment}{padding}{nextStackFrame.CountAsNode + nextStackFrame.CountAsLeaf, Padding} ");
            visitor.WriteFrameSeparator($"~~~~ ");
        }
    }
}

The IRenderer interface implementations are simply changing foreground color depending on what kind of information to display:

I have used the same “Visitor” pattern for the pstack tool/extension for WinDBG.

Not for Admin only

I always thought that I needed to be a member of the Administrator group and running elevated to be allowed to start a kernel profiling session. Well… This is in fact not the case! You have to dig into the documentation for configuring and starting a SystemTraceProvider session to read the following note:

If you want a non-administrators or a non-TCB process to be able to start a profiling trace session using the SystemTraceProvider on behalf of third party applications, then you need to grant the user profile privilege and then add this user to both the session GUID (created for the logger session) and the system trace provider GUID to enable the system trace provider. For more information, see the EventAccessControl function.

Long story short, you need a user to be part of the Performance Log Users group (makes sense) or grant her the TRACELOG_ACCESS_REALTIME permission. Obviously, you need an administrator account to do both but this can be done once on a machine by your IT in a secure way.

I wrapped a managed implementation of the corresponding code to add the permission in a ProfilingPermission class that hides all the P/Invoke and weird marshalling stuff to the native Windows API. Simply pass a user name to EnableProfileUser() and it should work just fine.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131


public static class ProfilingPermission
{
    private const uint TRACELOG_GUID_ENABLE = 0x0080;
    private const int NO_ERROR = 0;  // ERROR_SUCCESS in C++
    private const int ERROR_INSUFFICIENT_BUFFER = 122;

    // read https://docs.microsoft.com/en-us/windows/win32/etw/configuring-and-starting-a-systemtraceprovider-session 
    // for more details 
    public static void EnableProfilerUser(string accountName)
    {
        // Kernel provider from https://github.com/microsoft/perfview/blob/master/src/TraceEvent/Parsers/KernelTraceEventParser.cs#L43
        Guid kernelProviderGuid = new Guid("{9e814aad-3204-11d2-9a82-006008a86939}");
        byte[] sid = LookupSidByName(accountName);

        // from https://docs.microsoft.com/en-us/windows/win32/etw/configuring-and-starting-a-systemtraceprovider-session
        uint operation = (uint)EventSecurityOperation.EventSecurityAddDACL;
        uint rights = TRACELOG_GUID_ENABLE;
        bool allowOrDeny = ("Allow" != null);
        uint result = EventAccessControl(
            ref kernelProviderGuid,
            operation,
            sid,
            rights,
            allowOrDeny
        );

        if (result != NO_ERROR)
        {
            var lastErrorMessage = new Win32Exception((int)result).Message;
            throw new InvalidOperationException($"Failed to add ACL ({result.ToString()}) : {lastErrorMessage}");
        }
    }

    private static byte[] LookupSidByName(string accountName)
    {
        byte[] sid = null;
        uint cbSid = 0;
        StringBuilder referencedDomainName = new StringBuilder();
        uint cchReferencedDomainName = (uint)referencedDomainName.Capacity;
        SID_NAME_USE sidUse;

        int err = NO_ERROR;
        if (!LookupAccountName(null, accountName, sid, ref cbSid, referencedDomainName, ref cchReferencedDomainName, out sidUse))
        {
            err = Marshal.GetLastWin32Error();
            if (err == ERROR_INSUFFICIENT_BUFFER)
            {
                sid = new byte[cbSid];
                referencedDomainName.EnsureCapacity((int)cchReferencedDomainName);
                err = NO_ERROR;
                if (!LookupAccountName(null, accountName, sid, ref cbSid, referencedDomainName, ref cchReferencedDomainName, out sidUse))
                    err = Marshal.GetLastWin32Error();
            }
        }

        if (err != NO_ERROR)
        {
            var lastErrorMessage = new Win32Exception(err).Message;
            throw new InvalidOperationException($"LookupAccountName fails ({err.ToString()}) : {lastErrorMessage}");
        }

        // display the SID associated to the given user
        IntPtr ptrSid;
        if (!ConvertSidToStringSid(sid, out ptrSid))
        {
            err = Marshal.GetLastWin32Error();
            var lastErrorMessage = new Win32Exception(err).Message;
            Console.WriteLine($"No SID string associated to user {accountName} ({err.ToString()}) : {lastErrorMessage}");
        }
        else
        {
            string sidString = Marshal.PtrToStringAuto(ptrSid);
            ProfilingPermission.LocalFree(ptrSid);
            Console.WriteLine($"Account ({referencedDomainName}){accountName} mapped to {sidString}");
        }

        return sid;
    }

    [DllImport("Sechost.dll", SetLastError = true)]
    static extern uint EventAccessControl(
        ref Guid providerGuid,
        uint operation,
        [MarshalAs(UnmanagedType.LPArray)] byte[] Sid,
        uint right,
        bool allowOrDeny // true means ALLOW
        );

    [DllImport("kernel32.dll")]
    static extern IntPtr LocalFree(IntPtr hMem);

    [DllImport("advapi32.dll", SetLastError = true)]
    static extern bool LookupAccountName(
        string systemName,
        string accountName,
        [MarshalAs(UnmanagedType.LPArray)] byte[] Sid,
        ref uint cbSid,
        StringBuilder referencedDomainName,
        ref uint cchReferencedDomainName,
        out SID_NAME_USE nameUse);

    [DllImport("advapi32.dll", CharSet = CharSet.Auto, SetLastError = true)]
    static extern bool ConvertSidToStringSid(
        [MarshalAs(UnmanagedType.LPArray)] byte[] pSID,
        out IntPtr ptrSid); // can't be an out string because we need to explicitly call LocalFree on it;
                            // the marshaller would call CoTaskMemFree in case of a string

    // from http://pinvoke.net/default.aspx/advapi32/LookupAccountName.html
    enum SID_NAME_USE
    {
        SidTypeUser = 1,
        SidTypeGroup,
        SidTypeDomain,
        SidTypeAlias,
        SidTypeWellKnownGroup,
        SidTypeDeletedAccount,
        SidTypeInvalid,
        SidTypeUnknown,
        SidTypeComputer
    }

    // from evntcons.h
    enum EventSecurityOperation
    {
        EventSecuritySetDACL = 0,
        EventSecuritySetSACL,
        EventSecurityAddDACL,
        EventSecurityAddSACL,
        EventSecurityMax
    } // EVENTSECURITYOPERATION
}

You are now ready to profile your application memory allocation patterns and CPU consumption!

Thanks for checking in with us again on our C# series. Like what you are reading? Head over to our latest blog posts on the topic:

Build your own .NET memory profiler in C# *This post explains how to collect allocation details by writing your own memory profiler in C#.*medium.com Build your own .NET memory profiler in C# — call stacks (2/2–1) *This post explains how to get the call stack corresponding to the allocations with CLR events.*medium.com

If you are interested in joining our team, check out our open positions and apply today!

Careers at Criteo | Criteo jobs *Find opportunities everywhere. Choose your next challenge. Find the job opportunities at Criteo in Product, research &…*careers.criteo.com

How to write your own commands in dotnet-dump (2/2)

Mon, 09 Nov 2020 16:27:38 +0000

In the previous post, I presented the new commands that were added to dotnet-dump and how to use them. It is now time to show how to implement such a command.

But before jumping into the code, you should first ensure that you have a valid use case that the Diagnostics team is not currently working on. I recommend to create an issue in the Diagnostics repository to explain what is missing for which scenario and propose to implement the corresponding command.

What is a dotnet-dump command?

Here is the directory structure related to the dotnet-dump tool in the Diagnostics repository:

The built binaries are generated under artifacts\bin\dotnet-dump\netcoreapp2.1 folder if you need to test them outside of Visual Studio.

The eng folder contains the versions.props file that lists the versions for nuget dependencies. In my case, I had to reference the ParallelStacks.Runtime nuget so I added the following line:2.0.1

And in the dotnet-dump.csproj, this nuget is referenced with the same variable: `

Missed the first part of the story? Read it here:

How to extend dotnet-dump (1/2) — What are the new commands? *This first post describes the new commands, when to use them, and the git setup I used to implement them.*medium.com

Want to work with Christophe or other teams? Check out our open positions:

Careers at Criteo | Criteo jobs *Find opportunities everywhere. Choose your next challenge. Find the job opportunities at Criteo in Product, research &…*careers.criteo.com

How to extend dotnet-dump (1/2) — What are the new commands?

Tue, 29 Sep 2020 14:26:16 +0000

Introduction

To ease our troubleshooting sessions at Criteo, new high level commands for WinDBG have been written and grouped in the gsose extension. As we moved to Linux, Kevin implemented the plumbing to be able to load gsose into LLDB. In both cases, our extension commands are based on ClrMD to dig into a memory dump.

As Microsoft pushed for dotnet-dump as the new cross-platform way to deal with memory dump, it was obvious that we would have to be able to use our extension commands in dotnet-dump. Unfortunately dotnet-dump does not support any extension mechanism. In May this year, Kevin updated a minimum of code to load a list of extension assemblies at the startup of dotnet-dump. I followed another direction by adding a “load” command to dynamically add extensions commands.

However, the Diagnostics team was focusing on supporting .NET Core 5 release and adding extensibility was not planned before next year. Due to our own time constraints at Criteo, gsose extension commands were really needed so we agreed with the Diagnostics team to implement these commands directly into dotnet-dump as pull requests.

This first post describes the new commands, when to use them, and the git setup I used to implement them.

Commands details and challenges

Here is the list of extension commands as shown by the help command:

As of today with the last 3.1.141901 official version, only timerinfo is available but feel free to clone the repository and rebuild it to get all commands.

pstacks: almost Parallel Stacks a la Visual Studio

When you start an investigation, you are usually interested in getting a high level view of what the running threads are doing and this is what pstacks provides. When hundreds of threads are running, commands such as clrthreads are useless. Instead, pstacks merges the common parts of each call stack and present them in a tree:

Don’t be scared by the layout; here is a description of each section:

If you look at the threads 9c6c and 4020 (after the last ~~~~), it means that 2 threads (only the first 4 threads id are shown except if you pass — all as a parameter) share the same callstack:

~~~~ 9c6c,4020
2 System.Threading.Monitor.Wait(Object, Int32, Boolean)
4 System.Threading.Tasks.Task+c.cctor>b__278_1(Object)
  ...
4 System.Threading.Tasks.Task.ExecuteEntryUnsafe()
4 System.Threading.Tasks.Task.ExecuteWorkItem()
7 System.Threading.ThreadPoolWorkQueue.Dispatch()
7 System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

Compared to the “list” representation, the “tree” representation is useful when you have a lot of threads to deal with and see the different branches in the groups of stack frames.

ti: see all your timers

If you don’t find anything interesting in the running threads (i.e. not hundreds of threads with the same code stack blocked on a Wait call for example), you should look at what will run as timer callbacks with the ti command.

Here is example of what you will get:

> ti
(S) 0x00007F7D84529CD0 @    1000 ms every     1000 ms |  0000000000000000 () -> Kafka.Batching.AccumulatorByTopic.Tick
    ...
(L) 0x00007F7D844FA7A0 @    1000 ms every     1000 ms |  0000000000000000 () -> Kafka.Batching.AccumulatorByTopic.Tick
    ...
(L) 0x00007F7D842042F8 @     999 ms every     1000 ms |  0000000000000000 () -> Criteo.DevKit.TimeStamp+<>c.cctor>b__35_0
    ...
(L) 0x00007F7D846314C8 @     999 ms every     1000 ms |  00007F7D8462ED70 (Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.Heartbeat) ->
    ...
 
   190 timers
-----------------------------------------------
   1 | (S) @     999 ms every     1000 ms | 0000000000000000 () -> Kafka.Batching.AccumulatorByTopic.Tick
   1 | (L) @     996 ms every     1000 ms | 0000000000000000 () -> Kafka.Batching.AccumulatorByTopic.Tick
   1 | (L) @     993 ms every     1000 ms | 0000000000000000 () -> Kafka.Batching.AccumulatorByTopic.Tick
   1 | (L) @     999 ms every     1000 ms | 0000000000000000 () -> Criteo.DevKit.TimeStamp+<>c.cctor>b__35_0
   ...
   9 | (L) @    1000 ms every     1000 ms | 0000000000000000 () -> Kafka.Batching.AccumulatorByTopic.Tick
  33 | (L) @     999 ms every     1000 ms | 0000000000000000 () -> Kafka.Batching.AccumulatorByTopic.Tick
  34 | (L) @    2000 ms every     2000 ms | 0000000000000000 () -> Kafka.Batching.AccumulatorByTopicByPartition.Tick
  38 | (L) @     999 ms every     1000 ms | 0000000000000000 () -> Kafka.Batching.AccumulatorByTopic.Tick

The first part of the output lists each instance of Timer with start/repeat information followed by the callback parameter and callback if available. The second list groups timer so you could identify cases where the same timers have been created many times.

tpq: what is in the ThreadPool queues

If no culprit is identified in timers, it is time to look at what is in the ThreadPool with the tpq command.

> tpq
 
global work item queue________________________________
0x000002AC3C1DDBB0 Work | (ASP.global_asax)System.Web.HttpApplication.ResumeStepsWaitCallback
                       ...
0x000002AABEC19148 Task | System.Threading.Tasks.Dataflow.Internal.TargetCore.b__3
 
local per thread work items_____________________________________
0x000002AE79D80A00 System.Threading.Tasks.ContinuationTaskFromTask
                       ...
0x000002AB7CBB84A0 Task | System.Net.Http.HttpClientHandler.StartRequest
 
   7 Task System.Threading.Tasks.Dataflow.Internal.TargetCore.b__3
                       ...
  84 Task System.Net.Http.HttpClientHandler.StartRequest
----
6039
 
1810 Work  (ASP.global_asax) System.Web.HttpApplication.ResumeStepsWaitCallback
----
1810

Both work items from the global queue and the local queues are displayed with each identified callback. The final summary spits work items and tasks.

Miscellaneous commands

Two other helper commands are also available.

tks: Task State

Pass a Task object reference to tks to get its human readable state.

> help tks
-------------------------------------------------------------------------------
TaskState [hexa address] [-v ]
 
TaskState translates a Task m_stateFlags field value into human readable format.
It supports hexadecimal address corresponding to a task instance or -v .
 
> tks 000001db16cf98f0
Running
 
> tks -v 73728
WaitingToRun

dcq : Dump ConcurrentQueue

Dump elements stored in a ConcurrentQueue. Due to implementation details, more manual steps are required in case of value types.

> help dcq
-------------------------------------------------------------------------------
DumpConcurrentQueue
 
Lists all items in the given concurrent queue.
 
For simple types such as numbers, boolean and string, values are shown.
> dcq 00000202a79320e8
System.Collections.Concurrent.ConcurrentQueue
   1 - 0
   2 - 1
   3 - 2
 
In case of reference types, the command to dump each object is shown.
> dcq 00000202a79337f8
System.Collections.Concurrent.ConcurrentQueue
   1 - dumpobj 0x202a7934e38
   2 - dumpobj 0x202a7934fd0
   3 - dumpobj 0x202a7935078
 
For value types, the command to dump each array segment is shown.
The next step is to manually dump each element with dumpvc  <[item] address>.
> dcq 00000202a7933370
System.Collections.Concurrent.ConcurrentQueue
   1 - dumparray 202a79334e0
   2 - dumparray 202a7938a88

Note that for reference type items, the dumpobj command is provided to get the value of the item fields: you just copy and paste them to get instance fields.

GIT hell

Before digging into the implementation details, I want to spend some time on how to properly create a pull request. Since the Diagnostics repository is hosted on Github, you have to create a fork and push your changes on a dedicated branch to then submit a pull request.

Due to my previous Team Foundation Server experience, it seems that I have problems with Git commands: too many different ways to do simple things maybe. So I was faithfully relying on the documentation to configure/sync a fork and I ended up merging the Microsoft Master branch to our Criteo fork Master before creating my own branch dedicated to my pull request:

After a while, in the next pull requests, I started to have unrelated commits in the pull requests:

It was due to the fact that I was merging the Criteo fork master with Diagnostics master to stay in sync.

My coworker Guillaume explained to me that I should follow a different path. This time, I’m just creating a branch on the Microsoft repository (so no need to merge) and I’m passing by the Criteo fork just to push upstream:

git clone https://github.com/dotnet/diagnostics
cd diagnostics
git checkout -b PR_dotnet-dump_pstacks

Now I can change my local source code before pushing upstream to Criteo fork

git add/gui 
git commit
git remote add upstream https://github.com/criteo-forks/diagnostics
git push upstream PR_dotnet-dump_pstacks

That way, I can synchronize the Criteo fork without “impacting” the history of commits brought with my dedicated pull request branch.

The next episode will cover command implementation details.

What’s next? Read the second part of this article:

How to write commands for dotnet-dump This post describes the different steps, tips and tricks to write your own commands for dotnet-dumpmedium.com

Like what you are reading? Check out more articles from Christophe on our medium account.

The .NET Core Journey at Criteo *This post shows the challenges we faced during the migration to .NET Core on containerized Linux for our main…*medium.com

Are you interested to work with Christophe and other talented Engineers at Criteo? Take a look at our open positions:

Product, Research & Development | Criteo Careers careers.criteo.com

The .NET Core Journey at Criteo

Fri, 31 Jul 2020 15:55:50 +0000

Introduction

When I arrived at Criteo in late 2016, I joined the .NET Core “guild” (i.e. group of people from different teams dedicated to a specific topic). The first meeting I attended included Microsoft folks led by Scott Hunter (head of .NET program management) and including David Fowler (SignalR and ASP.NET Core). The goal for Criteo was simple: Moving a set of C# applications from Windows/.NET Framework to Linux/.NET Core. I guess that for Microsoft we were a customer with workloads that could be interesting to support with .NET Core. At that time, I did not realize how strong their commitment to work with us was. Our Open Source mindset was the selling point.

How complicated could it be? Well… this post will show you the challenges that we had to face to run, monitor and debug our applications.

Try it

Once we got a build of all .NET Core assemblies (more on this in a forthcoming blog post), it was time to run a few applications. The first issues that we faced were related to missing features between .NET Framework and .NET Core. For example, we need cryptography support of 3DES and AES with cypher mode CFB but it is (still) not available in .NET Core for Linux. Thanks to the Open Source status of .NET Core, we were able to add it to CoreFx. However, since we did not implement it on MacOS/Windows as Microsoft requested for our change to be accepted as a Pull Request, we had to keep our Criteo-forked branch.

The second class of runtime problems we had to solve were due to differences between Windows and Linux but also with the “containerization” of the runtime environment. Let’s take two examples involving the .NET Garbage Collector. First, our containers were using Linux cgroups to manage quotas including memory and number of CPU cores usable by applications. However, at CLR startup, the GC was counting the total count of CPU cores to compute the number of heaps to allocate instead of the one defined at the cgroup level: We ended up with instant Out Of Memory automatic killing. This time our fix was done and merged in the CLR repository.

The second example is related to a GC optimization: During background generation 2 collections, the CLR threads working underneath are affinitized to each different CPU core to avoid locks. We were lucky enough to welcome Maoni Stephens (Lead Dev on the GC) in our Paris office early 2018 to share our weird allocation patterns that impacted the GC. During her stay, she was kind enough to help us investigate a behavior on our servers: When SysInternals ProcessExplorer was running, the garbage collections were taking more time than usual. Maoni found out ProcessExplorer had an affinitized high priority thread conflicting with GC threads. During investigations related to longer response time on Linux compared to Windows. We realized that GC threads were not affinitized like it was the case on Windows and the issue was fixed by Jan Vorlicek.

Here is our lesson: Sometimes fixes are merged into the official release and sometimes they are not. If your workloads are pushing .NET to its limits, you will probably have to build and manage your own Core fork and make it available to your deployments.

Monitor it

At Criteo, our Grafana dashboards measuring .NET Framework application health were based on metrics computed from Windows performance counters. Even without going to Linux, .NET Core is no more exposing performance counters so we had to entirely rebuild our metrics collection system!

Based on Microsoft feedbacks, we decided to listen to CLR events emitted via ETW on Windows and LTTng on Linux. In addition to work for both Operating Systems, these events are also providing accurate details about thread contention, exceptions and garbage collections not available with Performance counters. Please refer to our series of blog posts for more details and reusable code samples to integrate these events into your own systems.

Our first Linux metrics collection implementation was based on LTTng and we presented our journey during the Tracing Summit in 2017. Microsoft already built TraceEvent, an assembly allowing .NET code to parse CLR events for both Windows and Linux. Unfortunately for us, the Linux part was only able to load traces files but we needed live session like on Windows where you can listen to events emitted by running applications. Since this code is Open Source, Gregory was able to add the live session feature to TraceEvent.

With .NET Core 3.0 Microsoft provided a way to exchange events common to Linux and Windows called EventPipes. So… we moved our collection implementation from LTTng to EventPipe (look at our blog series and DotNext conference session for more details and reusable code sample). With the new EventPipe implementation in the CLR came performance issues not seen by Microsoft. The reason is simple: Some of our applications are running hundreds of threads to process thousands of requests per second and allocate memory like crazy. In that kind of context, the CLR has a lot to do and so, has a lot of events to generate and emit via LTTng or EventPipes.

The initial implementation was lacking some filtering and too many events were generated or expensive event payload was created even though the events were not emitted. Based on our feedback, the Microsoft Diagnostic team was very responsive and quickly fixed the problem.

Microsoft did not “just” move to Open Source, the teams are working deeply integrated with the issue/pull request model of GitHub. So don’t be shy and if you find a problem, create an issue with a detailed reproduction and even better, provide a pull request with the fix. Everyone in the community will benefit!

Run it

With these metrics, we started to investigate some performance differences (mostly response time) between Windows and containerized Linux.

We saw a huge performance difference on Linux: Both response time (x2) and scalability (timeout increase with QPS). Our team spent a lot of time to improve the situation up to the point where it was possible to send the applications to production.

In the new containerized environment we faced the same kind of noisy neighbor symptoms that we had with Process Explorer. If the CPU cores are not dedicated to a container (as it was for us at the beginning), this scenario happens a lot. So we updated the scheduling system to dedicate CPU cores to containers.

On a totally different area, we found out that the way .NET Core handles network I/O continuation had an impact on our main application. To give a bit of context, this application has to handle a lot of requests and is response-time driven. During the processing of a request, the current thread might have to send an HTTP request before continuing its processing. Since this is done asynchronously, the thread is now available to process more incoming requests and this is good for throughput. However, it means that when the inner HTTP request comes back, all available threads might be processing new incoming requests and it will take time to complete the old one. The net effect is to increase the median response time and this is not something we want!

The .NET Core implementation is relying on the .NET ThreadPool that shares its threads with all the async/await magic and the incoming requests processing (The .NET Framework implementation is using a totally different implementation based on I/O completion ports on Windows). To solve the issue, Kevin implemented a custom thread pool to handle network I/O and we keep on optimizing it. When you work on this kind of deep area of code-shared by so many different workloads, you realize that it is impossible to find the silver bullet.

Debug it

What would you do if something would go wrong in an application? On Windows, with Visual Studio, we are able to remote debug a rogue application to set a breakpoint, look at fields and properties or even have a high-level view of what threads are doing with the ParallelStacks view. In the worst case, SysInternals procdump allows us to take a snapshot of the application and analyze it on our developer’s machine with WinDBG or Visual Studio.

In terms of remote debugging a Linux application, Microsoft provides an SSH-based solution to attach to a running application. However, for security reasons, it is not allowed to run an SSH server in our Criteo containers. The solution was to implement the communication protocol with VsDbg for Linux on top of WebSockets.

Well… this was not enough. Hosting architecture (Marathon and Mesos in our case) ensures that applications in containers are running smoothly by sending requests to health check endpoints. If the application replies that everything is fine, then the container is safe. If the application does not answer as expected (including retries), then Marathon/Mesos kills the application and cleans up the container. Now think about what will happen if you set a breakpoint in the application and you dig into the data structures content in Visual Studio Watch/Quick Watch panels for a few minutes. Behind the scene, the debugger has to freeze all application threads, including the ones from the thread pool responsible to answer health checks. As you have probably guessed already, the debugging session will not end well.

This is why the previous figure shows an arrow between Marathon and the Remote Debugger which acts as a proxy for the application health check. When a debugging session starts (i.e. when the WebSockets code executes the protocol), the Remote Debugger knows that it should answer OK instead of calling the application endpoint that might never answer.

When remote debugging is not enough, how do you take a memory snapshot of the application? For example, if the health check does not answer after a series of retry, the Remote Debugger is calling the createdump tool installed with the .NET Core runtime to generate a dump file. Again, since the memory dump creation of 40+ GB applications could take several minutes, the same health check proxy mechanism has been put in place.

Once the dump file is created, the remote debugger let Marathon kill the application. But wait! This is not enough because in that case, the container will be cleaned up and the disk storage will disappear. Not a problem, after a dump has been generated by createdump, the file is sent to a “Dump Navigator” application (one per data center). This application is providing a simple HTML user interface to get high-level details of the application state such as thread stacks or managed heap content.

On Windows, we have built our own set of extension commands that allow us to investigate memory, threadpool starvation, thread contention, or timer leak scenarios in a Windows memory dump with WinDBG as shown during this NDC Oslo conference session. Note that they are also usable with LLDB on Linux. These commands are leveraging the ClrMD Microsoft library that gives you access to a live process or a memory dump in C#. Thanks to the Linux support that has been added to this library by Microsoft developers, it was easy to reuse the code into our Dump Navigator application. I definitively recommend to look at the API provided by ClrMD to automate and build your own tools. The long Criteo blog series is a good start in addition to my DotNext conference session.

Conclusion

Even though some of our main applications moved to .NET Core running on containerized Linux with a large set of monitoring/debugging tools, the journey is not over. We are now testing the preview of .NET Core 5.0 (like we did for 3.0) to check if it supports Criteo specific needs. If this is not the case, we will figure out why and find solutions to integrate into the code. Same for the tools: I have started to add our extension commands to Microsoft dotnet-dump CLI tool used to analyze both Windows and Linux dumps.

At least we could say that we not only helped ourselves but also Microsoft to understand how far .NET Core could go and even the whole .NET Windows and Linux community. This is where Open Source shines!

Stay tuned for the next article in our mini-series. Don’t forget to head over to our previous articles of this journey:

Migrating Arbitrage to Apache Mesos *Lessons learned from migrating our largest application to our container platform.*medium.com Moving .NET to Linux at Scale *The story of a multi-year migration: How we changed Criteo’s whole foundation.*medium.com

Interested in joining the challenge? Head over to our career site!

Product, Research & Development | Criteo Careers careers.criteo.com

Build your own .NET memory profiler in C# — call stacks (2/2–2)

Fri, 19 Jun 2020 09:32:16 +0000

In the past two episodes of this series I have explained how to get a sampling of .NET application allocations and one way to get the call stack corresponding to the allocations; all with CLR events. In this last episode, I will detail how to transform addresses from the stack into methods name and possibly signature.

From managed address to method signature

In order to transform an address on the stack into a managed method name, you need to know where in memory (i.e. at which address) is stored the method JITted assembly code and what is its size:

For each JITted method, the MethodLoadVerbose/MethodDCStartVerboseV2 events are providing this information in addition to 3 properties to rebuild the full method name and signature (more on this later). I’m storing each method description as a MethodInfo into a MethodStore per process.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


public class PerProcessProfilingState : IDisposable
{
    ...
    private readonly Dictionary<int, MethodStore> _methods = new Dictionary<int, MethodStore>();

public class MethodStore : IDisposable
{
    // JITed methods information (start address + size + signature)
    private readonly List<MethodInfo> _methods;
    ...

The only interesting part of the MethodInfo class is the computation of the full method name stored in the _fullName field:

1
2
3
4
5
6


public class MethodInfo
{
    private readonly ulong _startAddress;
    private readonly int _size;
    private readonly string _fullName;
    ...

The ComputeFullName helper merges together the 3 properties given by the MethodxxxVerbose events including special processing for constructors:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


private string ComputeFullName(ulong startAddress, string namespaceAndTypeName, string name, string signature)
{
    var fullName = signature;

    // constructor case: name = .ctor | namespaceAndTypeName = A.B.typeName | signature = ...  (parameters)
    // --> A.B.typeName(parameters)
    if (name == ".ctor")
    {
        return $"{namespaceAndTypeName}{ExtractParameters(signature)}";
    }

    // general case: name = Foo | namespaceAndTypeName = A.B.typeName | signature = ...  (parameters)
    // --> A.B.Foo(parameters)
    fullName = $"{namespaceAndTypeName}.{name}{ExtractParameters(signature)}";
    return fullName;
}

private string ExtractTypeName(string namespaceAndTypeName)
{
    var pos = namespaceAndTypeName.LastIndexOf(".", StringComparison.Ordinal);
    if (pos == -1)
    {
        return namespaceAndTypeName;
    }
    
    // skip the .
    pos++;

    return namespaceAndTypeName.Substring(pos);
}

Only the parameters (not the return type) are extracted from the “return type SPACE SPACE (parameters)” signature format:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


private string ExtractParameters(string signature)
{
    var pos = signature.IndexOf("  (");
    if (pos == -1)
    {
        return "(???)";
    }

    // skip double space
    pos += 2;

    var parameters = signature.Substring(pos);
    return parameters;
}

With the starting address and the size of each JITted methods, it is easy to find the one corresponding to a given address on the stack: look for the MethodInfo where this address could be between the start address and the start address + the code size:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


public string GetFullName(ulong address)
{
    if (_cache.TryGetValue(address, out var fullName))
        return fullName;
    
    // look for managed methods
    for (int i = 0; i < _methods.Count; i++)
    {
        var method = _methods[i];
        
        if ((address >= method.StartAddress) && (address < method.StartAddress + (ulong)method.Size))
        {
            fullName = method.FullName;
            _cache[address] = fullName;
            return fullName;
        }
    }

    // look for native methods
    fullName = GetNativeMethodName(address);
    _cache[address] = fullName;

    return fullName;
}

For performance sake, the _cache dictionary property speeds up the process by keeping track of the address/full name mappings.

It is now time to look at the details of the GetNativeMethodName helper that takes care of the native functions scenario.

The native part of the symbols story

Unlike for JITted methods, the CLR does not send events to describe native functions even for the CLR itself. Instead, you need to find a way to map a call stack address to a native function by yourself. Unlike Perfview, I will be using the dbghelp native API instead of DIA mostly because my scenario is to get the stacks while the applications are still running:

After reading the march 2002 MSDN article about DBGHELP by Matt Pietrek, the updated symbols related Microsoft Docs and the dbghelp.h include a file from the Windows SDK, I wrote a C# wrapper around the dbghelp function needed to get a method name from an address in a process address space:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


internal static class NativeDbgHelp
{
    // from C:\Program Files (x86)\Windows Kits\10\Debuggers\inc\dbghelp.h
    public const uint SYMOPT_UNDNAME = 0x00000002;
    public const uint SYMOPT_DEFERRED_LOADS = 0x00000004;

    [StructLayout(LayoutKind.Sequential)]
    public struct SYMBOL_INFO
    {
        public uint SizeOfStruct;
        public uint TypeIndex;      // Type Index of symbol
        private ulong Reserved1;
        private ulong Reserved2;
        public uint Index;
        public uint Size;
        public ulong ModBase;       // Base Address of module containing this symbol
        public uint Flags;
        public ulong Value;         // Value of symbol, ValuePresent should be 1
        public ulong Address;       // Address of symbol including base address of module
        public uint Register;       // register holding value or pointer to value
        public uint Scope;          // scope of the symbol
        public uint Tag;            // pdb classification
        public uint NameLen;        // Actual length of name
        public uint MaxNameLen;
        [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 1024)]
        public string Name;
    }

    [DllImport("dbghelp.dll", SetLastError = true)]
    public static extern bool SymInitialize(IntPtr hProcess, string userSearchPath, bool invadeProcess);

    [DllImport("dbghelp.dll", SetLastError = true)]
    public static extern uint SymSetOptions(uint symOptions);

    [DllImport("dbghelp.dll", SetLastError = true, CharSet = CharSet.Ansi)]
    public static extern ulong SymLoadModule64(IntPtr hProcess, IntPtr hFile, string imageName, string moduleName, ulong baseOfDll, uint sizeOfDll);

    // use ANSI version to ensure the right size of the structure 
    // read https://docs.microsoft.com/en-us/windows/win32/api/dbghelp/ns-dbghelp-symbol_info
    [DllImport("dbghelp.dll", SetLastError = true, CharSet = CharSet.Ansi)]
    public static extern bool SymFromAddr(IntPtr hProcess, ulong address, out ulong displacement, ref SYMBOL_INFO symbol);

    [DllImport("dbghelp.dll", SetLastError = true)]
    public static extern bool SymCleanup(IntPtr hProcess);
}

Note that you will need to download the dbghelp.dll (SymSrv.dll if needed) from the Windows SDK and copy it next to your memory profiler binaries.

The usage of the dbghelp API is straightforward. First, for each new process, call SymSetOptions/SymInitialize** **with a handle of the process:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


private bool SymInitialize(IntPtr hProcess)
{
    // read https://docs.microsoft.com/en-us/windows/win32/api/dbghelp/nf-dbghelp-symsetoptions for more details
    // maybe SYMOPT_NO_PROMPTS and SYMOPT_FAIL_CRITICAL_ERRORS could be used
    NativeDbgHelp.SymSetOptions(
        NativeDbgHelp.SYMOPT_DEFERRED_LOADS |   // performance optimization
        NativeDbgHelp.SYMOPT_UNDNAME            // C++ names are not mangled
        );

    // https://docs.microsoft.com/en-us/windows/win32/api/dbghelp/nf-dbghelp-syminitialize
    // search path for symbols:
    //   - The current working directory of the application
    //   - The _NT_SYMBOL_PATH environment variable
    //   - The _NT_ALTERNATE_SYMBOL_PATH environment variable
    //
    // passing false as last parameter means that we will need to call SymLoadModule64 
    // each time a module is loaded in the process
    return NativeDbgHelp.SymInitialize(hProcess, null, false);
}
private IntPtr BindToProcess(int pid)
{
    try
    {
        _process = Process.GetProcessById(pid);

        if (!SymInitialize(_process.Handle))
            return IntPtr.Zero;

        return _process.Handle;
    }
    catch (Exception x)
    {
        Console.WriteLine($"Error while binding pid #{pid} to DbgHelp:");
        Console.WriteLine(x.Message);
        return IntPtr.Zero;
    }
}

In the case of protected processes, Process.GetProcessById might throw an exception. The _hProcess field storing the process handle will be cleaned up in the IDisposible.Dispose implementation of the MethodStore:

1
2
3
4
5
6
7
8


public void Dispose()
{
    if (_hProcess == IntPtr.Zero)
        return;
    _hProcess = IntPtr.Zero;

    _process.Dispose();
}

After a process has been bound, each time one of its modules is loaded, SymLoadModule64 must be called. You can be notified of such a loaded module by enabling the Kernel provider with the ImageLoad keyword.

1
2
3
4
5


session.EnableKernelProvider(
    KernelTraceEventParser.Keywords.ImageLoad |
    KernelTraceEventParser.Keywords.Process,
    KernelTraceEventParser.Keywords.None
);

The handler attached to the ImageLoaded event will be called each time a dll gets loaded.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


private void SetupListeners(ETWTraceEventSource source)
{
    ...

    // get notified when a module is load to map the corresponding symbols
    source.Kernel.ImageLoad += OnImageLoad;
}

const int ERROR_SUCCESS = 0;
private void OnImageLoad(ImageLoadTraceData data)
{
    if (FilterOutEvent(data)) return;

    GetProcessMethods(data.ProcessID).AddModule(data.FileName, data.ImageBase, data.ImageSize);
}
public void AddModule(string filename, ulong baseOfDll, int sizeOfDll)
{
    var baseAddress = NativeDbgHelp.SymLoadModule64(_hProcess, IntPtr.Zero, filename, null, baseOfDll, (uint)sizeOfDll);
    if (baseAddress == 0)
    {
        // should work if the same module is added more than once
        if (Marshal.GetLastWin32Error() == ERROR_SUCCESS) return;

        Console.WriteLine($"SymLoadModule64 failed for {filename}");
    }
}

Now everything is in place to get a native function name from an address on the stack:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


private string GetNativeMethodName(ulong address)
{
    var symbol = new NativeDbgHelp.SYMBOL_INFO();
    symbol.MaxNameLen = 1024;
    symbol.SizeOfStruct = (uint)Marshal.SizeOf(symbol) - 1024;   // char buffer is not counted
    // the ANSI version of SymFromAddr is called so each character is 1 byte long

    if (NativeDbgHelp.SymFromAddr(_hProcess, address, out var displacement, ref symbol))
    {
        var buffer = new StringBuilder(symbol.Name.Length);

        // remove weird "$##" at the end of some symbols
        var pos = symbol.Name.LastIndexOf("$##");
        if (pos == -1)
            buffer.Append(symbol.Name);
        else
            buffer.Append(symbol.Name, 0, pos);

        // add offset if any
        if (displacement != 0)
            buffer.Append($"+0x{displacement}");

        return buffer.ToString();
    }

    // default value is the just the address in HEX
    return $"0x{address:x}";
}

Note that I needed to remove some unexpected $## strings are the end of some symbols.

This is the last episode of the series about building your own memory profiler in C#. In case you missed the first episodes, check them out on Medium:

Build your own .NET memory profiler in C# — call stacks (2/2–1) *This post explains how to get the call stack corresponding to the allocations with CLR events.*medium.com Build your own .NET memory profiler in C# *This post explains how to collect allocation details by writing your own memory profiler in C#.*medium.com

Resources

Source code available on Github.
Download Debugging Tools for Windows for dbghelp.dll and SymSrv.dll
Matt Pietrek article in MSDN Magazine about DBGHELP
Dbghelp samples -http://www.debuginfo.com/examples/dbghelpexamples.html

Join the crowd!

Careers at Criteo | Criteo jobs *Find opportunities everywhere. Choose your next challenge. Find the job opportunities at Criteo in Product, research &…*careers.criteo.com

Build your own .NET memory profiler in C# — call stacks (2/2–1)

Mon, 18 May 2020 12:07:01 +0000

In the previous episode of this series, you have seen how to get a sampling of .NET application allocations thanks to the AllocationTick and GCSampleObjectAllocation(High/Low) CLR events. However, this is often not enough to investigate unexpected memory consumption: you would need to know which part of the code is triggering the allocations. This post explains how to get the call stack corresponding to the allocations, again with CLR events.

Introduction

If you look carefully at the payload of the TraceEvent object mapped by Microsoft TraceEvent library (not my fault if they have the same name) for each CLR event, you won’t see anything related to a call stack. However, in the TraceEvent sample 41, the following line looks promising:

var callStack = data.CallStack();

with data being a TraceEvent object received for each CLR event!

This CallStack method is an extension method provided by the TraceLog special kind of event source. You might not have noticed but I have used it in the AllocationTick code sample from the previous post. This class (and many more helper classes) is doing a lot of work to :

“attach” a call stack to each CLR event; i.e. a list of addresses of assembly code
to translate addresses into string symbols (method names or full signatures), listen to a bunch of JIT related events for managed methods (more on this later), using COM-based Debug Interface Access (a.k.a. DIA) and MetadataReaderProvider** **for native functions

Notice that since events from all managed processes on the machine are handled by TraceLog, the internal cache for JITted methods description could consume a lot of memory. During my tests with two Visual Studio running, my test profiler consumed more than 500 MB before even handling call stacks. If you are in such an environment with multiple .NET processes, I will show how to “manually” get the same stacks (+ symbols in the next episode) with CLR events and a few methods from dbghelp.dll in a cheaper way.

The new provider (more on ClrRundown later), keywords and events need to be received to make all this work:

TraceLog: the easy way

As you have seen in the previous posts, the TraceEventSession class exposes a Source property of ETWTraceEventSource type. This source has event parsers properties from which you register handler methods that will be called when CLR events are received. Instead of directly using this source, you should wrap it with a TraceLogEventSource object that provides the same event parsers.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


await Task.Factory.StartNew(() =>
{
    using (_session)
    {
        SetupProviders(_session);

        using (TraceLogEventSource source = TraceLog.CreateFromTraceEventSession(_session))
        {
            SetupListeners(source);

            source.Process();
        }
    }
});

What’s new with providers?

The code for mySetupProviders method is a little bit different from the previous post even though no new event listeners are needed:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


private void SetupProviders(TraceEventSession session)
{
    // Note: the kernel provider MUST be the first provider to be enabled
    // If the kernel provider is not enabled, the callstacks for CLR events are still received 
    // but the symbols are not found (except for the application itself)
    // TraceEvent implementation details triggered when a module (image) is loaded
    session.EnableKernelProvider(
        KernelTraceEventParser.Keywords.ImageLoad |
        KernelTraceEventParser.Keywords.Process,
        KernelTraceEventParser.Keywords.None
    );

    session.EnableProvider(
        ClrTraceEventParser.ProviderGuid,
        TraceEventLevel.Verbose,    // this is needed in order to receive AllocationTick_V2 event
        (ulong)(
        // required to receive AllocationTick events
        ClrTraceEventParser.Keywords.GC |
        ClrTraceEventParser.Keywords.Jit |                      // Turning on JIT events is necessary to resolve JIT compiled code 
        ClrTraceEventParser.Keywords.JittedMethodILToNativeMap |// This is needed if you want line number information in the stacks
        ClrTraceEventParser.Keywords.Loader |                   // You must include loader events as well to resolve JIT compiled code.
        ClrTraceEventParser.Keywords.Stack
        )
    );

    // this provider will send events of already JITed methods
    session.EnableProvider(ClrRundownTraceEventParser.ProviderGuid, TraceEventLevel.Informational,
    (ulong)(
        ClrTraceEventParser.Keywords.Jit |              // We need JIT events to be rundown to resolve method names
        ClrTraceEventParser.Keywords.JittedMethodILToNativeMap | // This is needed if you want line number information in the stacks
        ClrTraceEventParser.Keywords.Loader |           // As well as the module load events.  
        ClrTraceEventParser.Keywords.StartEnumeration   // This indicates to do the rundown now (at enable time)
        ));

}

The kernel provider needs to be enabled with the ImageLoad and Process keywords in order to let TraceEvent detect when a process loads “images” (i.e. dlls) and at which address (needed to convert Relative Virtual Addresses (RVA) to addresses in the address space). Note that this provider must be enabled before any other provider or your code will trigger an exception.
The CLR provider needs to be enabled with Jit, JittedMethodILToNativeMap, and Loader (in addition to the usual GC one).
The Stack keyword has to be set on the same CLR provider to receive call stacks events for “normal” CLR event (more on this later)
The CLR Rundown provider is enabled with the same Jit, JittedMethodILToNativeMap, and Loader keywords. That way, JIT events corresponding to already JITted methods will be received (not only the new ones). This is important because otherwise, you won’t be able to map these methods with the address in memory of their JITted native code in the case of processes that have been started before the profiler. This is the case for my AllocationTickProfiler sample.

Callstacks and symbols

Now, when an AllocationTick event is received, calling the CallStack extension method on the GCAllocationTickTraceData argument returns a TraceCallStack object. This class is a linked list of TraceCodeAddress representing each stack frame (i.e. address in assembly code). These classes are at the heart of TraceEvent and Perfview callstack management. The method names and signatures are retrieved behind the scene thanks to JIT events and the SymbolReader class that digs into .pdb files.

You first need to initialize a SymbolReader instance:

Set the path to find the .pdb; including the Microsoft HTTP endpoint for public .NET versions symbols,
Allow pdb next to the executable to be loaded.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


// By default a symbol Reader uses whatever is in the _NT_SYMBOL_PATH variable.  However you can override
// if you wish by passing it to the SymbolReader constructor.  Since we want this to work even if you 
// have not set an _NT_SYMBOL_PATH, so we add the Microsoft default symbol server path to be sure/
var symbolPath = new SymbolPath(SymbolPath.SymbolPathFromEnvironment).Add(SymbolPath.MicrosoftSymbolServerPath);
_symbolReader = new SymbolReader(_symbolLookupMessages, symbolPath.ToString());

// By default the symbol reader will NOT read PDBs from 'unsafe' locations (like next to the EXE)  
// because hackers might make malicious PDBs. If you wish ignore this threat, you can override this
// check to always return 'true' for checking that a PDB is 'safe'.  
_symbolReader.SecurityCheck = (path => true);

Then, displaying a TraceCallStack from a received CLR event in a human-readable format is simple:

Get one frame after the other from the linked list,
If the CodeAddress field is not cached yet, load the symbols for its module,
Display the FullMethodName field of the frame (or the address if not found).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


private void DumpStack(TraceCallStack callStack)
{
    while (callStack != null)
    {
        var codeAddress = callStack.CodeAddress;
        if (codeAddress.Method == null)
        {
            var moduleFile = codeAddress.ModuleFile;
            if (moduleFile == null)
            {
                Debug.WriteLine($"Could not find module for Address 0x{codeAddress.Address:x}");
            }
            else
            {
                codeAddress.CodeAddresses.LookupSymbolsForModule(_symbolReader, moduleFile);
            }
        }
        if (!string.IsNullOrEmpty(codeAddress.FullMethodName))
            Console.WriteLine($"     {codeAddress.FullMethodName}");
        else
            Console.WriteLine($"     0x{codeAddress.Address:x}");
        callStack = callStack.Caller;
    }
}

Note that the first frame in the linked list is the last on the stack (i.e. last executed method).

As I mentioned at the beginning of the post, I have been facing OutOfMemory errors due to the TraceEvent symbols management large memory usage when a few other .NET applications were running. Let’s see how to get the call stacks in a less memory consuming way.

Manually rebuilding the allocations call stack

Instead of using the call stack and symbol management provided by TraceLog in TraceEvent, I would prefer to manually get them. If you remember the last post, thanks to GCSampledObjectAllocation CLR events, it is possible to have a sampling of the allocation size and count per process and per type. What I would like to add to the type picture is the list of call stacks leading to these allocations.

How to manually get CLR events call stack

The first step is to understand how to get the CLR events call stacks. If you use the TraceLog-based code just presented, you should see the following kind of call stack:

The ETWCallout CLR helper function is in charge of sending a special event containing the call stack of other normal events from the four supported CLR providers. If you set the Stack keyword to the CLR provider, each time an event is sent by a thread, a ClrStackWalk event will be sent just after. It means after each SampleObjectAllocation event, a ClrStackWalk event containing the call stack will be immediately received. In fact, since an application will probably be using more than one thread, it is required to do the mapping between the two events on a per-thread basis.

Each allocation event received by the OnSampleObjectAllocation handler contains the ThreadID property so it is easy to keep track of the last received allocation event per thread. In my case, the ProcessAllocations class stores this information in its _perThreadLastAllocation field:

1
2
3
4
5


public class ProcessAllocations
{
    ...
    private readonly Dictionary<string, AllocationInfo> _allocations;
    private readonly Dictionary<int, AllocationInfo> _perThreadLastAllocation;

Now, each time a SampleObjectAllocation event is received, the id of the sending thread is passed to the updatedProcessAllocations.AddAllocation() method:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


public AllocationInfo AddAllocation(int pid, ulong size, ulong count, string typeName)
{
    if (!_allocations.TryGetValue(typeName, out var info))
    {
        info = new AllocationInfo(typeName);
        _allocations[typeName] = info;
    }

    info.AddAllocation(size, count);

    // the last allocation is still here without the corresponding stack
    if (_perThreadLastAllocation.TryGetValue(pid, out var lastAlloc))
    {
        Console.WriteLine("no stack for the last allocation");
    }

    // keep track of the allocation for the given thread
    // --> will be used when the corresponding call stack event will be received
    _perThreadLastAllocation[pid] = info;

    return info;
}

The _perThreadLastAllocation dictionary stores the AllocationInfo per thread. If an allocation happens, it is added into the dictionary. When a ClrStackWalk event is received for a given thread, the stack will be associated with the last AllocationInfo and removed from the dictionary. If some events are missed (it never happens during my tests but who knows), error message could be logged.

The ClrStackWalkTraceData argument received by the ClrStackWalk listener has a FrameCount property that returns the number of frames in the call stack. In addition, its InstructionPointer() method takes a frame position in the stack (starting at 0) and returns the address (in assembly code) at this position on the call stack.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


private void OnClrStackWalk(ClrStackWalkTraceData data)
{
    if (FilterOutEvent(data)) return;

    var callstack = BuildCallStack(data);
    GetProcessAllocations(data.ProcessID).AddStack(data.ThreadID, callstack);
}
private AddressStack BuildCallStack(ClrStackWalkTraceData data)
{
    var length = data.FrameCount;
    AddressStack stack = new AddressStack(length);

    // frame 0 is the last frame of the stack (i.e. last called method)
    for (int i = 0; i < length; i++)
    {
        stack.AddFrame(data.InstructionPointer(i));
    }

    return stack;
}

The AddressStack class returned by BuildCallStack stores the frames as a list of addresses so it can be stored in AllocationInfo.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36


public class AddressStack
{
    // the first frame is the address of the last called method 
    private readonly List<ulong> _stack;

    public AddressStack(int capacity)
    {
        _stack = new List<ulong>(capacity);
    }

    // No need to override GetHashCode because we don't want to use it as a key in a dictionary
    public override bool Equals(object obj)
    {
        if (obj == null) return false;

        var stack = obj as AddressStack;
        if (stack == null) return false;

        var frameCount = _stack.Count;
        if (frameCount != stack._stack.Count) return false;

        for (int i = 0; i < frameCount; i++)
        {
            if (_stack[i] != stack._stack[i]) return false;
        }

        return true;
    }

    public IReadOnlyList<ulong> Stack => _stack; 

    public void AddFrame(ulong address)
    {
        _stack.Add(address);
    }
}

This class overrides the Equals method for a single reason: I want to be able to detect when the “same” stack (i.e. with the exact same frame addresses) is received for a given type allocation. That way, I just need to keep a counter for each different AddressStack and not all call stacks in AllocationInfo. Remember that AllocationInfo is used to keep track of allocations per type:

1
2
3
4
5
6


public class AllocationInfo
{
    private readonly string _typeName;
    private ulong _size;
    private ulong _count;
    private List<StackInfo> _stacks;

The StackInfo class contains an AddressStack and how many times it led to this type of allocation.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


public class StackInfo
{
    private readonly AddressStack _stack;
    public ulong Count;

    internal StackInfo(AddressStack stack)
    {
        Count = 0;
        _stack = stack;
    }

    public AddressStack Stack => _stack;
}

So, when a stack event is received, AddStack is called on the last AllocationInfo for the same thread:

1
2
3
4
5
6
7
8
9


public void AddStack(int tid, AddressStack stack)
{
    if (_perThreadLastAllocation.TryGetValue(tid, out var lastAlloc))
    {
        lastAlloc.AddStack(stack);
        _perThreadLastAllocation.Remove(tid);
        return;
    }
}

The job of AllocationInfo.AddStack() the method is to check if a previous allocation was made with the same call stack (hence the Equals override). If this is the case, just increment the corresponding StackInfo count. Otherwise, create a new StackInfo for this call stack with a count set to 1.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


internal void AddStack(AddressStack stack)
{
    var info = GetInfo(stack);
    if (info == null)
    {
        info = new StackInfo(stack);
        _stacks.Add(info);
    }

    info.Count++;
}

private StackInfo GetInfo(AddressStack stack)
{
    for (int i = 0; i < _stacks.Count; i++)
    {
        var info = _stacks[i];
        if (stack.Equals(info.Stack)) return info;
    }

    return null;
}

Knowing the address in code of each frame for all events call stack is nice but it would be much more useful to translate them into method names… You have to deal with two different cases: managed and native methods. I will cover these topics in the next episode.

Resources

Source code available on Github.
TraceEvent sample 41 source code.

Missed the first part of this story? Check this out:

Build your own .NET memory profiler in C# *This post explains how to collect allocation details by writing your own memory profiler in C#.*medium.com

Interested in joining our journey? Check this out:

Product, Research & Development | Criteo Careers careers.criteo.com

Debugging Wednesday at Criteo — Cancel this task!

Fri, 21 Feb 2020 08:18:08 +0000

Introduction

Last Wednesday was a great day for Kevin and myself: We spent a lot of time investigating the reasons why a test was failing. Let’s share with you these frustrating but interesting minutes.

One of our colleagues came to us because an integration test would get stuck in some specific conditions. Here is the simplified code of the service that is supposed to do some background processing until it is stopped:

1
2
3
4
5
6
7
8


public class Service
{
    private CancellationTokenSource _cancellationSource;
    private Task _backgroundProcessing;
    private Task _cleanup;

    ...
}

In the test, the service is created: Two Task instances are created in the constructor (for background processing irrelevant for this discussion) that (1) are started when the service starts and (2) are canceled when the service is stopped. A CancellationTokenSource is used to cancel the tasks if needed.

1
2
3
4
5
6
7
8
9


public Service()
{
    _cancellationSource = new CancellationTokenSource();
    var cancellationToken = _cancellationSource.Token;
    _backgroundProcessing = new Task(DoStuffInTheBackground,
        cancellationToken, TaskCreationOptions.LongRunning);
    _cleanup = new Task(DoStuffInTheBackground, 
        cancellationToken, TaskCreationOptions.LongRunning);
}

However, in the specific conditions of this test, the Start method would never be called, skipping straight to the Stop method.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


public void Start()
{
    _backgroundProcessing.Start();
    _cleanup.Start();
}

public async Task Stop()
{
    _cancellationSource.Cancel();
    await _backgroundProcessing;
    await _cleanup;
}

The task is never started because the test skips the Service.Start() method, but it should transition to “Cancelled” state as soon as the CancellationTokenSource associated with the CancellationToken passed to the Task gets canceled. In that particular situation, we expect the await _backgroundProcessing code to immediately return in the Stop() method.

In our colleague Visual Studio, the debugger never came back from await _backgroundProcessing, indicating that the task never completed…

Reproduce the problem

I wanted to un-validate any side effect related to our test framework or custom Criteo libraries and confirm my understanding of task cancellation, so I wrote a small Console application with the same 4.7.2 version of .NET Framework with the following code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


static async Task ReproduceWorking()
{
    var source = new CancellationTokenSource();
    var token = source.Token;
    var t = new Task(
        () => Console.WriteLine("Done"), token, TaskCreationOptions.LongRunning);

    source.Cancel();

    try
    {
        await t;
    }
    catch (TaskCanceledException x)
    {
        Console.WriteLine(x.ToString());
    }
}

And guess what? I got the expected TaskCancellationException:

System.Threading.Tasks.TaskCanceledException: A task was canceled. at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.GetResult() at TaskCancellation.Program.d__2.MoveNext()

The next step was to double-check the understanding of how tasks are working related to cancellation. So I set a breakpoint after the task is instantiated

And its status is Created.

Doing the same after Cancel() is called on the CancellationTokenSource, gives the expected Canceled status.

So, let’s do the same when debugging the test code:

It gives the same Created status after the task creation but after canceling the source:

…the status does not switch to Canceled like in my Console repro!

Looking for cancellation

The only thing that came to my mind was: maybe the cancellation token is not taken into account. However, a CancellationToken is just a struct that keeps a reference to its CancellationTokenSource. It should be easy in Visual Studio debugger to double-check that our task keeps track of the token somewhere and goes back to the cancellation source. Well… the token is not kept as a field of the task but stored inside the m_contingentProperties field deep during the task construction code path.

Let’s look at the value in the Quick Watch after the cancellation source gets canceled:

It sounds like the cancellation token is not canceled… But if we look at the source Token property of our canceled CancellationTokenSource,

we don’t have the same value for IsCancellationRequested!

It’s like we don’t look at the same cancellation source… To find out, we just have to compare the reference to our CancellationTokenSource with the one we see in the m_contingentProperties token of the task. To achieve that, we could copy the expression from QuickWatch, paste it into the Debug | Windows | Memory… pane and press ENTER to get the address where the object is stored in memory

After having done the same with _cancellationSource, I did not get the same address:

It means that we were dealing with two different instances of CancellationTokenSource.

But this might be too C++ish for you… Kevin prefers leveraging the Make Object ID feature of the C# debugger. You simply right-click the Data Tip of the cancellation source and select Make Object ID:

Once this is done, a numeric identifier is displayed for this instance in Data Tip and any Watch window (#1 in this screenshot):

When we looked at the cancellation source of the token stored by the task,

we didn’t see any ID so it was not the same object.

So we decided to use Make Object ID on this m_source that became marked as #2.

And when we looked at the m_source of the second cleanup Task, we realized that it was the same object but not the one we created!

We started to think that we were becoming crazy. So, let’s restart from the beginning and follow the CancellationTokenSource from its creation because we are sure that we passed a valid cancellation token (linked to this source) to the task constructor. Or… Did we? The QuickWatch gives a different answer just after the task gets created compared to what we’ve seen already: a token with an empty source property now!

I closed the QuickWatch pane and reopened it for Kevin to confirm. And like in a nightmare, the source was not null anymore…

Visual Studio must be responsible for that weird behavior!

Kevin remembered the ”Enable property evaluation” settings in the Options dialog. If it is checked (which is the default), it means that the Debugger would fetch the value of an instance field and then call the Getter of each property in order to display its value.

However, if you uncheck it, only the fields are displayed. So in our case, we then always got a null m_contingentProperties field (and, as expected, all property would not be displayed):

The m_contingentProperties is initialized in EnsureContingentPropertiesInitialized() when called by AssignCancellationToken() from the TaskContructorCore() helper used by the Task constructor but it did not seem to be the case because it was definitively null…

Kevin decided to stop at the CancellationTokenSource constructor with a new breakpoint (more on how to set a breakpoint on a .NET Framework method soon) to see where the one shown in the Debugger was created but the breakpoint was never hit. So the CancellationTokenSource #2 must have been created even before our own was created by our code. In fact, a static CancellationTokenSource is created and is set to m_source when InitializeDefaultSource() gets called by one of the Getter. This explains why we saw the same instance #2 in both tasks token.

To sum up, we were now sure that the passed token was not “received” by the Task.

Eureka!

Maybe there is a magic trick done by the .NET Framework to lazily set the token source after the creation of the task. However, we did not find such a code in the .NET Framework and this is not what we see in our repro.

Back to the basics: are we sure that we are executing the code we think is executed? We looked for mscorlib in the Debug | Windows | Modules pane,

and we opened it with a decompiler: the code of the methods called during the Task** **construction was the same as the one shown in https://referencesource.microsoft.com/#mscorlib/system/threading/Tasks/Task.cs.

Next, in order to better follow the execution and the passing of parameters (including our token), we decided to set breakpoints on Task private method responsible for its initialization.

internal void TaskConstructorCore(object action, object state, CancellationToken cancellationToken, TaskCreationOptions creationOptions, InternalTaskOptions internalOptions, TaskScheduler scheduler)

In the Debug | Windows | Breakpoints pane, click New | Function Breakpoint…

and type the full name of the method. This is working even for a method of a class defined in the .NET Framework assembly for which you do not have the source code:

We checked that the breakpoints were well set (i.e. no typo in the full name) by looking at the filled red circle:

The cancellationToken parameter should contain the token that we passed at Task creation. Unfortunately, the QuickWatch pane displayed a “cannot read memory” error that we never saw in Visual Studio before!

At that time, we thought we were doomed but we looked at the Call Stack pane and we realized that the code was calling the wrong Task constructor:

public Task(Action action, object state, TaskCreationOptions creationOptions)

Its signature is compatible with our code:

_backgroundProcessing = new Task(DoStuffInTheBackground, cancellationToken, TaskCreationOptions.LongRunning);

and this is why the compiler did not complain.

Our CancellationToken was passed as the state parameter is given directly to our DoStuffInTheBackground Action: the created Task had no idea that it was supposed to be its CancellationToken.

Note that if we had noticed the Auto Completion (Ctrl + Shift + Space) hint, we might have figured out the root cause much sooner…

The fix was straightforward; just using the right constructor:

public Task(Action action, object state, CancellationToken cancellationToken, TaskCreationOptions creationOptions)

that accepts both a state for the callback and a CancellationToken for the Task to create:

_backgroundProcessing = new Task(DoStuffInTheBackground, cancellationToken, cancellationToken, TaskCreationOptions.LongRunning);

Under the debugger, we validated that the source was now the expected one:

and the test did not hang anymore.

If, from the beginning, we would have been able to step into .NET Framework compiled code as we do with Jetbrains Resharper integration in Visual Studio, we would have found the issue almost immediately. Thankfully, Microsoft has just announced decompilation of C# code made easy with Visual Studio.

We wish we had it last Wednesday…

Interested in reading more about Christophe’s & Kevin’s work? Check out their latest articles:

Build your own .NET memory profiler in C# *This post explains how to collect allocation details by writing your own memory profiler in C#.*medium.com Switching back to the UI thread in WPF/UWP, in modern C# Leveraging the async machinery to transparently switch to the UI thread when neededmedium.com

If you are looking for a change and would love to work with these two, head over to our careers page and let us know if there is something that sounds like you!

Product, Research & Development | Criteo Careers *Come and meet our teams …*careers.criteo.com

Getting another view on thread stacks with ClrMD

Tue, 31 Dec 2019 10:52:00 +0000

This post of the series details how to look into your threads stack with ClrMD.

Introduction

It’s been a long time (see the resources at the end) since I’ve been discussing what ClrMD could bring to .NET developers/DevOps! My colleague Kevin just wrote an article about how to emulate SOS DumpStackObjects command both on Windows and Linux with ClrMD. This implementation lists the objects on the stack but without their values (like strings content for example) nor the stack frames corresponding to the method calls.

The rest of the post will show you, with ClrMD, how to get an higher view, closer to what the SOS ClrStack command could provide.

Let’s take this simple application as an example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


class Program
{
    static void Main(string[] args)
    {
        Host h = new Host();
        h.Base();
    }
}

class Host
{
    public void Base()
    {
        int iValue = Guid.NewGuid().GetHashCode();
        bool bValue = (iValue % 2) == 0;
        string parameter = Guid.NewGuid().ToString().Substring(0,1) + "_1234567890";

        Console.WriteLine(parameter + " = " + First(iValue, bValue, parameter));
    }

    private int First(int iValue, bool bValue, string parameter)
    {
        Guid guid = Guid.NewGuid();
        return Second(bValue, guid) / 2;
    }

    private int Second(bool bValue, Guid guid)
    {
        return Third(guid).Length;
    }

    private string Third(Guid guid)
    {
        Console.WriteLine($"call   procdump -ma {Process.GetCurrentProcess().Id}");
        Console.ReadLine();
        return $"{guid.GetHashCode()}#{guid.GetHashCode()}#{guid.GetHashCode()}";
    }
}

As you can see, I’ve mixed value and reference types as parameters and local variables up to the call to the Third method that displays the procdump command line to execute in order to generate a memory dump of the process.

Use WinDBG + SOS Luke!

When you open it with WinDBG and load SOS, here is the result of the dso command:

The clrstack command shows the stacked method calls:

And if you use the -a parameter, you will get methods with their parameters and local variables (or -p for parameters only and -l for local variables only):

It is weird that SOS implementation does not give the type of both the parameters and locals. But wait! While researching for this post, I looked at the SOS implementation (now in the strike.cs file moved from the coreclr to the diagnostics repository) to find this nice comment:

So I tried clrstack with -i and I got the types for parameters (and locals unlike what the comments implies):

Even though clrstack supports the -all flag to dump the call stack of all managed threads, you might need to do your own automatic analysis on hundreds of threads and this is where ClrMD shines.

Merging methods and parameters/locals

When I read Kevin’s post, I immediately thought about adding the method call on the stack based on the work I’ve done in March 2019 to implement the pstacks tool. At that time, my goal was to aggregate the call stacks of a large number of threads in order to find out pattern of blocked threads, sharing the “same” call stacks. Visual Studio provides a great “Parallel Stacks” pane but I needed it for both Windows and Linux.

To list all the call stack with ClrMD, you simply enumerate the managed threads and for each one, its StackTrace property contains the list of StackFrame objects corresponding to each method call.

The StackPointer property of each frame contains the address of the frame in the call stack, allowing a mapping of the method call with its parameters and locals:

As always with stacks, lower addresses correspond to the last things added to the stack (i.e. last called method). While checking between what is shown by SOS, the parameters/locals addresses and frame stack pointers, you realize that all objects at an address in the stack equal or below the StackPointer of a frame are either parameters or local variables of the frame method.

Even better, for non static method, you can guess what is the this* *implicit parameter if the address is the same as the frame StackPointer; shown with the green = sign in the previous screenshot and prefixed by > in Kevin’s updated code that merges the method calls to the parameters and locals:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46


for (ulong ptr = stackTop; ptr <= stackLimit; ptr += (ulong)runtime.PointerSize)
{
    // look for the frame corresponding to the current position in the stack
    if (currentFrame.StackPointer <= ptr)
    {
        Console.WriteLine(FormatFrame(currentFrame));

        nFrame++;
        if (nFrame < frames.Count)
        {
            currentFrame = frames[nFrame];
        }
        else
        {
            break;
        }
    }

    ulong obj;
    if (!runtime.ReadPointer(ptr, out obj))
    {
        break;
    }

    if (!IsInHeap(heap, obj))
    {
        continue;
    }

    var type = heap.GetObjectType(obj);
    if (type == null || type.IsFree)
    {
        continue;
    }

    // try to find implicit "this" parameter in case of non-static method
    var myFrame = (nFrame == 0) ? currentFrame : frames[nFrame-1];
    separator = ((myFrame.StackPointer == ptr) &&
                 (myFrame.Method != null) &&
                 (!myFrame.Method.IsStatic))
        ? ">" 
        : " ";
    Console.Write($"{ptr:x16}   {separator} ");
    DumpObject(heap, type, (ulong)obj);

}

The FormatFrame helper method simply prefix static methods with # instead of | for instance methods:

Unfortunately, I did not find any way with ClrMD to make the difference between parameters and locals. Based on what you can see in SOS implementation of this part of the clrstack command, it relies on the EnumerateArguments and EnumetateLocalVariables methods of ICorDebugILFrame which is not exposed by ClrMD. There is another undocumented implementation based on private interfaces I could not leverage neither. For a larger discussion around stack walking in .NET, read this great post by Matt Warren.

Also, without any explicit access to specific parameter or local, I did not find a way to get the value of primitive and value type instances stored on the stack. However, it is still possible to get them for boxed ones and reference type instances such as string for example.

Getting instances from the stack

In the last code excerpt, I did not describe the DumpObject helper method used to display an object on the stack. The implementation provided by Kevin was used to show the address and the type of the object:

1

Console.WriteLine($"{objAddress:x16} {type.Name}");

The next step would be to display value for primitive types such as numbers, boolean, string and even array size:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


private static void DumpObject(ClrHeap heap, ClrType type, ulong objAddress)
{
    // get value for simple types
    string valueOrAddress = 
        (type.Name == "System.Char") ? $"{type.GetValue(objAddress),16}" :
        (type.Name == "System.String") ? $"{FormatString(type.GetValue(objAddress).ToString())}" :
        (type.Name == "System.Bool") ? $"{FormatString(((bool)type.GetValue(objAddress)).ToString())}" :
        (type.Name == "System.Byte") ? $"{FormatString(((byte)type.GetValue(objAddress)).ToString())}" :
        (type.Name == "System.SByte") ? $"{FormatString(((sbyte)type.GetValue(objAddress)).ToString())}" :
        (type.Name == "System.Decimal") ? $"{FormatString(((decimal)type.GetValue(objAddress)).ToString())}" :
        (type.Name == "System.Double") ? $"{FormatString(((double)type.GetValue(objAddress)).ToString())}" :
        (type.Name == "System.Single") ? $"{FormatString(((float)type.GetValue(objAddress)).ToString())}" :
        (type.Name == "System.Int32") ? $"{FormatString(((int)type.GetValue(objAddress)).ToString())}" :
        (type.Name == "System.SInt32") ? $"{FormatString(((uint)type.GetValue(objAddress)).ToString())}" :
        (type.Name == "System.Int64") ? $"{FormatString(((long)type.GetValue(objAddress)).ToString())}" :
        (type.Name == "System.SInt64") ? $"{FormatString(((ulong)type.GetValue(objAddress)).ToString())}" :
        (type.IsArray) ? $"{FormatString(GetArrayAsString(type, objAddress))}" :
        $"{objAddress:x16}";  // work also for IntPtr
    Console.WriteLine($"{valueOrAddress} {type.Name}");
}

Most of this code is based on the GetValue helper from ClrType: it returns the right “thing” for simple types. Look at ClrMD implementation details to get a better understanding of how the value is rebuilt.

The GetArrayAsString simply returns the number of elements in the array:

1
2
3
4
5


private static string GetArrayAsString(ClrType type, ulong objAddress)
{
    var elementCount = type.GetArrayLength(objAddress);
    return $"length = {elementCount}";
}

And the call stack is now complete!

Note that you may even get more locals or parameters than with WinDBG+SOS but don’t ask me why…

For more advanced object formatting cases such as dumping structs or enumerating fields and their value, I would highly recommend to look at the related ClrMD documentation page (just replace GCHeapType by ClrType and you’ll be safe).

Resources

Dumping stack objects with ClrMD by Kevin Gosse
Stack walking in the .NET Runtime by Matt Warren
Part 1: Bootstrap ClrMD to load a dump.
Part 2: Find duplicated strings with ClrMD heap traversing.
Part 3: List timers by following static fields links.
Part 4: Identify timers callback and other properties.
Part 5: Use ClrMD to extend SOS in WinDBG.
Part 6: Manipulate memory structures like real objects.
Part 7: Manipulate nested structs using dynamic.
Part 8: Spelunking inside the .NET Thread Pool
Part 9: Deciphering Tasks and Thread Pool items

WSL + Visual Studio = attaching/launching a Linux .NET Core application on my Window 10

Thu, 21 Nov 2019 15:57:01 +0000

This post shows how to attach to a .NET Core process running on Linux with WSL and also how to start a Linux process with Visual Studio debugger

Coming from the Windows world, I don’t find that easy to develop .NET Core applications for Linux. I’m used to code and debug in Visual Studio. Now, I need to build on Windows (due to our Criteo continuous integration), deploy an artifact to Marathon in order to get an application running inside a Mesos container. At Criteo, we had to build a whole set of services to allow remote debugging or memory dump analysis.

But what if I just wanted to test and debug a small scenario on my beloved Windows machine? Windows Subsystem for Linux (aka WSL) is perfect for running a Linux .NET Core application on Windows. However, how to attach to it or even start a debugging session from Visual Studio? In the rest of this post, I’ll explain how to setup your Windows 10 machine to achieve these miracles.

Linux on Windows: welcome WSL

It is obviously not the place to dig into Windows Subsystem for Linux. You just need to know that once installed, it allows you open up a Linux shell on your Windows machine without any virtual machine kind of technology. It is also possible to share folders between Windows (where you want to build your application) and Linux (where you want to execute the built assemblies).

The first step is to turn on WSL feature in your Windows:

After a reboot, you are able to start a WSL prompt:

You can also install your favorite Linux distro if you want.

The next step is to install .NET Core runtime or SDK so you are able to run your application on Linux.

It is now time to look at your hard drive from a Linux perspective:

Your drives are mapped under /mnt without the “:”. In my case, I created a wsl folder under my d: drive to do my experiments. With Visual Studio, I generated a TestConsole application with the following code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


using System;
 
namespace TestConsole
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Enter x to EXIT...");
            while(true)
            {
                var cmd = Console.ReadLine();
                if (cmd.ToLower() == "x")
                    return;
 
                Console.WriteLine($"> {cmd}");
            }
        }
    }
}

I’m publishing the application to get all needed assemblies:

In a WSL prompt, type dotnet with the name of your application assembly et voila!

A Linux .NET Core application is running on your Windows machine.

How to attach to a running Linux application

Let’s say that I’m detecting a problem and I want to debug the Linux application with Visual Studio. When attaching the Visual Studio debugger to a process, several connection types are available:

The SSH connection type will be used with WSL with the following kind of communications architecture:

The Visual Studio debugger is sending commands to the remote Linux debugger vsdbg via an SSH channel. Here are the steps to follow to install the missing components:

By default, an SSH server is installed with WSL. However, I was not able to make the whole pipeline work with it so I had to uninstall and reinstall it: sudo apt-get remove openssh-server sudo apt-get install openssh-server
The SSH configuration needs also to be changed in order to allow username/password kind of security needed by Visual Studio (if you prefer key-based security, look at the end of the post for available resources). If you don’t know how to use vi efficiently to simply edit a file, install nano (thanks @kookiz for the tips :^) sudo apt-get install nano
In /etc/ssh/sshd_config, change the PasswordAuthentication settings sudo nano /etc/ssh/sshd_config PasswordAuthentication yes
Restart the ssh server sudo service ssh start
You need to install unzip in order to get vsdbg sudo apt-get install unzip curl -sSL https://aka.ms/getvsdbgsh | bash /dev/stdin -v latest -l ~/vsdbg

You are now ready to select SSH as connection type and enter your machine name before clicking the Refresh button. At that time, a new dialog should pop up for you to enter your WSL credentials:

After you click the Refresh button, the list at the bottom should contain the Linux processes running in WSL

Select your .NET Core application and click Attach to select the Managed debugger:

Now, if you set a breakpoint in the code and trigger it with an appropriate action in your WSL prompt

then the debugger will break as expected:

Note that when you stop your debugging session, the Linux application is not stopped; just detached from the debugger and keeps on running.

Showtime for F5!

Attaching to a running Linux application is nice but it would be even better to start a Linux process from Visual Studio debugger. In order to achieve this goal you need to add another piece to the architecture puzzle:

As detailed in this WIKI page, it is possible to tell Visual Studio to execute debugging actions thanks to a launch.json file such as the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


{
  "version": "0.2.0",
  "adapter": "c:\\tools\\plink.exe",
  "adapterArgs": "-ssh -pw  chrisnas@DBGDQPZ1 -batch -T ~/vsdbg/vsdbg --interpreter=vscode",
  "configurations": [
    {
      "name": ".NET Core Launch",
      "type": "coreclr",
      "cwd": "/mnt/d/wsl/TestConsole/TestConsole/bin/Debug/netcoreapp2.1/publish",
      "program": "TestConsole.dll",
      "request": "launch"
    }
  ]
}

The plink tool from putty will be used as an adapter for Visual Studio to communicate with vsdbg running in WSL. The adapterArgs property gives the same SSH/machine/user/password information that you provide via Visual Studio UI in the Attach scenario. The configuration section defines which request (“launch” instead of attach and which folder/assembly to start) will be sent to vsdbg.

Once this file is created (in my case in d:\wsl\vs folder), you just need to type the following command in the Immediate pane of Visual Studio:

DebugAdapterHost.Launch /LaunchJson:d:\wsl\vs\launch.json

and if you had set a breakpoint on the first line of your application, the debugger should break there:

In a WSL prompt, you can see the expected 2 new spawned processes:

But wait!

I have a problem now: I don’t have any prompt in which typing input for my console application… However, as always in Linux, you simply need to write to a file to fix this. The stdin stream of your application is accessible under /proc//fd/0.

So, when I type the following command:

echo "Launching a Linux app is not a problem!" > /proc/18341/fd/0

my breakpoint is hit in Visual Studio:

Also note that everything that is sent to the console appears in Visual Studio Output pane:

Note that, unlike the Attach scenario, if you stop the debugging session, the Linux application (and vsdbg) will be terminated.

Resources

During my investigations I’ve found a few resources you might find useful (especially about configuring SSL with keys instead of passing clear user/password)

Basic VS + WSL
Good description about how to use VS 2017 to attach to .NET Core app running in WSL
Debugging .NET Core from VS 2017 and WSL
VS/C++ with WSL (describe how to install WSL and setup open ssh server)
Setup SSH on WSL
Wiki for VSDBG
Great description about how to setup your linux dev environment with VS Code and WSL

How to expose your custom counters in .NET Core

Thu, 17 Oct 2019 12:42:17 +0000

This post of the series explains how to implement your own counters.

Part 1: Replace .NET performance counters by CLR event tracing.

Part 2: Grab ETW Session, Providers and Events.

Part 3: CLR Threading events with TraceEvent.

Part 4: Spying on .NET Garbage Collector with TraceEvent.

Part 5: Building your own Java GC logs in .NET

Part 6: Spying on .NET Core Garbage Collector with .NET Core EventPipes

Part 7: .NET Core Counters internals: how to integrate counters in your monitoring pipeline

Introduction

The** EventPipe** counters are the .NET Core replacement for Windows performance counters. In the previous post, I’ve explained how to listen to CLR event pipes to get the counter’s value over time both on Windows and Linux. This post shows you how easy it is to provide your counters via the same infrastructure.

The example I’m using is based on a real-world case we had to investigate at Criteo. We needed to correlate request duration with garbage collections, so we decided to add new metrics to our testing dashboard: number and duration of requests but split between those processed without being interrupted by a GC and the others.

For the sake of the ASP.NET Core code example, a dedicated middleware is created: it simply measures the time spent to process a request and if the count of garbage collections has changed before and after the request is processed:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51


public class RequestMetricsMiddleware
{
    private readonly RequestDelegate _next;

    public RequestMetricsMiddleware(RequestDelegate next)
    {
        _next = next;
    }

    public async Task InvokeAsync(HttpContext context)
    {
        // get the count of GCs before processing the request
        var collectionCountBeforeProcessingTheRequest = GetCurrentCollectionCount();

        var sw = Stopwatch.StartNew();

        try
        {
            // Call the next delegate/middleware in the pipeline
            await _next(context);
        }
        finally
        {
            // compare the counter of GCs after processing the request
            // if the count changed, a garbage collection occurred during the processing 
            // and might have slowed it down and maybe reaching SLA limit: this could 
            // explain 9x-percentile in slow requests for example
            if (GetCurrentCollectionCount() - collectionCountBeforeProcessingTheRequest != 0)
            {
                // update with collection metric
               RequestCountersEventSource.Instance.AddRequestWithGcDuration(sw.ElapsedMilliseconds);
            }
            else
            {
                // update without collection metric
                RequestCountersEventSource.Instance.AddRequestWithoutGcDuration(sw.ElapsedMilliseconds);
            }
        }
    }

    private int GetCurrentCollectionCount()
    {
        int count = 0;
        for (int i = 0; i < GC.MaxGeneration; i++)
        {
            count += GC.CollectionCount(i);
        }

        return count;
    }
}

The interesting part is in the RequestCountersEventSource implementation.

Use an EventSource Luke!

As explained in the previous post, an EventSource instance is used as the “server” part of the EventPipe communication channel. It exposes a name that is used to identify it, but more important to listen to it with dotnet-trace, dotnet-counters, or your own listener as the provider name.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


[EventSource(Name = RequestCountersEventSource.SourceName)]
public class RequestCountersEventSource : EventSource
{
    // this name will be used as "provider" name with dotnet-counters
    // ex: dotnet-counters monitor -p  Sample.RequestCounters
    //
    const string SourceName = "Sample.RequestCounters";

    public RequestCountersEventSource() 
        : base(RequestCountersEventSource.SourceName, EventSourceSettings.EtwSelfDescribingEventFormat)
    {
        // create the counters: they'll be bound to this event source + CounterGroup
        CreateCounters();
    }

This name is exposed via an EventSourceAttribute that decorates your EventSource-derived class (you could also pass it to the constructor). The counters are created in the constructor through the CreateCounters helper.

Pick the right Counter class

Before looking at the implementation of the CreateCounters method, you need to understand what kind of counters are available for you. In the previous post, I mentioned that the CLR was using Mean (that provides mean, max, and min values over the update interval) and Sum (to increment a single value) kinds of counters. Note that dotnet-counter will only show the mean value for Mean counters. In addition, the counters could either automatically poll the value from a callback (the method used by the CLR today), or your code could change a counter value by calling the WriteMetric method. The EventCounter class provides this helper and it does its best to compute the min/max/mean in a lock-free way.

The next question to answer is which one should you use.

In the case of the request with(out) GC example, I want to expose different metrics:

Request count: a PollingCounter will be used in addition to an int field incremented when a request is received.
Request count delta: an IncrementingCounter associated with the same int value will provide the delta (i.e., number of requests processed during an interval)
Request with GC and without GC counts: two PollingCounter instances based on two int fields incremented when a request with (or without respectively) GC are processed.
Duration of requests with and without GC: two EventCounter instances updated when requests are processed.

Here is the implementation ofCreateCounters:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


private void CreateCounters()
{
    // the same request count can be used for two counters:
    // - raw request counter that will always increase
    // - increment counter that will automatically compute the delta
    //   between the current value and the value when the counter
    //   was previously sent
    _requestCount ??= new PollingCounter("request-count", this,
        () => _requestCountValue)
    { DisplayName = "Requests count" };
    _requestCountDelta ??= new IncrementingPollingCounter("request-count-delta", this,
        () => _requestCountValue)
    { DisplayName = "New requests", DisplayRateTimeScale = new TimeSpan(0, 0, 1) };

    // split the request counts between those for which a GC occured or not 
    // during their processing
    _noGcRequestCount ??= new PollingCounter("no-gc-request-count", this,
        () => _noGcRequestCountValue)
    { DisplayName = "Requests (processed without GC) count" };
    _withGcRequestCount ??= new PollingCounter("with-gc-request-count", this,
        () => _withGcRequestsCountValue)
    { DisplayName = "Requests (processed during a GC) count" };

    // request duration counters (with or without GC happening during the processing)
    _noGcRequestDuration ??= new EventCounter("no-gc-request-duration", this)
    { DisplayName = "Requests (processed without GC) duration in milli-seconds" };

    _withGcRequestDuration ??= new EventCounter("with-gc-request-duration", this)
    { DisplayName = "Requests (processed during a GC) duration in milli-seconds" };
}

The request processing code of the ASP.NET Core middleware is relying on the following helper methods to update the counters:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


internal void AddRequestWithoutGcDuration(long elapsedMilliseconds)
{
    IncRequestCount();
    Interlocked.Increment(ref _noGcRequestCountValue);

    // compute min/max/mean
    _noGcRequestDuration?.WriteMetric(elapsedMilliseconds);
}

internal void AddRequestWithGcDuration(long elapsedMilliseconds)
{
    IncRequestCount();
    Interlocked.Increment(ref _withGcRequestsCountValue);

    // compute min/max/mean
    _withGcRequestDuration?.WriteMetric(elapsedMilliseconds);
}

private void IncRequestCount()
{
    Interlocked.Increment(ref _requestCountValue);
}

And that’s it!

How to get these custom counters?

The controller of the ASP.NET Core sample application is triggering (or not) garbage collections based on the parameters passed via the url:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


[Route("api/[controller]")]
[ApiController]
public class RequestController : ControllerBase
{

    // GET: api/Request/5
    [HttpGet("{id}")]
    public string Get(int id)
    {
        if (id == -1)
        {
            return $"pid = {Process.GetCurrentProcess().Id}";
        }
        else
        if ((id >= 0) && (id <= 2))
        {
            GC.Collect(id);
            return $"triggered GC {id}";
        }
        else
        if (id <= 10)
        {
            // trigger a given number of GCs up to 10
            TriggerGCs(id);
            return $"triggered {id} garbage collections";
        }

        return $"value = {id}";
    }

    private void TriggerGCs(int count)
    {
        for (int current = 0; current < count; current++)
        {
            GC.Collect(0);
        }
    }

As explained earlier, it is possible to see the counter values with dotnet-counters by using the event source name as a provider with the following command line:

dotnet counters monitor -p Sample.RequestCounters

Then if you trigger a few requests with and without GC, you should see the numbers change:

The code available on Github has been updated to provide the middleware and the event source classes that demonstrate how to expose custom .NET Core counters.

.NET Core Counters internals: how to integrate counters in your monitoring pipeline

Tue, 23 Jul 2019 15:30:37 +0000

This post of the series digs into the implementation details of the new .NET Core counters.

Part 1: Replace .NET performance counters by CLR event tracing.

Part 2: Grab ETW Session, Providers and Events.

Part 3: CLR Threading events with TraceEvent.

Part 4: Spying on .NET Garbage Collector with TraceEvent.

Part 5: Building your own Java GC logs in .NET

Part6: Spying on .NET Core Garbage Collector with .NET Core EventPipes

Introduction

As explained in a previous post, .NET Core 2.2 introduced the EventListener class to receive in-proc CLR events both on Windows and Linux. Starting with .NET Core 3.0 Preview 6, the EventPipe-based infrastructure makes it now possible to get these events from another process. The diagnostics repository contains the cross-platform tools leveraging this infrastructure:

dotnet-dump: take memory snapshot and allow analysis based on most SOS commands
dotnet-trace: collect events emitted by the Core CLR and generate trace file to be analyzed with Perfview
dotnet-counters: collect the metrics corresponding to some performance counters that used to be exposed by the .NET Framework

At Criteo, our metrics are exposed in Grafana dashboards and it is interesting to figure out how the new counters are implemented and see how to fetch them via the EventPipe infrastructure. With this knowledge in hand, I’ve implemented helpers to let you get counters in less than 10 lines of code:

1
2
3
4
5
6
7


_counterMonitor = new CounterMonitor(_pid, GetProviders());
_counterMonitor.CounterUpdate += // receive the value of one counter after the other

Task monitorTask = new Task(() => {
    _counterMonitor.Start();
});
monitorTask.Start();

At the end of this post you will be able to very easily integrate any counter to your own monitoring pipeline!

.NET Core replacement for .NET Framework Performance Counters

With .NET Core being cross-platform, performance counters were gone and, as explained in the previous posts of the series, CLR events were the only way to get metrics about how your .NET Core applications were behaving. However, with .NET Core 3.0, it is now possible to view a few metrics thanks to the dotnet-counters tool.

You can download and install the tools automatically if you have installed .NET Core SDK 2.1+. Microsoft is currently working to provide other ways to directly download the tools binaries without having to install the SDK or recompile the diagnostics repository.

Use the following command line to install dotnet-counters: dotnet tool install --global dotnet-counters --version 3.0.0-preview7.19365.2

Note that you need to have the same version both for the Core CLR runtime and for the tools because, as you will soon see, the monitoring and the monitored applications are communicating via a dedicated protocol (that have changed between previews) on top of a transport layer different between Windows and Linux.

After the installation, use the following command line dotnet counters monitor -p and you get a 1 second auto-refreshed view of counters.

These counters are exposed by the System.Runtime provider and are detailed with the list argument:

This list is currently hard-coded in the CreateKnownProviders method. However, you are free to create your own provider and expose your application metrics as shown in this tutorial (and in the next forthcoming post). In addition, if you are using ASP.NET Core, starting from Preview 7, then you could get a few counters from the “Microsoft.AspNetCore.Hosting” provider defined in HostingEventSource.cs.

What are these “counters”

Even though it is nice to have a console-based cross-platform tool to see the values of counters change, what would be the cost to get them into your own monitoring pipeline? For example, at Criteo, we are pushing our metrics to Graphite in order to get nice Grafana dashboards. These graphical representations allow us to have a visual representation of the evolution of metrics over time. In addition, it is also possible to define alerts based on threshold for some metrics values (when CPU > 85% for more than 5 seconds for example).

In a nutshell, dotnet-counters tool is listening to another application via EventPipe. Unlike .NET Framework performance counters that are polled by the monitoring application, the counters are pushed by the monitored .NET Core process.

In term of implementation, these counters are values that you could get via .NET internal or public APIs if you were running in-proc as shown in RuntimeEventSource.cs:

Unlike most of the events that previous posts of this series presented, counters are metrics that are computed by the CLR in the monitored application. They are supposed to provide a set of values changing over time in the monitored application without impacting the performance nor flooding the listener client. I highly recommend to take a look at this issue for a deeper discussion about EventCounters compared to regular events.

As of Preview 7, two types of counters are used:

Mean: supposed to contain a mean of all values during the polling interval with its min and max values. However, based on the current implementation, all contain only the current value.
Sum: contains an increment between the previous value and the current one

The question is now to figure out how to get the values of the counters.

How to receive the counters?

Like the Perfview tool that relies on TraceEvent library, dotnet-counters uses an API exposed by Microsoft.Diagnostics.Tools.RuntimeClient assembly. Note that it is currently not (yet) available from nuget so you need to recompile it with the diagnostics git repo.

To receive counters, you need to create an EventPipe session that communicates via IPC (named pipes on Windows and domain sockets on Linux) with the CLR of the monitored process. Here is an excerpt of the CounterMonitor.StartMonitoring implementation that connects and listens to counter events:

1
2
3
4
5
6
7
8
9


var configuration = new SessionConfiguration(
    circularBufferSizeMB: 1000,
    outputPath: "",
    providers: Trace.Extensions.ToProviders(providerString));

var binaryReader = EventPipeClient.CollectTracing(_processId, configuration, out _sessionId);
EventPipeEventSource source = new EventPipeEventSource(binaryReader);
source.Dynamic.All += ProcessEvents;
source.Process();

The important method call is call is EventPipeClient.CollectTracing() that returns a Stream from which an EventPipeEventSource instance gets created. This class has been added to TraceEvent so you can now leverage the event parsing infrastructure on top of EventPipe! As shown in a previous post, it is easy to attach a listener to the source All .NET event and get notified each time an event is received after the Process method is called.

A few parameters are given to CollectTracing via the SessionConfiguration object: the size of the circular buffer used by the CLR and no file path because we want a live session. The last one is supposed to filter which providers and counters you would like to listen to: it expects a list of Provider instances. This struct is created with a few parameters:

1
2
3
4
5
6


    public struct Provider
    {
        public Provider(string name, ulong keywords = ulong.MaxValue, 
                        EventLevel eventLevel = EventLevel.Verbose, 
                        string filterData = null)
        { ... }

As we have already mentioned, the name of the provider is “System.Runtime” for the Core CLR counters. The keywords and event level are expected to have these max values. The filter data string starts with “EventCounterIntervalSec=” followed by the refresh interval in seconds. Internally, the CLR in the monitored application is creating a timer with that frequency to push the counters via EventPipe (more on this later).

Here is a helper class to easily create your providers:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


public class CounterHelpers
{
    public static Provider MakeProvider(string name, int refreshIntervalInSec)
    {
        var filterData = BuildFilterData(refreshIntervalInSec);
        return new Provider(name, 0xFFFFFFFF, EventLevel.Verbose, filterData);
    }

    private static string BuildFilterData(int refreshIntervalInSec)
    {
        if (refreshIntervalInSec < 1)
            throw new ArgumentOutOfRangeException(nameof(refreshIntervalInSec), $"must be at least 1 second");

        return $"EventCounterIntervalSec={refreshIntervalInSec}";
    }
}

Note that dotnet-counters allows you to pass a subset of the counters with the System.Runtime[counter1,counter2,counter2] syntax: events for all System.Runtime counters will be received but only these three will be displayed in the console.

Show time for counter events!

Next, the important part of the job takes place in the EventSourc.All event listener. Each new counter value is received in the payload of an event named “EventCounters”.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


private void ProcessEvents(TraceEvent data)
{
    if (data.EventName.Equals("EventCounters"))
    {
        IDictionary<string, object> countersPayload = (IDictionary<string, object>)(data.PayloadValue(0));
        IDictionary<string, object> kvPairs = (IDictionary<string, object>)(countersPayload["Payload"]);

        var name = string.Intern(kvPairs["Name"].ToString());
        var displayName = string.Intern(kvPairs["DisplayName"].ToString());

        var counterType = kvPairs["CounterType"];
        if (counterType.Equals("Sum"))
        {
            OnSumCounter(name, displayName, kvPairs);
        }
        else
        if (counterType.Equals("Mean"))
        {
            OnMeanCounter(name, displayName, kvPairs);
        }
        else
        {
            throw new InvalidOperationException($"Unsupported counter type '{counterType}'");
        }
    }
}

The Name and DisplayName values are self-explanatory. The Sum/Mean type is retrieved from CounterType.

The value for each counter type is retrieved from the payload with “Increment” (Sum type) or “Mean” (*Mean *type) keys.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


 private void OnSumCounter(string name, string displayName, IDictionary<string, object> kvPairs)
{
    double value = double.Parse(kvPairs["Increment"].ToString());

    // send the information to your metrics pipeline
}

private void OnMeanCounter(string name, string displayName, IDictionary<string, object> kvPairs)
{
    double value = double.Parse(kvPairs["Mean"].ToString());

    // send the information to your metrics pipeline
}

The CounterMonitor class has been added on my Github to expose a CounterUpdate C# event when a counter event is received:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


public class CounterMonitor
{
    ...
    public event Action<CounterEventArgs> CounterUpdate;
    private void OnSumCounter(string name, string displayName, IDictionary<string, object> kvPairs)
    {
        double value = double.Parse(kvPairs["Increment"].ToString());

        // send the information to your metrics pipeline
        CounterUpdate(new CounterEventArgs(name, displayName, CounterType.Sum, value));
    }

    private void OnMeanCounter(string name, string displayName, IDictionary<string, object> kvPairs)
    {
        double value = double.Parse(kvPairs["Mean"].ToString());

        // send the information to your metrics pipeline
        CounterUpdate(new CounterEventArgs(name, displayName, CounterType.Mean, value));
    }
}

The event argument contains the expected properties but other could be added if needed such as the timestamp for example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


public class CounterEventArgs : EventArgs
{
    internal CounterEventArgs(string name, string displayName, CounterType type, double value)
    {
        Counter = name;
        DisplayName = displayName;
        Type = type;
        Value = value;
    }

    public string Counter { get; set; }
    public string DisplayName { get; set; }
    public CounterType Type { get; set; }
    public double Value { get; set; }
}

public enum CounterType
{
    Sum = 0,
    Mean = 1,
}

Let’s show some graphs!

With these helpers in hand, it is easy to integrate any counter to your monitoring pipeline. As an example, let’s see how to generate a .csv file used to create visual representations in Excel.

With a refresh rate of 1 second, one line containing the value of the CLR counters should be added to the .csv file every second. Since we get one event per counter, we need to know which is the “last” counter event sent by the CLR for a given 1 second counters push.

As mentioned earlier the RuntimeEventSource class defines the CLR counters. Each one is an instance of a type derived from the DiagnoticCounter class that associates its instances to a CounterGroup also bound to the RuntimeEventSource. The CounterGroup class will setup a repeating timer responsible for creating the payload for its DiagnosticCounter-derived instances and ask the event source to send each to the monitoring application via EventPipe.

So we can rely on the order defined by the counters creation code in RuntimeEventSource: for a given push of counters, the name of the last one will be “assembly-count”. Beware that in a case of new counters (such as for ASP.NET Core), you would need to check what would be the last one of the counters series. Another way to work around would be to rely on the timestamps of each event but this could become flaky over time. It would have been great if a “CounterSeries”event containing the list of counter names would have been sent before any “EventCounters” of a series push (good idea for a pull request :^)

The CsvCounterListener class wraps the few lines of code needed to handle the events and add a line into the .csv file each time a series of counters is received:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115


public class CsvCounterListener
{
    private readonly string _filename;
    private readonly int _pid;
    private CounterMonitor _counterMonitor;
    private List<(string name, double value)> _countersValue;

    public CsvCounterListener(string filename, int pid)
    {
        _filename = filename;
        _pid = pid;
        _countersValue = new List<(string name, double value)>();
    }

    public void Start()
    {
        if (_counterMonitor != null)
            throw new InvalidOperationException($"Start can't be called multiple times");

        _counterMonitor = new CounterMonitor(_pid, GetProviders());
        _counterMonitor.CounterUpdate += OnCounterUpdate;

        Task monitorTask = new Task(() => {
            try
            {
                _counterMonitor.Start();
            }
            catch (Exception x)
            {
                Environment.FailFast("Error while listening to counters", x);
            }
        });
        monitorTask.Start();
    }

    private void OnCounterUpdate(CounterEventArgs args)
    {
        _countersValue.Add((args.DisplayName, args.Value));

        // we know that the last CLR counter is "assembly-count"
        if (args.Counter == "assembly-count")
        {
            SaveLine();
            _countersValue.Clear();
        }
    }

    bool isHeaderSaved = false;
    private void SaveLine()
    {
        if (!isHeaderSaved)
        {
            File.AppendAllText(_filename, GetHeaderLine());
            isHeaderSaved = true;
        }

        File.AppendAllText(_filename, GetCurrentLine());
    }

    private string GetHeaderLine()
    {
        StringBuilder buffer = new StringBuilder();
        foreach (var counter in _countersValue)
        {
            buffer.AppendFormat("{0}\t", counter.name);
        }

        // remove last tab
        buffer.Remove(buffer.Length - 1, 1);

        // add Windows-like new line because will be used in Excel
        buffer.Append("\r\n");

        return buffer.ToString();
    }

    private string GetCurrentLine()
    {
        StringBuilder buffer = new StringBuilder();
        foreach (var counter in _countersValue)
        {
            buffer.AppendFormat("{0}\t", counter.value.ToString());
        }

        // remove last tab
        buffer.Remove(buffer.Length - 1, 1);

        // add Windows-like new line because will be used in Excel
        buffer.Append("\r\n");

        return buffer.ToString();
    }

    public void Stop()
    {
        if (_counterMonitor == null)
            throw new InvalidOperationException($"Stop can't be called before Start");

        _counterMonitor.Stop();
        _counterMonitor = null;

        _countersValue.Clear();
    }

    private IReadOnlyCollection<Provider> GetProviders()
    {
        var providers = new List<Provider>();

        // create default "System.Runtime" provider with a refresh every second
        var provider = CounterHelpers.MakeProvider("System.Runtime", 1);
        providers.Add(provider);

        return providers;
    }
}

What’s next?

You have seen how easy it is to be notified of CLR counters update. The integration to your own monitoring system should not be more complicated. However, you need to pay attention to the meaning of counter types between *Mean *and Sum. For example, the value you get for gen-0-count (Sum) counters is a difference between now and the previous computation. It means that you can’t have the “current” number of gen 0 collection at a given time.

This is not a problem in the Excel example because you can “rebuild” a column that will contain the “current” count based on the previous value + the diff returned by the counter.

Here is the resulting graph:

In other cases, you might need to feed your monitoring system with real count values and benefit from advanced charting such as non derivative computation to show a rate based on a series of values. At the end of the day, it is just a question of initial value from which rebuild a count. And if you think about it, you are often more interested in unexpected variations (i.e. differences returned by counters) when monitoring your application.

In addition to your business metrics, .NET Core Counters are usually enough to monitor the health of your applications. However, in order to investigate situations where counters value are showing weird results, you often need more details. For example spikes in garbage collections count might not be a problem if the pause time is not too long. Listening to specific CLR events as shown in previous posts of this series is a great way to unveil important metrics such as GC pause time, contentions duration or exception names without performance hit.

The code available on Github has been updated to provide the CounterMonitor and CsvCounterListener classes that demonstrates how to get .NET Core counters and generate .csv file usable in Excel.

Let’s debug the Core CLR with WinDBG!

Thu, 04 Apr 2019 13:32:51 +0000

This post of the series shows how we debugged the Core CLR to figure out insane contention duration.

Part 1: Replace .NET performance counters by CLR event tracing.

Part 2: Grab ETW Session, Providers and Events.

Part 3: CLR Threading events with TraceEvent.

Part 4: Spying on .NET Garbage Collector with TraceEvent.

Part 5: Building your own Java GC logs in .NET.

Introduction

Long before migrating our .NET applications to Linux, our first step was to build a monitoring pipeline based on LTTng instead of ETW on Windows. To achieve this goal, the open source TraceEvent Nuget package needed to be updated in order to listen to LTTng live session (only a file based implementation was provided by Microsoft; mostly to allow Perfview to be able to open traces taken on Linux machines). This was a huge development task that led sometimes to weird results. Among the metrics we wanted to monitor, the contention duration gave insane value such as thousands of minutes… per minute:

As shown in a previous episode, this duration is computed by comparing the time between the two events ContentStart and ContentionStop. What could be the possible reasons to get such insane values?

A lot of small contentions are happening
A few very long contentions are happening

As a first step, it would be great to be able to debug the Core CLR and figure out what call stacks end up to triggering these contention events. Unfortunately for us, the .NET debugging ecosystem on Linux is far from being as rich as on Windows. So this episode is detailing the steps to compile and debug the Core CLR on Windows with WinDBG.

From the source to debugging the runtime

To better understand the implementation details in the CLR, we needed to find where the two events are emitted. In fact, during the CLR compilation, a lot of helpers are created based on the name of the event. In our case, FireEtwContentionStart_V1 and FireContentionStop are the two helpers in charge. Both are called in the AwareLock::EnterEpilogHelper function.

As a Windows developer, I would like to debug the CLR code and set a breakpoint in the EnterEpilogHelper with Visual Studio to see what are the call stacks that end up to contention. However, I did not find a way to do it with Visual Studio. I turned to WinDBG and things gets “easier”… in a certain way.

Here are the different steps you need to follow before setting a breakpoint on any Core CLR function in WinDBG:

Clone the Core CLR repository from https://github.com/dotnet/coreclr
Build it:
Get the Visual Studio, .NET Core SDK, CMake, Python, Powershell prerequisites from the documentation
Goto the root folder and type .\build -skiptests to build a DEBUG version of the Core CLR
Leave your desk and go to lunch (ok… maybe just take a coffee break)

When you go back, the result of the compilation should be available in the following folder:

…\coreclr\bin\Product\Windows_NT.x64.debug.

the next step is to use your custom Core CLR build in the application:

the application must be self-contained by adding win-x64 (or linux-x64 for Linux) in a PropertyGroup section of the .csproj.
publish the application by running dotnet publish or from within Visual Studio

Click the Configure link and select Debug configuration

after clicking Save and Publish, you should now have the result under the \bin\Debug\netcoreapp2.2\publish folder.
after clicking Save and Publish, you should now have the result under the \bin\Debug\netcoreapp2.2\publish folder.

It is now time to copy the following files from the Core CLR output to your application publication folder:

coreclr.dll (for the native part of the CLR) and System.Private.CoreLib.dll (if the CLR C# code has been modified)
in the PDB subfolder, coreclr.pdb and System.Private.CoreLib.pdb
note that you might also need the sos.dll and mscordaccore.dll files for any investigation in WinDBG.

If you wonder why the CoreFx repo is not rebuilt, the answer is simple: the contention related code is in the CoreCLR. Also, most of the managed “mscorlib” is defined in System.Private.CoreLib.dll that gets built with CoreCLR. The rest of the BCL is covered by CoreFX and not needed in this investigation.

From running to debugging in WinDBG

You should use corerun.exe instead of dotnet.exe to run an application with the debug version of the Core CLR you’ve just built.

Open up a command prompt in the coreclr\bin\Product\Windows_NT.x64.debug folder and type corerun** c:bin\Debug\netcoreapp2.2\publish folder of your application>**

You have to tell corerun where to find the CoreFx assemblies via the CORE_LIBRARY environment variable:

CORE_LIBRARIES=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.0

If you forget about it, don’t be surprised if the application stops with FileNoteFoundException for a missing assembly (usually System.Runtime)…

If, like me, your applications are running with server mode GC, you know that it is set in the application .csproj file to end up into the runtimeconfig.json file. Unfortunately, this is not taken into account by corerun (yet?) and you need to set it (and if you need concurrent version too) explicitly through the following environment variables:

COMPlus_gcServer=1 COMPlus_gcConcurrent=1

From there, (install WinDBG if not already done and) start the debugger: click the File menu and select Launch Executable (advanced) to setup a debugging session:

The Executable text field points to the corerun.exe file generated during the compilation of the Core CLR. The same folder is used as Start Directory and the Arguments text field contains the full path of the application to debug. You could also attach to a running process but sometimes you need to access Core CLR data structures before any C#-compiled managed code of your application starts executing (to see how the garbage collector initializes for example).

As soon as you click the Ok button, the application starts but is almost immediately stopped by WinDBG

Don’t be scared by the last lines of the output: even through you read the word exception, this int 3** **assembly instruction tells you that a breakpoint has been set for you by WinDBG, has been hit when the application reached it and the application is now paused just before calling its entry point.

As you can see from the list of loaded modules, even though CoreRun.exe is there, no managed assembly (especially your application) has been loaded yet; not even the Core CLR itself! This means that you have to tell WinDBG to keep on executing the application until a point you would be interested in. To achieve that task, you will first need a quick tour of WinDBG user interface even though this post is not there to replace the WinDBG online help nor provide a detailed walkthrough.

The debugging section of the Home tab is not too different from what you get in Visual Studio:

The icons are even easier to understand because their action is also displayed. If you want to see the current call stack, select the View tab and click the Stack icon:

Like in Visual Studio, you are able to pin each panel wherever you want into the IDE

The next step would be to set a breakpoint on the line of code you are interested in. But let’s be clear here: I’m talking about a line of code in a function exported by a native dll; not a line of C# code in a managed assembly. Remember that WinDBG is a native debugger and it debugs only native code. If you want to debug managed code with WinDBG, you need to use commands from the sos extension; but this is another story.

So let’s go back to the native world. Even though WinDBG does not have the notion of “solution” like Visual Studio provides, it is still possible to open a C++ file and set a breakpoint in it. Click File |Open Source File menu and go to your Core CLR github repo to select syncblk.cpp under the \src\vm folder. Look for AwareLock::EnterEpilogHelper with CTRL+F (yes: search is working in WinDBG) and go down to the call to the FireEtwContentionStart_V1 helper. Setting a breakpoint on this line is as simple as pressing **F9 **like in Visual Studio. Press the View tab and click the Breakpoint button to see the result:

Since the dll in which the breakpoint is set is not loaded yet, you can’t see the details of the breakpoint.

There is a way to tell WinDBG to continue the execution of the application until a dll get’s loaded. For coreclr.dll, type the following command:

sxeld:coreclr

and type F5 (or type g as a command or click the green triangle in the Home toolbar) to resume the execution of the application. The Breakpoints panel shows more details now:

Press F5 to resume the execution and the breakpoint should be triggered when the first contention happens.

From symbols to call stacks in WinDBG

Before digging into call stacks, I would like to show you one of the differences between native dlls and managed assemblies. As a .NET developer, you are used to Intellisense and strongly typed environment provided by the metadata stored in an assembly itself. For native dll, the story is different. Exported functions are visible with tools such as Dependency Walker or dumpbin /exports from the SDK. If the dll exports symbols built by the C++ compiler, their name gets mangled to describe their signature. To get human readable symbols, you need the associated .pdb file. It will also be required to map a function address to its name in call stacks.

WinDBG allows you to browse these symbols with the dt command. For example, if you want to know all members defined by the AwareLock class, use the following command:

dt CoreClr!AwareLock::*

Like what was shown in the previous Breakpoints screenshot, the prefix of a name is the dll in which the symbol is defined. Next, use ! as separator before the class name. Since Visual Studio is really slow to navigate the source code of the Core CLR or search in the thousands of include and C/C++ files, this is a very convenient way to navigate and learn its different parts. Don’t forget that the compilation could also inline functions (that won’t be visible in the symbols) and expand macros.

If you want to set a breakpoint on a function, use the bp command with the same syntax as dt. For example, the following command:

bp coreClr!AwareLock::EnterEpilogHelper

sets a breakpoint at the beginning of the function in which I already set a breakpoint.

This is the very basics of breakpoints in WinDBG. You are also able to define which actions to start when a breakpoint is hit. This is extremely powerful! For example, in the case of thread contention, you typically don’t want to stop the execution of the application because it will pause all threads and disturb the normal flow of execution that could lead to thread contention. Instead, you could ask WinDBG to dump the call stack leading to the function we are interested in and lets the execution resume with the following syntax:

bp coreClr!AwareLock::EnterEpilogHelper "!clrstack; g"

The commands to execute after the breakpoint is hit are defined between quotes. In this example, I’m using the clrstack command exported by the sos.dll extension (that must be previously loaded via .loadby sos coreclr) and once it is done, g resumes the execution.

What’s next?

Due to automatic suspension of all threads when the clrstack command gets executed (before g resumes), the interactions between threads are not the same as normal execution outside of a debugger. I have even used some code available in DEBUG to dump the callstacks outside of a debugger if the contention last more than a threshold. However, it was not possible to reproduce the problem on Windows.

In parallel on Linux, another colleague investigated another lead: some events may also be skipped by our LTTng implementation. Due to complicated event management, if a ContentionStop and ContentionStart are missed, a possible previous ContentionStart could be used by the next ContentionStop and the duration would be unrelated to the real contention that happened.

So there could be a simpler way to narrow down the issue: instead of relying on two events, why not simply compute the duration of the contention in the AwareLock::EnterEpilogHelper function and emit only one new event with the duration as payload? Well… this will be the topic of the next episode of this series.

References

Series of videos from the Defrag Tools show where Maoni Stephens explains how to debug the Garbage Collector for a better understanding of its arcana

Debugging Friday — Hunting down race condition

Fri, 22 Feb 2019 09:25:50 +0000

Introduction

At Criteo, CLR metrics are collected by a service that listens to ETW events (see the related series). On a few servers, the metrics stopped being collected and we had to fix the problem by manually polling new and dead processes. After deploying the new version, the same scenario started to happen: on some servers, the metrics were no more collected.

In an investigation, the first step is always trying to check the environment. In our case, on a server where the metrics collector is up and running, a dedicated ETW session should be created to listen to the CLR events.

The name given to the session allows us to easily detect if the session is present or not. In the case of a faulted server, the session was not present.

If you look at the code described in a previous post, it is not easy to guess why the session would be stopped by the metrics collector:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


private void ListenToEtw(TraceEventSession etwSession)
{
    try
    {
        // this call is blocking... until ewtSession.Stop is called (done in Dispose)
        etwSession.Source.Process();
    }
    finally
    {
        etwSession.Dispose();
    }
}

A TraceEventSession is created and passed to a dedicated thread to process the events until Stop is called at the end of the application.

The second step of an investigation is trying to reproduce the issue in a controlled environment such as… my developer machine. I setup the Exception Settings of the debugger to stop on any managed exception:

That way, if something bad happens while I’m debugging the application, Visual Studio tells me exactly where the exception was thrown.

After starting and stopping applications monitored by the metrics collector a few times, an exception was thrown in the code responsible for mapping the application id and the component in charge of storing the metrics of this process. When looking at the code, it seems that there was a “timing” conflict between the code in charge of detecting new and dead processes (in a timer) and the code receiving the events from ETW (in the dedicated thread described earlier). A CLR event was received after the corresponding process was detected as being dead. The net effect of the uncaught exception was fast: the TraceEvent session stopped its execution and the Process method returned. Nothing special visible outside of a debugger with the right exception settings. This is a great scenario to understand why swallowing all exceptions is not a good pattern…

Still not working

The next step is to fix the code to handle the dead process case, build it and test it. Unfortunately, the new metrics collector, from time to time, does not seem to receive any CLR event. Even worse, this time the ETW session is still here as shown by logman -ets. Going back inside the Visual Studio debugger, everything is working fine: the TraceEvent session is created, its Process method called and blocked in a dedicated thread and… the events are received! It means that I’m not able to reproduce the problem while running under debugger control. Maybe the problem could come from the code responsible to filter out events sent by unmonitored applications.

So, I’m adding a Breakpoint in the method responsible for checking monitored process to ensure that there is no bug there (events could be received but skipped due to invalid process ID mapping for example):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


private bool IsMonitoredEvent(TraceEvent traceEvent)
{
    var isMonitored = traceEvent.ProcessID == _monitoredPid;
    if (isMonitored)
    {
        NotifyProcessedEvent(traceEvent);
    }

    return isMonitored;
}

The breakpoint is hit and I’m able to validate that there is no problem in the mapping code

I can even check that the events I’m expecting are all received by asking Visual Studio to trace the event name with a tracepoint when isMonitored is true:

And I get all expected events in the Output Window. If some events were missing, it could have explained that metrics based on events series (such as contention duration) were not computed.

I’m now running the application outside of the debugger… and no event. Just to confirm that I’m not crazy, I decide to add traces in the source code:

But how to get the output without an attached debugger? The trick is to start SysInternals Debug View and wait for the event names to appear: nothing. Even by moving the Debug.WriteLine call outside of the if block, no event is ever received, even from unmonitored processes.

Navigating memory by stack frame

Let’s summarize the investigation status:

The metrics collector is working only when under the control of a debugger.
If the debugger is attached after it is started, the events are not received.

I don’t know why but this kind of weird behaviors is always happening at Criteo on a Friday. So let’s start a joined debugging session with Kevin, Jean-Philippe and Gregory!

To better understand what is the state of the metrics collector when the events are not received, the application is launched and the debugger is then attached to it. I open the Parallel Stacks panel and double-click the stack frame with a valuable context:

In our case, it would be interesting to get a view on the state of the ETWTraceEventSource object used by TraceEvent to process the events. Even if you don’t have the source code, it is still possible for the debugger to get a view of the object used as implicit “this” pointer by the ProcessOneFile method. Summon the Quick Watch dialog (Shift-F9 with my old VC 6 keyboard shortcut) and type “this”

Based on own understanding of how the ETWTraceEventSource is working, we know that registered event handlers are associated to entries in its templates field.

Instead of the expected GC, thread pool, exception and contention events, only kernel related events are defined:

But breakpoints have been set and hit on the code that registers our own event handlers! Well… it was the case when we debugged the application from start. What if… the event source we are looking at now is not the one our code has registered its handlers to?

Without the address of the object available like in C++, it is complicated to easily check if two references actually point to the same object in memory. However, it is still possible to associate a numeric ID to an object with Make Object ID:

And its ID {$1} is now visible after the type name:

To compare with the ETWTraceEventSource object manipulated by our own source code, double-click the right frame in Parallel Stack:

In the ListenToEtw method, this refers to our SessionManager in which the ClrTraceEventParser property references the ETWTraceEventSource object we believe we initialized with the right event handlers:

And… this object does not have any ID: it should be {$1}.

To confirm that we are not looking at the source object {$1} that TraceEvent uses to receive events, it is just a question of checking its template field:

And here are the expected CLR events we are interested in!

Race condition… again

So now the question is: how is it possible to have two instances of a TraceEvent internal class?

Here is the sequence of execution in our code:

Thread 1 is creating a TraceEventSession object
Thread 1 starts Thread 2
Thread 1 accesses the clrTraceEventParser via _currentSession?.Source.Clr

and

Thread 2 calls etwSession.Source.Process()

So the Source property getter could be called from two different threads. Unfortunately, in the getter, the source is lazily created in a non thread-safe way.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


public ETWTraceEventSource Source
{
   get
   {
      if (m_source == null)
      {
         ... // long code 
         m_source = new ETWTraceEventSource(SessionName, TraceEventSourceType.Session);
      }
      return m_source;
   }
}

When both threads enter the getter, m_source could be null and in that case, two ETWTraceEventSource objects are created and returned. One is used by TraceEvent to listen to events and the other by our code to register handlers to events that will never be received.

The fix is simply to force the initialization of the Source object in the first thread.

It is now a good time to go back home… to take a well deserved vacation!

Building your own Java-like GC logs in .NET

Tue, 12 Feb 2019 10:11:39 +0000

This post of the series focuses on logging each GC details in a file and how to leverage it during investigations.

Part 1: Replace .NET performance counters by CLR event tracing.

Part 2: Grab ETW Session, Providers and Events.

Part 3: Monitor Finalizers, contention and threads in your application.

Part 4: Spying on .NET Garbage Collector with TraceEvent.

Introduction

I’m working in a team where we investigate issues in production: both for Java and .NET applications. This is a good opportunity to learn what are the features provided by Java that are missing in .NET. One of the features heavily discussed with my colleague Jean-Philippe is called the GC Log. It is possible to start an application with parameters that tell the GC to save tons of details about each garbage collection in a file : the GC Log. Based on this file, it is possible to extract the reason of a collection, the times of the different phases including the suspension time. This is a great source of information during investigations… when you know how GC is working or by leveraging automatic report generation.

In addition, you can also build your own UI to more easily understand what is going on and get a more visual representation of the situation.

In the short video above you can see the heap evolution during several days. Then, as this is an interactive HTML page you can zoom in an interesting period to have a more detailed view of the evolution between GCs.

Also for the pause time graph, you can follow the behavior of the GC with different kinds of pauses and associated phases. In this example, we have minor GCs happening and then an “initial mark” is triggered, followed by “final remark” and “cleanup” pauses. After an extra minor GC, we have a series of mixed GCs that is the result of what was planned by the GC after the marking phase.

In the .NET world, there is no such thing as a GC Log. However, as shown in the previous post, it is possible to use Perfview to analyze traces corresponding to collected CLR events. The GCStats view shows high level details in the “All GC Events” section. In addition to this HTML rendering, you can get access to the data itself in different formats

The more complete one is the Raw Data XML file that you could parse to extract the details you need. This is very close to a .NET GC Log but it is complicated to build an automated process to get it from a production machine.

It would be great if you could tell a .NET application to generate such a GC Log like in Java instead of relying on manual steps with Perfview (and more scripts on Linux). This post will show you how to achieve this goal!

Defining the goal: basic GcLog implementation

In Java, you have to set on or off the GC log before the application starts and you can’t change it while it runs. Since I’m working with server applications, I would prefer to enable/disable the generation of a GC log file without having to stop and restart the application.

So I’ve defined the simple IGcLog interface:

1
2
3
4
5
6


public interface IGcLog
{
    void Start(string filename);

    void Stop();
}

In a dedicated administration handler (i.e. http endpoint of the application), the code could just use a class that implements this interface and call Start when the log is enabled and Stop when it is no more needed.

To make the implementation easier, I’ve written the following GcLogBase abstract class:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


public abstract class GcLogBase : IGcLog
{
    protected string Filename;
    private StreamWriter _fileWriter;

    public void Start(string filename)
    {
        if (string.IsNullOrEmpty(filename))
            throw new ArgumentNullException(nameof(filename));

        if (_fileWriter != null)
            throw new InvalidOperationException("Start can't be called twice: Stop must be called first.");

        _fileWriter = new StreamWriter(filename);
        Filename = filename;

        OnStart();
    }

    public void Stop()
    {
        if (string.IsNullOrEmpty(Filename))
            return;

        OnStop();
        Filename = null;
        _fileWriter.Flush();
        _fileWriter.Dispose();
        _fileWriter = null;
    }

    protected bool WriteLine(string line)
    {
        if (_fileWriter == null)
            return false;   // just in case the method is called AFTER Stop

        _fileWriter.WriteLine(line);

        return true;
    }

    protected abstract void OnStart();

    protected abstract void OnStop();
}

Its main goal is to hide the file management by providing the WriteLine method that child classes would call to save the details of a garbage collection into a single line of text. The write operations are flushed when Stopis called. This combination allows asynchronous writes with low performance impact: don’t be scared if you don’t see the file size change because the StreamWriter class is caching write operations.

The next step is to implement OnStart and OnStop in a derived class to enable/disable GC details retrieval.

How to get the GC details: the easy way?

As already discussed in the previous posts of the series, the CLR is emitting traces (via ETW on Windows and LTTng on Linux) that can be collected in C#. You have already seen how TraceEvent could help collecting and parsing GC traces from any application like what Perfview is doing. With TraceEvent, the TraceGC instance received when a garbage collection ends contains tons of information: it’s mapped to the GarbageCollectionArgs structure that you get while listening to the GarbageCollection event of my ClrEventsManager helper class. The only information to provide is the ID of the .NET process I’m interested in: that way, is it easy to filter the events for this process only.

1
2
3
4
5
6
7
8
9


EtwGcLog gcLog = EtwGcLog.GetProcessGcLog(pid);
var filename = GetUniqueFilename(pid);
gcLog.Start(filename);

// in a simple Console application, wait for the user to press ENTER.
// in a more realistic case, keep track of the EtwLog instance and 
// call Stop to end the processing of events when needed.

gcLog.Stop();

The GetUniqueFilename method builds a filename based on the process ID and the time of the day:

1
2
3
4
5
6
7
8


private static string GetUniqueFilename(int pid)
{
    var now = DateTime.Now;
    string filename = Path.Combine(Environment.CurrentDirectory, 
        $"{pid.ToString()}_{now.Year}{now.Month}{now.Day}_{now.Hour}{now.Minute}{now.Second}.csv"
        );
    return filename;
}

The GetProcessGcLog method is a factory-like helper to build an instance bound to the given process ID.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


public static EtwGcLog GetProcessGcLog(int pid)
{
    EtwGcLog gcLog = null;
    try
    {
        var process = Process.GetProcessById(pid);
        process.Dispose();

        gcLog = new EtwGcLog(pid);
    }
    catch (System.ArgumentException)
    {
        // there is no running process with the given pid
    }

    return gcLog;
}

The implementation of OnStart and OnStop overrides is straightforward based on the previous post:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


protected override void OnStart()
{
    string sessionName = $"GcLogEtwSession_{_pid.ToString()}_{Guid.NewGuid().ToString()}";
    Console.WriteLine($"Starting {sessionName}...\r\n");
    _userSession = new TraceEventSession(sessionName, TraceEventSessionOptions.Create);

    Task.Run(() =>
    {
        // only want to receive GC event
        ClrEventsManager manager = new ClrEventsManager(_userSession, _pid, EventFilter.GC);
        manager.GarbageCollection += OnGarbageCollection;

        // this is a blocking call until the session is disposed
        manager.ProcessEvents();
        Console.WriteLine("End of CLR event processing");
    });

    // add a header to the .csv file
    WriteLine(Header);
}

protected override void OnStop()
{
    // when the session is disposed, the call to ProcessEvents() returns
    _userSession.Dispose();
}

The created TraceEventSession is passed to the ClrEventManager with the process ID with a filter to receive only GarbageCollection event notifications. The OnGarbageCollection handler is super simple:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


private void OnGarbageCollection(object sender, GarbageCollectionArgs e)
{
    _line.Clear();
    _line.AppendFormat("{0},", e.StartRelativeMSec.ToString());
    _line.AppendFormat("{0},", e.Number.ToString());
    _line.AppendFormat("{0},", e.Generation.ToString());
    _line.AppendFormat("{0},", e.Type);
    _line.AppendFormat("{0},", e.Reason);
    _line.AppendFormat("{0},", e.IsCompacting.ToString());
    _line.AppendFormat("{0},", e.SuspensionDuration.ToString());
    _line.AppendFormat("{0},", e.PauseDuration.ToString());
    _line.AppendFormat("{0},", e.BGCFinalPauseDuration.ToString());
    _line.AppendFormat("{0},", e.Gen0Size.ToString());
    _line.AppendFormat("{0},", e.Gen1Size.ToString());
    _line.AppendFormat("{0},", e.Gen2Size.ToString());
    _line.AppendFormat("{0},", e.LOHSize.ToString());
    _line.AppendFormat("{0},", e.ObjSizeBefore[0].ToString());
    _line.AppendFormat("{0},", e.ObjSizeBefore[1].ToString());
    _line.AppendFormat("{0},", e.ObjSizeBefore[2].ToString());
    _line.AppendFormat("{0},", e.ObjSizeBefore[3].ToString());
    _line.AppendFormat("{0},", e.ObjSizeAfter[0].ToString());
    _line.AppendFormat("{0},", e.ObjSizeAfter[1].ToString());
    _line.AppendFormat("{0},", e.ObjSizeAfter[2].ToString());
    _line.AppendFormat("{0}", e.ObjSizeAfter[3].ToString());

    WriteLine(_line.ToString());
}

Each garbage collection appears as a textual line with the following columns separated by a comma:

The last twelve pieces of information require some explanation:

· **xxxBefore **: size of a generation before the collection; without free list

· **xxxAfter **: size of a generation after the collection; without free list

· **xxxSize **: size of a generation (including LOH) after the collection; including free list (i.e. fragmentation)

The computation of these sizes relies on inner fields of the TraceGC argument receives from TraceEvent. The xxxSize are grouped in the GenerationSize0/1/2/3 fields of the HeapStat property. It is a little bit more complicated for the Before/After sizes. The Garbage Collector keeps track of these numbers in the PerHeapHistories field: an array of GCPerHeapHistory elements; one per heap (i.e. one per core for server GC). The next level is provided by the GenData field storing an array of GCPerHeapHistoryGenData elements; one per generation with LOH as the last index 3. So, to compute the size of each generation, it is needed to iterate on each heap:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


private long[] GetGenerationSizes(TraceGC gc, bool before)
{
    var sizes = new long[4];
    if (gc.PerHeapHistories == null)
    {
        return sizes;
    }

    for (int heap = 0; heap < gc.PerHeapHistories.Count; heap++)
    {
        // LOH = 3
        for (int gen = 0; gen <= 3; gen++)
        {
            sizes[gen] += before ? 
                gc.PerHeapHistories[heap].GenData[gen].ObjSpaceBefore:
                gc.PerHeapHistories[heap].GenData[gen].ObjSizeAfter;
        }
    }

    return sizes;
}

The code of the GetGenerationSizes helper method does that sum the value of either ObjSpaceBefore or ObjSizeAfter.

As you have probably noticed from the implementation, it is possible that the PerHeapHistories field is not filled up and all Before/After values are zero. This happens for a background gen2 collection. Also note that for gen0 and gen1 collection the value for gen2 and LOH is also 0 (make sense that gen2 and LOH do not change during such a collection).

A little bit of UI

Now that a .csv file containing all garbage collections details is available, it is time to provide some UI on top of it such as the following for GC pauses:

Let’s start what you can get for Excel champions:

Generation ratio

Sizes of generations including Large Object Heap

Top 10 pauses (including suspension time comparison)

But you can get better interaction thanks to Jean-Philippe. My colleague adapted his script for JVM to my .NET GC log .csv format: it generates some nice zoomable HTML UI.

This short video above shows the heap evolution during ~20 minutes. Then, as this is an interactive HTML page you can focus on gen2 and LOH impact on memory consumption.

For the pause time graph on the same period, it is very easy to detect long pauses (even for gen0 collection) and zoom into smaller period to figure out the impact of different collections.

The code available on Github has been updated to make the EtwGcLog class available to you.

[C#] Get-process-name challenge on a Friday afternoon

Tue, 13 Nov 2018 10:29:22 +0000

Unexpected CPU consumption

At Criteo, CLR metrics are collected by a service that listens to ETW events (see the related series). This metrics collector is given the process name of applications to monitor. Since applications could crash, be stopped or restarted, the metrics collector must be able to detect such an event. The previous implementation was using ETW kernel events (TraceEvent ProcessStart and ProcessStop events from ETWTraceEventSource.Kernel). However, in rare cases, it seems that a new application start was not detected and therefore the metrics were not collected for it.

An easy fix for this situation is to poll the list of running processes every second and detect which one is new or has left since the last time the list was polled. The implementation is straightforward: just call Process.GetProcesses() and get the process name from the Process MainModule.FileName property. After a few seconds testing this implementation on my laptop the fan started spinning: a quick look at Task Manager shows that the metrics collector is using ~10% CPU time!

I’ve used these P/Invoked PSAPI functions 20 years ago but I don’t remember such an impact: for our monitoring service, we would like to keep the CPU impact below 1%.

Measure, measure… and profile

This was a good opportunity to start profiling the metrics collector with dotTrace on a Friday afternoon!

NtProcessManager.GetModuleInfos and NtProcessManager.GetProcessIds are at the methods top list of CPU consumption the worst offenders by far:

The callstack to reach GetModuleInfos() shows the following:

The GetProcessPath() method of the metrics collector is asking for the value of Process.MainModule.FileName property that ends up calling GetModuleInfos.

And the callstack for the MainModule getter execution shows the following:

This call stack looks weird for two reason:

Since the Process object exists, why is it needed to call OpenProcess again to get the main module?
Why would OpenProcess need to call GetProcessIds (i.e. get the list of running processes) since its id is already known?!

Just take a look at the decompiled source code to get the answer:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


public static SafeProcessHandle OpenProcess(int processId, int access, bool throwIfExited)
{
  SafeProcessHandle safeProcessHandle = NativeMethods.OpenProcess(access, false, processId);
  int lastWin32Error = Marshal.GetLastWin32Error();
  if (!safeProcessHandle.IsInvalid)
    return safeProcessHandle;
  
  // error handling 
  if (processId == 0)
    throw new Win32Exception(5);
  if (ProcessManager.IsProcessRunning(processId))
    throw new Win32Exception(lastWin32Error);
...

And IsProcessRunning calls GetProcessIds:

1
2
3
4


public static bool IsProcessRunning(int processId)
{
   return IsProcessRunning(processId, GetProcessIds());
}

It does not appear in the callstack most probably because it was inlined by the JIT.

So, the code of ProcessManager.OpenProcess calls the Win32 OpenProcess API to get… the handle of the process corresponding to the given id and desired access rights. From there, it is spending most of its CPU time dealing with error cases (i.e. when a process information cannot be accessed maybe due to access right limitation). We definitively don’t need all that in our case!

What next?

At that point of the investigation, my colleague Kevin and I went to different directions. A few decades ago, I spent a lot of time digging into Windows internals using Win32 APIs and I remember that calling PSAPI.GetModuleFilenameEx with a pid and 0 as module handle should return the path name of the process (BTW, this is also what GetModuleInfos ends up calling but more on that later). So it should not be too complicated to P/Invoke this function from PSAPI.dll.

At the beginning of .NET programming, the https://pinvoke.net/ web site was very useful to figure out the right syntax for a lot of APIs if you did not want to read the 1579 pages of the Complete Interoperability Guide! The description of GetModuleFileNameEx is available, and even come with a code sample.

1
2


[DllImport("psapi.dll", BestFitMapping = false, CharSet = CharSet.Auto, SetLastError = true)]
private static extern int GetModuleFileNameEx(SafeProcessHandle processHandle, IntPtr moduleHandle, StringBuilder baseName, int size);

Note that marshaling strings should always be done with care: the Win32 API is not always consistent when a pointer to a C-like string is supposed to be filled up by a function. An additional parameter is given to state the size of the buffer in which the characters of the string will be copied. In some cases, this parameter counts the number of characters and in some others, it counts the number of bytes available in the buffer. I let you imagine what a nightmare it was when you had to deal with ANSI/UNICODE strings. In the GetModuleFileNameEx case, the size parameter takes the number of characters.

If you take a look at the NtProcessManager.GetModuleInfos implementation, you find the following code in the implementation:

1
2
3
4
5
6
7


StringBuilder stringBuilder2 = new StringBuilder(1024);
if (Microsoft.Win32.NativeMethods.GetModuleFileNameEx(
    safeProcessHandle, 
    new HandleRef(null, handle), 
    stringBuilder2, 
    stringBuilder2.Capacity * 2
 ) == 0)

Since the capacity of the StringBuilder is set to 1024, this code tells GetModuleFileNameEx that it is allowed to write up to 2048 characters. This looks like a bug… but hard to trigger with the usual 260 characters limitation for filenames. However, if, one day, you decide to use the extended syntax with the “\?\” prefix syntax to create a looooong folder for your application, the bug will trigger an AccessViolationException beyond 1025 characters.

Here is my safer implementation:

private readonly StringBuilder _baseNameBuilder = new StringBuilder(1024);
public static string GetProcessNameNative(Process p)
{
    _baseNameBuilder.Clear();
    if (GetModuleFileNameEx(p.SafeHandle, IntPtr.Zero, _baseNameBuilder, _baseNameBuilder.Capacity) == 0)
    {
        _baseNameBuilder.Append("???");
    }

    return _baseNameBuilder.ToString();
}

When I presented Kevin my oldies but goodies solution, he told me that he found a smarter solution. While I was digging into my memories, he kept decompiling the implementation of the Process class and realized that it contains a processInfo private field:

And its internal class exposes a public field called… processName: exactly what we needed!

So I was ready to implement a reflection-based solution like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


private static Type _processInfoType = null;
private static FieldInfo _processNameField = null;
public static string GetProcessNameByReflection(Process p)
{
    var processInfoField = typeof(System.Diagnostics.Process)
        .GetField("processInfo", BindingFlags.Instance | BindingFlags.NonPublic);
    var processInfo = processInfoField.GetValue(p);

    if (_processInfoType == null)
    {
        _processInfoType = processInfo.GetType();
        _processNameField = _processInfoType.GetField("processName");
    }

    return _processNameField.GetValue(processInfo).ToString();
}

And Kevin was able to give me a definitively smarter solution based on compiled expressions:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


private static Func<Process, string> GetProcessNameAccessor()
{
    var param = Expression.Parameter(typeof(Process), "arg");
    var processInfoMember = Expression.Field(param, "processInfo");
    var processNameMember = Expression.Field(processInfoMember, "processName");

    var lambda = Expression.Lambda(typeof(Func<Process, string>), processNameMember, param);

    return (Func<Process, string>)lambda.Compile();
}

private static readonly Func<Process, string> _getProcessNameFunc = GetProcessNameAccessor ();
public string GetProcessNameByExpression()
{
    return _getProcessNameFunc(_process);
}

However, when I tested them on my laptop, I got null reference exceptions while it was working fine on Kevin’s machine… There was a tiny difference between us: I was calling Process.GetProcessById while Kevin was using Process.GetProcesses to get the Process instance. It looks like the implementation of both methods is not doing the same initialization. This kind of things happen when you are trying to use undocumented implementation details… Also note that the .NET Core implementation is different (and does not contain the string length bug).

Comparing the different solutions

So which solution should I pick for our metrics collector?

It’s time to do some benchmarking thanks to BenchmarkDotNet and the results give different order of magnitude!

The winner is without contest based on expressions and the worst one is… the initial implementation that does not even fall into the error case during our tests!

After updating the implementation with the compiled expression-based solution, the CPU usage of our metrics collector seems more reasonable:

It is now a good time to go back home… to write this article :^)

RyuJIT and the never-ending ThreadAbortException

Thu, 06 Apr 2017 13:11:29 +0000

When you see this, you know for sure that something is wrong with a server:

This chart counts the number of first-chance exceptions thrown by the server. We have here an average of 840K exceptions thrown per minute, or 14K exceptions per second. That’s a lot, especially considering that this server only processes about 400 requests per second. Impossible to find anything meaningful or even related in the logs, and the server seems to respond properly to requests: head scratching situation for sure.

The ratio between processed requests and exceptions thrown does not make any sense. It would be great if we could see what are these unbelievable exceptions with our own eyes. Thankfully, the procdump tool from Sysinternals lists the first chance exceptions with the following magical command line:

1

procdump -ma -e 1 -f E0434352.CLR

Instead of capturing a memory dump, the tool will display all CLR related exceptions in the console as they are thrown. We expected a never ending flow of exception details but… For some reason, procdump seemed to completely slow down the process, generating an output polluted by timeout exceptions, that supposedly aren’t thrown when procdump isn’t attached.

It was the time for drastic measures: remote debugging the faulty process with Visual Studio could give us hints about what was going on. This time, we were able to catch a ThreadAbortException we never saw before. Retrying a few times helped us to confirm that it was indeed the culprit.

The likely culprit

As good .NET citizen, we know that it is bad not recommended to call Thread.Abort but to be sure, we did a search on our code, to discover… that we never abort any thread. We started to search by dichotomy any code change since the last known good run of the application. We found a fix for a code that was using Thread.Abort: maybe the server was still running with the old code! We checked the version of the assembly and double-checked by decompiling it (just in case) but it was the fixed code without any Thread.Abort.

Back to scratching our head, we kept Visual Studio remotely attached to the process and thankfully, we managed to catch a CannotUnloadAppDomainException from time to time. As the name indicates, this exception is thrown by the AppDomain when it doesn’t manage to unload in a timely fashion. It turns out that unloading an AppDomain is a known cause ofThreadAbortException, so we’ve got a serious lead.

That ASP.NET would try to unload an appdomain isn’t surprising. It can occur, for instance, when a configuration file is modified. But it’s certainly not enough to explain the 14K exceptions per second, unless we’re continuously creating and unloading new appdomains.

Depressingly, we also managed to explain why we weren’t getting anything in the log files. For performance and scalability reason, our logging API is asynchronous: the logger thread responsible for saving the traces was actually one of the first threads being aborted, thus killing our main source of information. Speak about bad luck.

Back to the problem and digging further into the callstacks, we found out that the exception was always thrown in the same method:

Program.cs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


public void SomeMethod()
{
   while (someCondition)
   {
      try
      {
         // Do some stuff

         // Sleep 15 minutes
         Thread.Sleep(15 * 60 * 1000);
      }
      catch (Exception ex)
      {
         // Log the exception
      }
   }
}

Did we find the culprit? This loop was a good candidate to explain the repeated exceptions: if an exception is thrown then we will go to the catch block, to immediately try again. However, ThreadAbortException is a weird beast: whenever caught, it will automatically be rethrown at the end of the catch block. So in our case, it means that we would leave the “while” block when the exception is rethrown behind our back, and exit the thread. There was no way the code could loop to generate the expected 14k exceptions per second!

The only other option was that appdomains would be created and unloaded at a crazy pace, but why?

After turning around for a while, we noticed something peculiar: the ThreadAbortException was always thrown in the same thread; just like if the code was really looping despite what we mentioned earlier…. What’s going on?

The plot twist

When you’ve excluded every other possible cause, you start doubting of your own knowledge. And so we’ve put together a small program to test the behavior of the ThreadAbortException:

Program.cs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


static void Main(string[] args)
{
   var mutex = new ManualResetEventSlim();
   var t = new Thread(() =>
   {
      while (true)
      {
         try
         {
            if (!mutex.IsSet)
            {
               mutex.Set();
            }

            // Do some stuff
            Thread.Sleep(100);
         }
         catch (Exception ex)
         {
            Console.WriteLine("Exception: " + ex.Message);
         }
      }
   });

   t.Start();

   // Wait for the thread to start
   mutex.Wait();

   t.Abort();

   Console.ReadLine();
}

We start a background thread, doing some fake work in a loop, catching any thrown exception. The main thread then waits for the background thread to start before asking the CLR to stop it by calling Thread.Abort. Following the abort request, a ThreadAbortException will be magically thrown in the context of the background thread. The exception will be caught there, logged in the console, then automatically be rethrown because of its special nature. Then the thread will exit because the exception is rethrown outside of the try/catch block.

Compile, run, and lo and behold…

It seems everything we knew about ThreadAbortException is actually wrong!

The behavior can only be reproduced in a release build while , and running in 64 bit. That last part tipped us towards suspecting the JIT. And indeed, disabling RyuJIT in the configuration seems to fix the issue:

App.config

1
2
3
4
5
6


xml version="1.0" encoding="utf-8"?>
<configuration>
  <runtime>
    <useLegacyJit enabled="1" />
  runtime>
configuration>

How could the JIT be involved? The call to Thread.Abort is in fact asynchronous and sets the AbortRequested flag on the Thread object. The CLR will throw a ThreadAbortException as soon as it is reaching a safe place as Jeffrey Richter explains page 580 of his “Clr via C#” book. The code responsible for throwing the exception at the right time is generated by the JIT compiler.

A JIT bug! After having notified Microsoft, we still had to fix our code. We just need to catch the ThreadAbortException and… rethrow it because it is not done for us by the JIT generated code.

In the end…

Sometimes, the odds are really against you when investigating an issue. Starting with Procdump changing the behavior of the process and the logger crashing during the appdomain unloading, to end up with a bug in the least visible part of the .NET framework. During those times, it is important to keep a clear head, have a few coworkers around to bounce ideas on, and methodically test every single assertion you make, no matter how confident you feel about it.

Co-authored with Kevin Gosse