In my previous posts, I explained how to use DIA and DbgHelp to map a method to its line in source code. I forgot to mention that it was correct for .NET Core but not for the “old” .NET Framework Windows PDB format. Instead of encoding the method token in the name, the symbol file contains the name of the methods. So, how to do the mapping for .NET Framework assemblies? You will find the answer (plus some tricks) in this article.

When I started to work on the support of the old Windows PDB format, I looked at what existed to parse the raw format and… I decided to try DbgHelp instead. With this first implementation, I realized that some token where missing and source code information was not retrieved for most of the methods.

So, I looked for another API to use and I found ISymUnmanagedReader. The usage philosophy is totally different from DIA or DbgHelp.

A little bit of magic

This interface is implemented in diasymreader.dll that comes with every .NET Framework installation. But you need to do COM magic to get it. After having called CoInitialize to setup COM, you ask for an instance of ISymUnmanagedBinder from CLSID_CorSymBinder_SxS:

1
2
3
4
5
6
7
8

CComPtr<ISymUnmanagedBinder> pBinder;
hr = CoCreateInstance(
    CLSID_CorSymBinder_SxS,
    NULL,
    CLSCTX_INPROC_SERVER,
    IID_ISymUnmanagedBinder,
    (void**)&pBinder
);

From the binder, you can get the ISymUnmanagedReader interface corresponding to the assembly you are interested in with GetReaderForFile. However, there are two tiny details to consider.

First, one parameter expects the path to the assembly, not to the .pdb file. That symbol file has to be stored in the same folder but note that the documentation states that you could have more flexible search with ISymUnmanagedBinder2::GetReaderForFile2 but I did not test it.

The second detail is the first parameter: an instance of IMetaDataImport for the same assembly. The steps to get it are… complicated.

Hosting the CLR

The idea is to host the .NET Framework and get the corresponding ICLRMetaHost interface:

1
2

CComPtr<ICLRMetaHost> pMetaHost;
HRESULT hr = CLRCreateInstance(CLSID_CLRMetaHost, IID_ICLRMetaHost, (void**)&pMetaHost);

Calling the CLRCreateInstance API allows you to get an instance of ICLRMetaHost from which you could enumerate installed version of .NET Framework. In my case, I know which version I want:

1
2
3

// Get the installed .NET Framework runtime (v4.0+)
CComPtr<ICLRRuntimeInfo> pRuntimeInfo;
hr = pMetaHost->GetRuntime(L"v4.0.30319", IID_ICLRRuntimeInfo, (void**)&pRuntimeInfo);

The ICLRRuntimeInfo interface allows you to get access to runtime services via GetInterface:

1
2

CComPtr<IMetaDataDispenser> pDispenser;
hr = pRuntimeInfo->GetInterface(CLSID_CorMetaDataDispenser, IID_IMetaDataDispenser, (void**)&pDispenser);

The service I’m interested in is the IMetadataDispenser interface that allows you to “open a scope” on the assembly you are interested in:

1
2
3
4
5
6

hr = pDispenser->OpenScope(
    wModulePath.c_str(),
    ofRead,
    IID_IMetaDataImport,
    (IUnknown**)&_pMetaDataImport
);

Note that the first parameter is the path to the assembly not to the .pdb file. The scope is abstracted by an IMetadataImport interface I have already described and that is needed to call GetReaderForFile: and get the ISymUnmanagedReader:

hr = pBinder->GetReaderForFile(_pMetaDataImport, wModulePath.c_str(), nullptr, &_pReader);

The road to get symbol details for a method

The ISymUnmanagedReader interface implements GetMethod to get details about a given method token via an ISymUnmanagedMethod interface. So, the next question is how to get these tokens. If you remember the previous article, these tokens are from the 06 MethodDef table in the assembly metadata; starting from 06000001 to the last one.

This means that you could write a simple loop starting from 1 up to a hardcoded maximum value, call TokenFromRid(index, mdtMethodDef) to get the corresponding token. However, since you are a professional developer, you would search for the exact number of tokens from IMetadataTables retrieved from IMetadataImport:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

ULONG cRows = 0;

// Get IMetaDataTables interface to query the MethodDef table
CComPtr<IMetaDataTables> pTables;
hr = _pMetaDataImport->QueryInterface(IID_IMetaDataTables, (void**)&pTables);
if (FAILED(hr) || pTables == nullptr)
{
    cRows = LAST_METHODDEF_TOKEN;
}
else
{
    // Get the number of rows in the MethodDef table (table index 0x06 = Method)
    hr = pTables->GetTableInfo(
        0x06,           // MethodDef table
        NULL,           // cbRow (not needed)
        &cRows,         // pcRows (number of methods)
        NULL,           // pcCols (not needed)
        NULL,           // piKey (not needed)
        NULL            // ppName (not needed)
    );

    if (FAILED(hr))
    {
        cRows = LAST_METHODDEF_TOKEN;
    }
}

Now that you have the number of rows (i.e. number of methods defined in the metadata), it is easy and safe to get method information from symbols:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

for (uint32_t i = 1; i <= cRows; i++)
{
    mdMethodDef token = TokenFromRid(i, mdtMethodDef);

    CComPtr<ISymUnmanagedMethod> pMethod;
    hr = _pReader->GetMethod(token, &pMethod);
    if (SUCCEEDED(hr) && pMethod != nullptr)
    {
        MethodInfo info;
        if (GetMethodInfoFromSymbol(pMethod, info))
        {
            _methods.push_back(info);
        }
    }
}

Note that GetMethod might fail (returning E_FAIL) for P/Invoked functions, abstract methods, or methods decorated with DebuggerHidden attribute.

Give me line and source code!

For the other methods with symbol information, you can get its token via the GetToken method. The ISymUnmanagedMethod interface allows low level access to line/column mapping that is beyond the scope of this article. At a high level, positions in source file are named sequence points. Call GetSequencePointCount to get… the number of sequence points for a given method.

The next step is to call GetSequencePoints with the number of points you want and the corresponding arrays of offsets, lines, columns, end lines, end columns and ISymUnmanagedDocument. In my case, I’m only interested in where the method starts so the first sequence point is good enough:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

// Get sequence points (source line information)
ULONG32 cPoints = 0;
hr = pMethod->GetSequencePointCount(&cPoints);
if (SUCCEEDED(hr) && (cPoints > 0))
{
    cPoints = 1; // We only need the first sequence point for start line
    std::vector<ULONG32> offsets(cPoints);
    std::vector<ULONG32> lines(cPoints);
    std::vector<ULONG32> columns(cPoints);
    std::vector<ULONG32> endLines(cPoints);
    std::vector<ULONG32> endColumns(cPoints);
    std::vector<ISymUnmanagedDocument*> documents(cPoints);

    ULONG32 actualCount = 0;
    hr = pMethod->GetSequencePoints(
        cPoints,
        &actualCount,
        &offsets[0],
        &documents[0],
        &lines[0],
        &columns[0],
        &endLines[0],
        &endColumns[0]
    );

    if (SUCCEEDED(hr) && (actualCount > 0))
    {

The source file is described by ISymUnmanagedDocument that provides its name when GetURL is called:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

// Get the first sequence point's document and line
        ISymUnmanagedDocument* pDoc = documents[0];
        if (pDoc != nullptr)
        {
            // Get document URL (file path)
            ULONG32 urlLen = 0;
            hr = pDoc->GetURL(0, &urlLen, NULL);
            if (SUCCEEDED(hr) && (urlLen > 0))
            {
                std::vector<WCHAR> url(urlLen);
                hr = pDoc->GetURL(urlLen, &urlLen, &url[0]);
                if (SUCCEEDED(hr))
                {
                    // Convert wide string to narrow string
                    int len = WideCharToMultiByte(CP_UTF8, 0, &url[0], urlLen, NULL, 0, NULL, NULL);
                    std::string narrowUrl(len, '\0');
                    WideCharToMultiByte(CP_UTF8, 0, &url[0], urlLen, &narrowUrl[0], len, NULL, NULL);
                    info.sourceFile = narrowUrl;
                }
            }

            // NOTE: 0xFEEFEE is a special value indicating hidden lines
            info.lineNumber = lines[0];
            pDoc->Release();
        }
    }
}

The final interesting trick is that the line number might have the special 0xFEEFEE value. It means that the line is hidden. I have seen it for methods generated by the C# compiler such as MoveNext for async state machines or anonymous methods:

The source code is available from my Github repository.

Happy coding!

References

• Archived Microsoft-pdb repository
• DIA implementation article
• DbgHelp implementation article

Introduction

In our Datatog continuous .NET profiler implementation, we collect the call stack of a thread when something interesting happens such as an exception is thrown for example. In addition to the method name we would like to figure out at what line in which source code file this method is implemented.

This information is usually stored in the program database (.pdb) file that is generated by the compiler when the assembly is generated from the source code. The type and the name of the method are stored in the metadata of the assembly itself but I already told this story before. The .NET compilers support two formats of .pdb: the Portable format for .NET Core and the Windows format for .NET Framework.

I’ve explained how to use the DIA API and it is now time to show how to leverage the DbgHelp API that is available on all Windows machines (even though it is recommended to always install the latest release via the Debugging Tools for Windows.

This time, my goal is to extract from a Windows .pdb file the source code and line information for a given managed method. You can find the corresponding source code of this DumpLine tool in my Github repository.

Starting with DbgHelp

There are two major ways to get access to symbols with DbgHelp: either from a running process (i.e. to map the currently loaded .dll to their associated .pdb files) or from a tool that would explicitly load a .dll or a .pdb file.

Before anything, you tell DbgHelp which options you want by calling SymSetOptions:

1
2
3
4
5
6
7
8

DWORD options = SymGetOptions();
    options |= SYMOPT_DEBUG;
    options |= SYMOPT_LOAD_LINES;           // Load line number information
    options |= SYMOPT_UNDNAME;              // Undecorate symbol names
    //options |= SYMOPT_DEFERRED_LOADS;       // Defer symbol loading
    options |= SYMOPT_EXACT_SYMBOLS;        // Require exact symbol match
    options |= SYMOPT_FAIL_CRITICAL_ERRORS; // Don't show error dialogs
    SymSetOptions(options);

This is where I ask that line number information should be collected.

The next step is to call SymInitialize to setup DbgHelp environment. The first parameter expects a process handle (returned by GetCurrentProcess in my case). You could pass a path where to find the .pdb files for your dlls as a second parameter. In my case, since I will provide a .pdb file path, I don’t need it, and NULL will be passed. It means that, if needed, DbgHelp will use the current folder and the path set in _NT_SYMBOL_PATH and _NT_ALTERNATE_SYMBOL_PATH environment variables.

The last boolean parameter tells DbgHelp if you want that SymLoadModule64 to be called for each and every loaded .dll in the given process. Definitively not what I want so I’m passing FALSE.

1
2
3
4
5

_hProcess = GetCurrentProcess();
    if (!SymInitialize(_hProcess, NULL, FALSE))
    {
        _hProcess = NULL;
    }

At that point, I’m ready to load a .pdb file.

Loading a .pdb file

The API is straightforward: just call SymLoadModuleEx :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

_baseAddress = SymLoadModuleEx(
        _hProcess,
        NULL,
        pdbFilePath.c_str(),
        NULL,
        0x10000000, // arbitrary base address
        0,
        NULL,
        0
    );

    if (_baseAddress == 0)
    {
        return false;
    }

The important parameters are the process handle (same as for SymInitialize) and the path of the .pdb file. I’ve lost some time trying to understand why my code was not working due to a weird behavior of this function. You know that it succeeds when the returned address is not 0. Well… This is not 100% correct. If the path you provide does not exist, you won’t get 0 but the base address that you also provide. Even worth, when you call the functions I’ll detail later on, no error will happen but nothing will work as expected. So, I simply check that the file exists:

1
2
3
4
5

// BUG? : dbghelp does not fail if the .pdb file does not exist...
    if (GetFileAttributesA(pdbFilePath.c_str()) == INVALID_FILE_ATTRIBUTES)
    {
        return false;
    }

Note that it is possible to unload the symbols of a given loaded module by calling SymUnloadModule with the same process handle and its base address: this will reduce the memory consumption if you don’t need the symbols anymore.

In case of deferred load symbols, it is needed to call SymGetModuleInfo64 before trying to access the symbols:

1
2
3
4
5
6

IMAGEHLP_MODULE64 moduleInfo = { 0 };
    moduleInfo.SizeOfStruct = sizeof(IMAGEHLP_MODULE64);
    if (!SymGetModuleInfo64(_hProcess, _baseAddress, &moduleInfo))
    {
        return false;
    }

In addition, this will fill up an IMAGEHLP_MODULE64 structure with possibly interesting details:

The .pdb signature and age could be useful to build urls to communicate with symbol servers; but this is another story:

1
2
3
4
5
6
7
8
9

_age = moduleInfo.PdbAge;
    GUID guid = moduleInfo.PdbSig70;
    char strGUID[80];
    sprintf_s(strGUID, 80, "%08x%04x%04x%02x%02x%02x%02x%02x%02x%02x%02x",
        guid.Data1, guid.Data2, guid.Data3,
        guid.Data4[0], guid.Data4[1], guid.Data4[2], guid.Data4[3],
        guid.Data4[4], guid.Data4[5], guid.Data4[6], guid.Data4[7]
        );
    _guid = strGUID;

You also know if line numbers are available or not thanks to the LineNumbers field.

Note that if you asked for deferred symbols option, you won’t get any interesting details:

Only the module name and path are provided but nothing else.

Enumerating the methods

It is now time to iterate on the symbols in the loaded .pdb thanks to SymEnumSymbols:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

if (!SymEnumSymbols(
            _hProcess,
            _baseAddress,
            "*!*",  // Mask (all symbols)
            EnumMethodSymbolsCallback,
            this    // User context to store the methods in _methods instance field
    ))
    {
        return false;
    }

In addition to the obvious parameters, this function expects a callback function that will be called for each symbol in the module specified by the process handle and the base address. Note that you can pass any context as the last parameter. In my case, the instance of my DbgHelpParser class is passed to be able to store the methods in a dedicated _methods field:

1
2
3
4
5
6
7
8
9

struct MethodInfo
{
    std::string name;
    uint64_t address;
    uint32_t size;
    std::string sourceFile;
    uint32_t lineNumber;
};
std::vector<MethodInfo> _methods;

The “*!*” mask tells DbgHelp to look for symbols in all modules. This might sound counter intuitive, but the syntax is similar to what you find in WinDBG or Visual Studio: !. This could be useful if you load more than one .pdb.

The job of the callback function is to detect the symbols you are interested in from the SYMBOL_INFO structure passed for each matching symbol:

1
2
3
4
5
6
7
8
9

BOOL CALLBACK DbgHelpParser::EnumMethodSymbolsCallback(PSYMBOL_INFO pSymInfo, ULONG SymbolSize, PVOID UserContext)
{
    DbgHelpParser* parser = reinterpret_cast<DbgHelpParser*>(UserContext);

    if (
        (pSymInfo->Tag == SymTagFunction) &&
        ((pSymInfo->Flags & (SYMFLAG_CLR_TOKEN | SYMFLAG_METADATA)) == (SYMFLAG_CLR_TOKEN | SYMFLAG_METADATA))
        )
    {

The Tag field contains a value from SymTagEnum but, for a managed .pdb file, you will only get SymTagFunction. Also, the Flags field should contain SYMFLAG_CLR_TOKEN and SYMFLAG_METADATA because we are only interested in managed methods.

Next, you get the name, address and size from other fields before looking for the source file and line details by calling SymGetLineFromAddr64:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

MethodInfo info;
        info.name = pSymInfo->Name;
        info.address = pSymInfo->Address;
        info.size = pSymInfo->Size;

        // Try to get source file and line information
        IMAGEHLP_LINE64 line = { 0 };
        line.SizeOfStruct = sizeof(IMAGEHLP_LINE64);
        DWORD displacement = 0;

        if (SymGetLineFromAddr64(parser->_hProcess, pSymInfo->Address, &displacement, &line))
        {
            info.sourceFile = line.FileName ? line.FileName : "";
            info.lineNumber = line.LineNumber;
        }
        else {
            info.sourceFile = "";
            info.lineNumber = 0;
        }

        parser->_methods.push_back(info);
    }

    return TRUE; // Continue enumeration
}

The callback returns TRUE to continue the enumeration. You could return FALSE if you would look for specific symbols and wanted to speed up the processing.

The managed side of the story

When the tool is run on a managed assembly, you get the following kind of output:

The name of the method does not match at all with the names in my test assembly! Why do all methods have this generic Method.# format?

Well… the number corresponds to the RID of the corresponding method in the metadata of the assembly. Let’s have a look at what the MethodDef metadata table of this assembly looks like in ILSpy:

The RID column corresponds to the number in the name in the tool output. So, the Method.#3 is the get_Records property getter in PDBFormatReaderTest.cs:28. And this is exactly what I can see in the test source code:

You can also check that the lines look correct compared to what is listed by the tool:

• FilePath getter at line 19
• FilePath setter at line 20
• Records getter at line 28
• Records setter at line 29

Feel free to look at the source code from my Github repository.

DbgHelp provides many more services to look into symbols but that’s all for today!

During this R&D week at Datadog, I wanted to implement a tool accepting a .pdb file and generate a .sym file listing functions symbols with their address, size, name with signature and if they are public or private. This post dig into the implementation details of using Microsoft Debug Interface Access (DIA) COM API to achieve these objectives. If you want to see what my vibe coding experience in Cursor was, read this other post instead.

One self-contained tool please!

I would like the tool to be self-contained but since DIA is based on a COM server, it would require registering msdia40.dll on the machine. Not a good idea. In case the dll is in the same folder as the tool, one could “emulate” the magic done by CoCreateInstance to get an instance of IDiaDataSource (more on this interface soon) by:

• Call LoadLibrary to load the dll in memory
• Call GetProcAddress to get the DllGetClassObject implementation
• Call this function to get the IClassFactory implementation
• Call its CreateInstance method to get an object implementing IDiaDataSource

Here is the corresponding code (without error checking for readability)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

// Create DIA data source without registration using DLL loading
HRESULT PdbSymbolExtractor::NoRegCoCreate(const std::wstring& dllPath, REFCLSID rclsid, REFIID riid, void** ppv) {
    HMODULE hDll = LoadLibraryW(dllPath.c_str());

    typedef HRESULT(__stdcall* DllGetClassObjectFunc)(REFCLSID, REFIID, LPVOID*);
    DllGetClassObjectFunc pDllGetClassObject = (DllGetClassObjectFunc)GetProcAddress(hDll, "DllGetClassObject");

    CComPtr<IClassFactory> pClassFactory;
    HRESULT hr = pDllGetClassObject(rclsid, IID_IClassFactory, (void**)&pClassFactory);

    hr = pClassFactory->CreateInstance(NULL, riid, ppv);

    // Note: We intentionally don't call FreeLibrary here because the DLL needs to stay loaded
    // The COM object references will keep it alive
    return S_OK;

This function is called with the path of msdia40.dll and the UUID of the expected IDiaDataSource:

1
2

// Create DIA data source without registration
hr = NoRegCoCreate(dllPath, CLSID_DiaSource, __uuidof(IDiaDataSource), (void**)&_pDiaDataSource);

But still, I don’t want to have two binaries!

The trick is to embed the msdia40.dll inside the tool as a Windows resource. In the .rc file, add an RCDATA entry that points to the dll:

1

IDR_MSDIA_DLL      RCDATA      "x64\\Release\\msdia140.dll"

You should see it in the Resource View in Visual Studio:

Here is the code that extracts it as a file on disk is straightforward (error checking has been removed for readability):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

// Extract embedded msdia140.dll from resources
bool PdbSymbolExtractor::ExtractEmbeddedDll(const std::wstring& outputPath) {
    // Find the resource
    HMODULE hModule = GetModuleHandle(NULL);
    HRSRC hResource = FindResource(hModule, MAKEINTRESOURCE(IDR_MSDIA_DLL), RT_RCDATA);

    // Load the resource
    HGLOBAL hLoadedResource = LoadResource(hModule, hResource);

    // Lock the resource to get a pointer to the data
    LPVOID pResourceData = LockResource(hLoadedResource);

    // Get the size of the resource
    DWORD resourceSize = SizeofResource(hModule, hResource);

    // Write the DLL to disk
    std::ofstream outFile(outputPath, std::ios::binary);
    outFile.write(static_cast<const char*>(pResourceData), resourceSize);
    outFile.close();

    return true;
}

Where are my function symbols?

After calling NoRegCoCreate(), _pDiaDataSource stores a reference to the entry point into the DIA APIs. Here are the steps to follow before being able to list the symbols:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

HRESULT PdbSymbolExtractor::ExtractSymbolsFromPdb(const std::wstring& pdbPath, std::vector<FunctionSymbol>& symbols) 
{
    // Load the PDB file
    HRESULT hr = _pDiaDataSource->loadDataFromPdb(pdbPath.c_str());

    // Open a session
    CComPtr<IDiaSession> pSession;
    hr = _pDiaDataSource->openSession(&pSession);

    // Get the global scope
    CComPtr<IDiaSymbol> pGlobal;
    hr = pSession->get_globalScope(&pGlobal);

Now, you have the global scope of the symbols, you can ask for an enumerator for the type of symbols you are interested in; SymTagFunction in my case:

1
2
3
4
5
6
7

// Enumerate all function symbols
    CComPtr<IDiaEnumSymbols> pEnumSymbols;
    hr = pGlobal->findChildren(SymTagFunction, NULL, nsNone, &pEnumSymbols);

    LONG count = 0;
    pEnumSymbols->get_Count(&count);
    std::wcout << L"Found " << count << L" function symbols" << std::endl;

The pEnumSymbols iterator allows you to loop on each SymTagFunction symbol and get its name:

1
2
3
4
5
6
7
8
9

while (SUCCEEDED(pEnumSymbols->Next(1, &pSymbol, &celt)) && celt == 1) {
        FunctionSymbol func;

        // Get function name
        BSTR bstrName;
        if (pSymbol->get_name(&bstrName) == S_OK) {
            func.name = bstrName;
            SysFreeString(bstrName);
        }

Note that each symbol details are stored in a FunctionSymbol instance:

1
2
3
4
5
6
7
8

struct FunctionSymbol {
    std::wstring name;
    ...
    std::wstring signature; // Function signature (parameters only, no return type)
    DWORD rva;              // Relative Virtual Address
    ULONGLONG length;
    bool isPublic;
};

with the rest of the code in the while() loop:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

// Get function signature (parameters only, no return type)
    func.signature = ExtractFunctionSignature(pSymbol);

    // Get relative virtual address
    DWORD rva;
    if (pSymbol->get_relativeVirtualAddress(&rva) == S_OK) {
        func.rva = rva;
    }

    // Get function length
    ULONGLONG length;
    if (pSymbol->get_length(&length) == S_OK) {
        func.length = length;
    }

    // Determine if function is public or private
    // Check access level - default to private
    func.isPublic = false;
    DWORD access;
    if (pSymbol->get_access(&access) == S_OK) {
        func.isPublic = (access == CV_public);
    }

    pSymbol.Release();

I did not have the time to do more trial for private/public state, but I should have tried by enumerating SymTagPublicSymbol or SymTagExport that could be considered as public.

Better with a signature

The final step is to figure out the signature of each function. This is where the genericity of DIA could be confusing because so many things are represented by IDiaSymbol: a symbol, a function, the type of a function, or the type of a parameter…

So, the type of the function is retrieved as an IDiaSymbol by calling getType() on the function symbol. From that IDiaSymbol, findChildren() lets you iterate on the parameters:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

// Extract function signature (parameters only, no return type)
std::wstring PdbSymbolExtractor::ExtractFunctionSignature(IDiaSymbol* pSymbol) {
    if (!pSymbol) {
        return L"()";
    }

    // Get function type
    CComPtr<IDiaSymbol> pFunctionType;
    if (pSymbol->get_type(&pFunctionType) != S_OK || !pFunctionType) {
        return L"()";
    }

    // Enumerate function arguments
    CComPtr<IDiaEnumSymbols> pEnumArgs;
    if (FAILED(pFunctionType->findChildren(SymTagFunctionArgType, NULL, nsNone, &pEnumArgs))) {
        return L"()";
    }

    LONG argCount = 0;
    pEnumArgs->get_Count(&argCount);

    if (argCount == 0) {
        return L"()";
    }

Now, the same Next() method is called on the enumerator to iterate on each parameter:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

// Build signature string
    std::wstring signature = L"(";
    CComPtr<IDiaSymbol> pArg;
    ULONG argCelt = 0;
    bool first = true;

    while (SUCCEEDED(pEnumArgs->Next(1, &pArg, &argCelt)) && argCelt == 1) {
        if (!first) {
            signature += L", ";
        }
        first = false;

        // Get the argument type
        CComPtr<IDiaSymbol> pArgType;
        if (pArg->get_type(&pArgType) == S_OK && pArgType) {
            signature += GetTypeName(pArgType);
        } else {
            signature += L"?";
        }

        pArg.Release();
    }

    signature += L")";
    return signature;
}

The final step is to get the name of the type from the IDiaSymbol returned by get_type(). If it is a custom type, call get_name() like any other symbol. Otherwise, for basic types, call get_baseType() and get_length() as shown by the code below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

std::wstring PdbSymbolExtractor::GetTypeName(IDiaSymbol* pType) {
    if (!pType) {
        return L"?";
    }

    // Try to get type name directly
    BSTR bstrTypeName;
    if (pType->get_name(&bstrTypeName) == S_OK && bstrTypeName && wcslen(bstrTypeName) > 0) {
        std::wstring typeName = bstrTypeName;
        SysFreeString(bstrTypeName);
        return typeName;
    }

    // For basic types or unnamed types, try getting basic type info
    DWORD baseType = 0;
    ULONGLONG length = 0;

    if (pType->get_baseType(&baseType) == S_OK) {
        pType->get_length(&length);

        // Map basic types to names
        switch (baseType) {
            case btVoid: return L"void";
            case btChar: return L"char";
            case btWChar: return L"wchar_t";
            case btBool: return L"bool";

            case btInt:
            case btLong:
                if (length == 1) return L"char";
                else if (length == 2) return L"short";
                else if (length == 4) return L"int";
                else if (length == 8) return L"__int64";
                else return L"int" + std::to_wstring(length * 8);

            case btUInt:
            case btULong:
                if (length == 1) return L"unsigned char";
                else if (length == 2) return L"unsigned short";
                else if (length == 4) return L"unsigned int";
                else if (length == 8) return L"unsigned __int64";
                else return L"uint" + std::to_wstring(length * 8);

            case btFloat:
                if (length == 4) return L"float";
                else if (length == 8) return L"double";
                else return L"float" + std::to_wstring(length * 8);

            default:
                return L"?";
        }
    }

    return L"?";
}

This is a “simple” implementation that does not take pointers, addresses, arrays, and more into account. For a more complete solution, I would recommend looking at the PrintType() implementation in the DIA2Dump code sample that is installed with Visual Studio.

I hope this will get your foot in the door of symbol parsing and make you want to dig further into DIA.

References

• Corresponding source code is available in my github repository.
• Archived Microsoft documentation/implementation of .pdb format including a symbol dumper code.
• DIA2Dump Visual Studio code sample.

In the previous article, I presented what is needed (i.e. listen to WaitHandleWait events) to compute lock/wait durations and call stacks for Mutex, Semaphore, SemaphoreSlim, Manual/AutoResetEvent, ManualResetEventSlim, ReaderWriterLockSlim .NET synchronization constructs for a running process.

However, since the application is already running, some JIT-related events are missing, and some frames of the call stacks cannot be symbolized. Also, it would be great to monitor an application’s startup to see if it could be faster.

This post will detail how to monitor a .NET application since the very beginning of its life and the issues you might face.

Preparing a new .NET process to be monitored

From .NET 5, the dotnet-trace CLI tool allows you to pass a command line to execute and trace it from startup. In a very interesting article, Olivier Coanet presented the gory details about how to tell the .NET runtime to start an application in a pseudo-suspended mode as shown in the following diagram:

The first step is to create a ReverseDiagnosticsServer instance with a specific port (i.e. dotnet-wait_1234 in the diagram). Next, the process to monitor is spawned with the DOTNET_DiagnosticPorts environment variable set to the same port (i.e. dotnet-wait_1234). Look at the Diagnostics documentation of the Diagnostic Ports with DOTNET_DiagnosticPorts environment variable for more details. The .NET runtime is the new process will listen to this port and… wait.

When the tool is ready, it sends a resume command via a DiagnosticsClient: from that point in time, the CLR executes the normal flow of actions to run the application and… you will receive all events without missing one!

Get my command line please

Following the dotnet-trace example, my dotnet-wait tool accepts the command line of the child process in its final arguments that follow the** — **trigger. For example, dotnet-wait — dotnet foo.dll will start the program in foo.dll by using dotnet.exe. I’m reusing the code in ReversedServerHelper.cs to deal with arguments containing spaces:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

else if (current == "--")  // this is supposed to be the last one
{
    i++;
    if (i < args.Length)
    {
        parameters.pathName = args[i];

        // use the remaining arguments as the arguments for the child app to spawn
        i++;
        if (i < args.Length)
        {
            parameters.arguments = "";
            for (int j = i; j < args.Length; j++)
            {
                if (args[j].Contains(' '))
                {
                    parameters.arguments += $"\"{args[j].Replace("\"", "\\\"")}\"";
                }
                else
                {
                    parameters.arguments += args[j];
                }

                if (j != args.Length)
                {
                    parameters.arguments += " ";
                }
            }
        }

        // no need to look for more arguments
        break;
    }
    else
    {
        throw new InvalidOperationException($"Missing path name value...");
    }
}

The code to spawn the child process is simple:

1
2
3
4
5
6
7
8
9

// start the monitored app
var psi = new ProcessStartInfo(pathName);
if (!string.IsNullOrEmpty(arguments))
{
    psi.Arguments = arguments;
}
psi.EnvironmentVariables["DOTNET_DiagnosticPorts"] = port;
psi.UseShellExecute = false;
var process = System.Diagnostics.Process.Start(psi);

Here is an example with the following prompt:

-- dotnet "C:\CommandLineTest.dll" one two "t h r e e" four 'five six'

that generates the output (the test application is just listing its arguments):

dotnet-wait v1.0.0.0 - List wait duration
by Christophe Nasarre

Press ENTER to exit...
6 arguments
   1 | one
   2 | two
   3 | t h r e e
   4 | four
   5 | 'five
   6 | six'

This test reminded me to never use simple quotes in prompts :^)

It’s my console!

Once I implemented these steps, I immediately faced a very simple problem: my dotnet-wait tool and the test application are console applications. It means that they will share the same console for both input and output. For example, both are waiting for the RETURN key to (1) stop for the tool and (2) start for the test application: too bad for me because the tool will stop as soon the application starts…

Going back in time in my Windows memories, I remembered that the Win32 CreateProcess API accepts CREATE_NEW_CONSOLE as creation flag to automagically start the child process into its own new console. Unfortunately, it is not possible to pass this flag in .NET; maybe a limitation due to Linux support.

One simple solution could be to redirect the output of the tool or the application to a file: that would avoid mixing them in the console. Note that, by default, dotnet-trace discards output from the child process (by setting RedirectStandardOutput, RedirectStandardError and RedirectStandardInput to false and by ignoring the error and output streams) except if you pass — show-child-io on the command line. In this case, no output for dotnet-trace.

I decided to do the opposite for dotnet-wait: by default, you also get the child output but you can redirect the output of the tool to a file with -o

. Still, this does not solve the input problem in case of common expected keys.

If you remember the interactions between the tool and the monitored application, the latter is suspended until DiagnosticsClient::ResumeRuntime is called. So, why not starting the tool that spawns the application in one console and another instance of the tool in a new console that will resume the application? This is exactly what my friend Kevin Gosse imagined and how dotnet-wait works.

After the timeout that you give to diagnosticsServer.AcceptAsync(cancellation.Token) has elapsed, the runtime in the child process will display the following message:

The runtime has been configured to pause during startup and is awaiting a Diagnostics IPC ResumeStartup command from a Diagnostic Port.
DOTNET_DiagnosticPorts="dotnet-wait_34296"
DOTNET_DefaultDiagnosticPortSuspend=0

And this is exactly what the -r 34296 parameter will do!

You can now install dotnet wait and monitor the lock and wait contentions of your .NET9+ applications.

Introduction

In an old post, I detailed how to use ContentionStart and ContentionStop events to measure the lock contentions duration for a .NET application. In a .NET 9 pull request, a former Criteo’s colleague Grégoire Verdier has added new events to be notified when wait time similar to lock contention is happening for Mutex, Semaphore, Manual/AutoResetEvent. Read his post for more details about what he was trying to investigate.

With asynchronous and multi-threaded algorithms, it is essential to detect unexpected wait/locks in our applications. This post shows you how to leverage these events to measure the duration of these waits and get the call stack when the wait started:

New WaitHandleWait events

These new events are emitted by the Microsoft-Windows-DotNETRuntime CLR provider when you enable the WaitHandle (= 0x40000000000) keyword with Verbose verbosity. Each time WaitOne is called on a waitable object and this object is already owned, a WaitHandleWaitStart event is emitted. When the object is released, a WaitHandleWaitStop event is emitted.

For example, the following code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

static Mutex mutex = new Mutex();

static void Main()
{
    var owningThread = new Thread(OwningThread);
    owningThread.Start();
    var mutexThread = new Thread(MutexThread);
    mutexThread.Start();

    owningThread.Join();
    mutexThread.Join();
}

static void OwningThread()
{
    Console.WriteLine($"    [{GetCurrentThreadId(), 8}] Start to hold resources");
    Console.WriteLine("___________________________________________");
    mutex.WaitOne();

    Thread.Sleep(3000);  // the wait should last ~3 seconds

    Console.WriteLine("    Release resources");
    mutex.ReleaseMutex();
}

static void MutexThread()
{
    Console.WriteLine($"    [{GetCurrentThreadId(), 8}] waiting for Mutex...");
    mutex.WaitOne();  // events are emitted in the implementation when a contention happens
    mutex.ReleaseMutex();
    Console.WriteLine("    <-- Mutex");
}

generates a Start and Stop events pair:

1
2

125980 | 00000000-0000-0000-0000-000000000000 > event 301 __ [ 1| Start] WaitHandleWait/Start
125980 | 00000000-0000-0000-0000-000000000000 > event 302 __ [ 2|  Stop] WaitHandleWait/Stop

There is no associated activity ID so you rely on the fact that the same waiter thread (125980 in the previous example) is emitting for both events.

Listening to the new Wait events

As usual, you should rely on the TraceEvent nuget to start an EventPipe session with an already running .NET application. The last version already contains the definition of the keyword:

1

keywords |= ClrTraceEventParser.Keywords.WaitHandle; // .NET 9 WaitHandle kind of contention

and the C# events for Start and Stop:

1
2

source.Clr.WaitHandleWaitStart += OnWaitHandleWaitStart;
source.Clr.WaitHandleWaitStop += OnWaitHandleWaitStop;

The handler’s implementation is straightforward. The start of the wait is recorded for the current thread:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

private void OnWaitHandleWaitStart(WaitHandleWaitStartTraceData data)
{
    // get the contention info for the current thread
    ContentionInfo info = _contentionStore.GetContentionInfo(data.ProcessID, data.ThreadID);
    if (info == null)
        return;

    // keep track of the wait start
    info.ContentionStartRelativeMSec = data.TimeStampRelativeMSec;
}

When the wait ends, the duration is computed based on the recorded wait start because it is not provided in the payload like for ContentionStop:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

private void OnWaitHandleWaitStop(WaitHandleWaitStopTraceData data)
{
    ContentionInfo info = _contentionStore.GetContentionInfo(data.ProcessID, data.ThreadID);
    if (info == null)
        return;
    // unlucky case when we start to listen just after the WaitHandleStart event
    if (info.ContentionStartRelativeMSec == 0)
    {
        return;
    }

    // Too bad the duration is not provided in the payload like in ContentionStop...
    var contentionDurationMSec = data.TimeStampRelativeMSec - info.ContentionStartRelativeMSec;
    info.ContentionStartRelativeMSec = 0;
    var duration = TimeSpan.FromMilliseconds(contentionDurationMSec);
    Console.WriteLine($"{e.ThreadId,7} | {e.Duration.TotalMilliseconds} ms");
}

This is nice but it would be more useful if we could get the call stack of long waits.

Call stacks with EventPipe

In a previous post, I explained that it is possible to get the call stack when an event is emitted thanks to the ClrStackWalk event that follows the event you are interested in. Unfortunately, this is not more the case for .NET 5+ that is using EventPipe instead of ETW.

As Olivier Coanet presents in his post, you can get the call stack as an array of addresses from the hidden event record that is mapped by the TraceEvent parameter passed to each event handlers. This EVENT_RECORD structure contains a ExtendedData field that is an array of EVENT_HEADER_EXTENDED_DATA_ITEM:

1
2
3
4
5
6
7
8

public struct EVENT_HEADER_EXTENDED_DATA_ITEM
{
    public ushort Reserved1;
    public ushort ExtType;
    public ushort Reserved2;
    public ushort DataSize;
    public ulong DataPtr;
}

If the ExtType value is EVENT_HEADER_EXT_TYPE_STACK_TRACE64 (=6) then DataPtr points to a EVENT_EXTENDED_ITEM_STACK_TRACE64 structure:

1
2
3
4
5

public struct EVENT_EXTENDED_ITEM_STACK_TRACE64
{
    public ulong MatchId;
    public unsafe fixed ulong Address[1];
}

that contains an array of 64-bit addresses. The size of this array is given by DataSize — sizeof(ulong).

For 32-bit applications, you will get EVENT_HEADER_EXT_TYPE_STACK_TRACE32 (=5) as ExtType value and DataPtr will point to EVENT_EXTENDED_ITEM_STACK_TRACE32:

1
2
3
4
5

public struct EVENT_EXTENDED_ITEM_STACK_TRACE32
{
    public ulong MatchId;
    public unsafe fixed uint Address[1];
}

that stores an array of 32-bit addresses.

Knowing that makes writing the code to get the call stacks as an array of 64-bit addresses (same with 32-bit applications for simplicity sake) pretty straightforward:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

public static EventPipeUnresolvedStack ReadStackUsingUnsafeAccessor(TraceEvent traceEvent)
{
    return GetFromEventRecord(traceEvent.eventRecord);
}

private static EventPipeUnresolvedStack GetFromEventRecord(TraceEventNativeMethods.EVENT_RECORD* eventRecord)
{
    if (eventRecord == null)
        return null;

    var extendedDataCount = eventRecord->ExtendedDataCount;

    for (var dataIndex = 0; dataIndex < extendedDataCount; dataIndex++)
    {
        var extendedData = eventRecord->ExtendedData[dataIndex];
        if (extendedData.ExtType == TraceEventNativeMethods.EVENT_HEADER_EXT_TYPE_STACK_TRACE64)
        {
            var stackRecord = (TraceEventNativeMethods.EVENT_EXTENDED_ITEM_STACK_TRACE64*)extendedData.DataPtr;
            var addresses = &stackRecord->Address[0];
            var addressCount = (extendedData.DataSize - sizeof(UInt64)) / sizeof(UInt64);
            if (addressCount == 0)
                return null;

            var callStackAddresses = new ulong[addressCount];
            for (var index = 0; index < addressCount; index++)
            {
                callStackAddresses[index] = addresses[index];
            }
            return new EventPipeUnresolvedStack(callStackAddresses);
        }
        else if (extendedData.ExtType == TraceEventNativeMethods.EVENT_HEADER_EXT_TYPE_STACK_TRACE32)
        {
            var stackRecord = (TraceEventNativeMethods.EVENT_EXTENDED_ITEM_STACK_TRACE32*)extendedData.DataPtr;
            var addresses = &stackRecord->Address[0];
            var addressCount = (extendedData.DataSize - sizeof(UInt32)) / sizeof(UInt32);
            if (addressCount == 0)
                return null;

            var callStackAddresses = new ulong[addressCount];  // store the 32 addresses as 64 bit addresses
            for (var index = 0; index < addressCount; index++)
            {
                callStackAddresses[index] = addresses[index];
            }

            return new EventPipeUnresolvedStack(callStackAddresses);
        }
    }

    return null;
}

Note that the last version of TraceEvent nuget provides a public access to the eventRecord field so it is no more needed to use the UnsafeAccessor attribute used by Olivier.

Symbolize the call stack addresses

Address is good but the corresponding method name is better. I won’t repeat what I’ve already detailed in an older post that shows how to get the name of a native and managed name from an instruction pointer address. Instead, I want to pinpoint a big limitation of this solution to listen to CLR provider MethodLoadVerbose/MethodDCStartVerboseV2 events. If the methods you are interested in are jitted BEFORE your tool attaches to the application, you will never get these events.

You could get the same mapping address span/method name via the other “Microsoft-Windows-DotNETRuntimeRundown” provider and its MethodDCEndVerbose event that contains the expected MethodStartAddress, MethodSize and MethodName in its payload. But I need this information before the end of the application…

Looking at the documentation, it seems that the rundown provider accepts the StartRundownKeyword value to emit the DCStart events when the provider is enabled! Since .NET 9, it is possible to pass the keywords you want (before, the default value did not contain StartRundownKeyword) when creating the EventPipe session

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

//                V-- this is the default rundown keyword
rundownKeywords = 0x80020139 | (long)ClrTraceEventParser.Keywords.StartEnumeration;
var config = new EventPipeSessionConfiguration(GetProviders(), 256, rundownKeywords, true);
using (var session = client.StartEventPipeSession(config))
{
    var source = new EventPipeEventSource(session.EventStream);
    RegisterListeners(source);

    // this is a blocking call
    source.Process();
}

Note that you should not add the rundown provider to the list passed as parameter.

Unfortunately, there is currently an issue in the runtime since September 2020 that pinpoints this exact problem. I even tried to create and close a session to get the DCStop events before recreating a new one, but I failed.

The next episode will talk about how it is possible to start a .NET application and get the events since its startup… with the problems that are happening.

I’ve presented in depth the events emitted by the CLR in many posts to get insightful details about how the .NET runtime is working (lock contention, GC, allocations, …). Some .NET features are not implemented at the runtime level but at the Base Class Library (a.k.a. BCL) level. For example, if you are using HttpClient, you might want to measure how long it takes to get the response to your HTTP requests.

In this new series, I will describe how the BCL is using EventSource to emit the events, how you can listen to them with TraceEvent; focusing on HTTP requests.

Where do these events come from?

When you look for these BCL events in the documentation, you end up to this well-known event provides in .NET page and in the Framework libraries section. It lists the providers with their name and the emitted events with their keyword and verbosity. However, there is no detail about their payload! For example, for the “System.Net.Http” provider, you know that the RequestStart informal event is emitted each time “an HTTP request has started”. But you don’t know the corresponding url… Let me tell you how to get these tiny details.

Within the BCL, the classes responsible for emitting events derive from EventSource with the “Telemetry” suffix as a naming convention. They are decorated with an EventSource attribute to define their name (used as provider name by a listener) and their Guid that will be provided with each event. For example, here is the declaration of the class responsible for events related to sending HTTP requests:

1
2
3

[EventSource(Name = "System.Net.Http")]
internal sealed partial class HttpTelemetry : EventSource
{

If a Guid is not provided, one is automatically computed (see the corresponding code in Perfview).

Next, public helper methods decorated with a NonEvent attribute are provided to be used in the BCL code when it is needed to emit events. These methods are calling private methods decorated with an Event attribute to define their unique ID and their verbosity level:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

[NonEvent]
public void RequestStart(HttpRequestMessage request)
{
    ...
    RequestStart(
        request.RequestUri.Scheme,
        request.RequestUri.IdnHost,
        request.RequestUri.Port,
        request.RequestUri.PathAndQuery,
        (byte)request.Version.Major,
        (byte)request.Version.Minor,
        request.VersionPolicy);
} 

[Event(1, Level = EventLevel.Informational)]
private void RequestStart(string scheme, string host, int port, string pathAndQuery, byte versionMajor, byte versionMinor, HttpVersionPolicy versionPolicy)
{
    ...
    WriteEvent(eventId: 1, scheme, host, port, pathAndQuery, versionMajor, versionMinor, versionPolicy);
}

The different WriteEvent overloads are responsible for filling an array of EventData elements (each one contains a pointer to the data and its size) that is passed to EventSource.WriteEventCore. This helper methods dispatches the event to EventPipe/ETW pipelines.

Document the undocumented

Unlike CLR events, their payload is not explicitly described in a text file but, instead, you have to look at the implementation of the different WriteEvent overloads. The rest of this section provides the details of the sources I’ve looked at and the corresponding events payload.

HTTP

Sources: https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.Http/src/System/Net/Http/HttpTelemetry.cs Name: System.Net.Http Guid: d30b5633–7ef1–5485-b4e0–94979b102068

Sockets

Sources: https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketsTelemetry.cs

Name: System.Net.Sockets Guid: d5b2e7d4-b6ec-50ae-7cde-af89427ad21f

DNS

Sources: https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.NameResolution/src/System/Net/NameResolutionTelemetry.cs

Name: System.Net.NameResolution Guid: 4b326142-bfb5–5ed3–8585–7714181d14b0

Network Security

Sources: https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.Security/src/System/Net/Security/NetSecurityTelemetry.cs

Name: System.Net.Security  Guid: 7beee6b1-e3fa-5ddb-34be-1404ad0e2520

The next episode will describe how to listen to these events and extract their payload in C#.

Testing the statistical results

In parallel of the performance impact, it is important to validate the expected statistical distribution of the sampled allocations. Basically, I need to execute the same run of allocations multiple times in a row. Each run allocates the same number of instances of different types. For example, it is interesting to know if sampling instances of types with sizes proportional to a base value gives good results. Same question for totally different sized types or with Finalizers.

I would like to pass the number of runs to execute and a given scenario to a C# runner program and listen to the emitted events in another C# listener.

I’m facing 3 issues here:

• How many instances are allocated to validate the upscaling algorithm (sampled vs real count)
• What are the types I want to focus on because I don’t want to hard code them in the listener application.
• When does each run start?

It would be great if I could send the answer to these questions via events so the listener would know at runtime. Well… This is exactly what a class inherited from EventSource allows you to do!

In the runner application, I’ve defined the AllocationsRunEventSource that is decorated with the EventSource attribute to set its name that will be used as a provider name like Microsoft-Windows-DotNETRuntime for the .NET runtime provider.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

[EventSource(Name = "Allocations-Run")]
public class AllocationsRunEventSource : EventSource
{
    public static readonly AllocationsRunEventSource Log = new AllocationsRunEventSource();

The four implemented methods are defining which event ID for which verbosity will be emitted with which payload:
    [Event(600, Level = EventLevel.Informational)]
    public void StartRun(int iterationsCount, int allocationCount, string listOfTypes)
    {
        WriteEvent(eventId: 600, iterationsCount, allocationCount, listOfTypes);
    }

    [Event(601, Level = EventLevel.Informational)]
    public void StopRun()
    {
        WriteEvent(eventId: 601);
    }

    [Event(602, Level = EventLevel.Informational)]
    public void StartIteration(int iteration)
    {
        WriteEvent(eventId: 602, iteration);
    }

    [Event(603, Level = EventLevel.Informational)]
    public void StopIteration(int iteration)
    {
        WriteEvent(eventId: 603, iteration);
    }
}

To make the payload serialization and parsing easy, the list of types that will be allocated is passed as a string with the following format allocatedTypes = “Object24;Object48;Object72;Object32;Object64;Object96”.

The code of the runner calls these methods as expected at different moment of the execution:

1
2
3
4
5
6
7
8

AllocationsRunEventSource.Log.StartRun(iterations, allocationsCount, allocatedTypes);
 for (int i = 0; i < iterations; i++)
 {
     AllocationsRunEventSource.Log.StartIteration(i);
     allocationsRun.Allocate(allocationsCount);
     AllocationsRunEventSource.Log.StopIteration(i);
 }
 AllocationsRunEventSource.Log.StopRun();

Instead of recording the events with dotnet-trace, this time I’m using TraceEvent and Microsoft.Diagnostics.NETCore.Client to code a listener application. The code is very similar to what was presented for my dotnet-fullgc CLI tool except that I’m enabling the AllocationsRun provider corresponding to the event source of the runner in addition to the .NET runtime one:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

public static void PrintEventsLive(int processId)
{
    var providers = new List<EventPipeProvider>()
    {
        new EventPipeProvider(
                "Microsoft-Windows-DotNETRuntime",
                EventLevel.Verbose, // verbose is required for AllocationTick
                (long)0x80000000001 // new AllocationSamplingKeyword + GCKeyword
                ),
        new EventPipeProvider(
                "Allocations-Run",
                EventLevel.Informational
                ),
    };

The custom events from that provider are received via the source.Dynamic.All C# event:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

var client = new DiagnosticsClient(processId);
    using (var session = client.StartEventPipeSession(providers, false))
    {
        Console.WriteLine();

        Task streamTask = Task.Run(() =>
        {
            var source = new EventPipeEventSource(session.EventStream);
            _source = source;
            ClrTraceEventParser clrParser = new ClrTraceEventParser(source);
            clrParser.GCAllocationTick += OnAllocationTick;
            source.Dynamic.All += OnEvents;
            ...

Because TraceEvent is not already aware of the new AllocationSampled event emitted by the PR code, it will also be received via the same OnEvent handler:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

private static void OnEvents(TraceEvent eventData)
{
    if (eventData.ID == (TraceEventID)303)
    {
        // AllocationSampled parsing 
        ...
        return;
    }

    if (eventData.ID == (TraceEventID)600)  // Start run
    {
        // keep track of the expected types and the number of allocated instances 
        ...
        return;
    }

    if (eventData.ID == (TraceEventID)601)  // Stop run
    {
        // show the results of the run
        ...
        return;
    }

    if (eventData.ID == (TraceEventID)602)  // Start an iteration in a run
    {
        // reset for a new iteration
        ...
        return;
    }

    if (eventData.ID == (TraceEventID)603)  // Stop an iteration in a run
    {
        // Show iteration results
        ...
        return;
    }
}

The parsing of the payload of the run related events is done the same way as for AllocationSampled by a dedicated xxxData class:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

class AllocationsRunData
{
    const int EndOfStringCharLength = 2;
    private TraceEvent _payload;

    public AllocationsRunData(TraceEvent payload)
    {
        _payload = payload;

        ComputeFields();
    }

    public int Iterations;
    public int Count;
    public string AllocatedTypes;

    private void ComputeFields()
    {
        int offsetBeforeString = 4 + 4;

        Span<byte> data = _payload.EventData().AsSpan();
        Iterations = BitConverter.ToInt32(data.Slice(0, 4));
        Count = BitConverter.ToInt32(data.Slice(4, 4));
        AllocatedTypes = Encoding.Unicode.GetString(data.Slice(offsetBeforeString, _payload.EventDataLength - offsetBeforeString - EndOfStringCharLength));
    }
}

By keeping track of this data, it is possible to show each iteration results:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

> starts 100 iterations allocating 1000000 instances
0|
Tag  SCount  TCount          SSize          TSize   UnitSize     UpscaledSize  UpscaledCount  Name
--------------------------------------------------------------------------------------------------
 ST     247     384           5928           9216         24         24702711        1029279  Object24
 ST     322     106          10304           3392         32         32205122        1006410  Object32
 ST     435     509          20880          24432         48         43510266         906463  Object48
 ST     587     776          37568          49664         64         58718825         917481  Object64
 ST     747     481          53784          34632         72         74726662        1037870  Object72
 ST     958     916          91968          87936         96         95845392         998389  Object96

that integrates AllocationTick numbers too.

At the end of the run, error distribution per type is also computed over the iterations:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

Object72
-------------------------
   1  -10.5 %
   2   -8.9 %
   3   -8.4 %
   4   -7.8 %
   5   -6.4 %
        ...
  49    0.2 %
  50    0.2 %
  51    0.2 %
        ...
  96    6.8 %
  97    6.8 %
  98    7.7 %
  99    8.6 %
 100   10.0 %

You could use the same mechanisms (a custom EventSource to emit additional information and an EventPipe listener to aggregate the data) for your own usage. This is a different way to use EventSource rather than emitting events for monitoring like what is done by the BCL.

Testing the standalone GC

In addition to the usual .NET GC, it is needed to validate that the changes are also working for the standalone GC. Long story short, it is possible to replace the existing .NET garbage collector by your own implementation. For the test, I needed to check that the standalone GC clrgcexp.dll generated by the .NET compilation generates the expected AllocationSampled events when the corresponding keyword with informational verbosity is enabled.

Debugging NativeAOT scenarios

The final step was to implement the feature for the NativeAOT scenario. When you build your C# application for NativeAOT, a lot happens behind the scenes, based on compilers known by Visual Studio corresponding to the official released version of the .NET runtime. In my case, I needed to use the brand-new code of my local branch and debug some simple C# applications. The steps to reach that goal are not that simple.

First, you follow the steps given by the documentation to build AOT CLR in debug and libs in release:

1
2

build clr.aot+lib -rc debug -lc release
build -c release

Then, open src\coreclr\tools\aot\ilc.sln

the repro project contains a program.cs file where you write the C# code you want to test and debug.

When you build a NativeAOT application, you need to select if you want the runtime that emits events or not. This is done by setting the EventSourceSupport to true in the .csproj:

Add the following in the .csproj to get events:

1

<EventSourceSupport>true</EventSourceSupport>

However, with the repro project, you need to change ILCompiler.csproj in a different way:

1

<ReproResponseLines Include="--feature:System.Diagnostics.Tracing.EventSource.IsSupported=true" />

Also, change the reproNative.vcxproj file to bind to eventpipe-enabled.lib instead of eventpipe-disabled.lib for the platform/configuration you want to debug:

1
2
3
4
5
6

<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
    ...
    <Link>
      ...
      <AdditionalDependencies>...$(ArtifactsRoot)bin\coreclr\windows.x64.Debug\aotsdk\eventpipe-enabled.lib;...</AdditionalDependencies>
    </Link>

Then, build repro in Debug x64

Next, change the target for the ILCompiler project:

Build and run it to generate the .obj file corresponding to the repro project

Finally, open src\coreclr\tools\aot\ILCompiler\reproNative\reproNative.vcxproj that will allow you to debug the program.cs you’ve just built!

Introduction

During the implementation of our .NET allocation profiler, we realized that the current sampling mechanism based on a fixed threshold did not provide a good enough statistical distribution. With the help of Noah Falk from the CLR Diagnostics team, I started to implement a randomized sampling based on a Bernoulli distribution model for .NET.

With this kind of changes, you need to ensure that you don’t break any existing code, the impact on performance is limited and the mathematical results map the expected mathematical distribution.

The rest of this blog series details the different tests I wrote and the corresponding tips and tricks that could be reused when you write C# code.

Testing the basics

From a high-level view, the code change does something simple: each time an allocation context is needed to fulfill an allocation, the code checks if it should be sampled. In that case, a new AllocationSampled event is emitted with the same information as the existing AllocationTick event plus an additional field. So, the first level of testing is to validate that the events are emitted when the keyword and verbosity are enabled for the .NET runtime provider.

The runtime has already some tests in place to validate that some events are emitted under the** \src\tests\tracing\eventpipe** folder. Here is the code of my XUnit test that mimics the existing ones such as simpleruntimeeventvalidation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

[Fact]
public static int TestEntryPoint()
{
    // check that AllocationSampled events are generated and size + type name are correct
    var ret = IpcTraceTest.RunAndValidateEventCounts(
        new Dictionary<string, ExpectedEventCount>() { { "Microsoft-Windows-DotNETRuntime", -1 } },
        _eventGeneratingActionForAllocations,
        // AllocationSamplingKeyword (0x80000000000): 0b1000_0000_0000_0000_0000_0000_0000_0000_0000_0000_0000
        new List<EventPipeProvider>() { new EventPipeProvider("Microsoft-Windows-DotNETRuntime", EventLevel.Informational, 0x80000000000) },
        1024, _DoesTraceContainEnoughAllocationSampledEvents, enableRundownProvider: false);
    if (ret != 100)
        return ret;

    return 100;
}

The IpcTraceTest.RunAndValidateEventCounts helper method accepts:

• The list of providers to enable with which keyword and verbosity level.
• How many events are expected (using -1 in my case because I can’t predict how many random events will be generated
• A callback with the code that will generate events (allocating a lot of instances of a custom type in my case)
• A callback that looks at emitted events

The last callback code relies on TraceEvent to listen to emitted events:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

private static Func<EventPipeEventSource, Func<int>> _DoesTraceContainEnoughAllocationSampledEvents = (source) =>
{
    int AllocationSampledEvents = 0;
    int Object128Count = 0;
    source.Dynamic.All += (eventData) =>
    {
        if (eventData.ID == (TraceEventID)303)  // AllocationSampled is not defined in TraceEvent yet
        {
            AllocationSampledEvents++;

            AllocationSampledData payload = new AllocationSampledData(eventData, source.PointerSize);
            // uncomment to see the allocation events payload
            // Logger.logger.Log($"{payload.HeapIndex} - {payload.AllocationKind} | ({payload.ObjectSize}) {payload.TypeName}  = 0x{payload.Address}");
            if (payload.TypeName == "Tracing.Tests.SimpleRuntimeEventValidation.Object128")
            {
                Object128Count++;
            }
        }
    };
    return () => {
        Logger.logger.Log("AllocationSampled counts validation");
        Logger.logger.Log("Nb events: " + AllocationSampledEvents);
        Logger.logger.Log("Nb object128: " + Object128Count);
        return (AllocationSampledEvents >= MinExpectedEvents) && (Object128Count != 0) ? 100 : -1;
    };
};

In my case, I’m adding a new event that is emitted when a new keyword is enabled. It means that TraceEvent does not know yet its ID (hence the 303 hardcoded value) or how to unpack the new event payload. This is why I created the AllocationSampleData type to expose the payload as public fields:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

class AllocationSampledData
{
    const int EndOfStringCharLength = 2;
    private TraceEvent _payload;
    private int _pointerSize;
    public AllocationSampledData(TraceEvent payload, int pointerSize)
    {
        _payload = payload;
        _pointerSize = pointerSize;
        TypeName = "?";

        ComputeFields();
    }

    public GCAllocationKind AllocationKind;
    public int ClrInstanceID;
    public UInt64 TypeID;
    public string TypeName;
    public int HeapIndex;
    public UInt64 Address;
    public long ObjectSize;
    public long SampledByteOffset;

    ...
}
And the extraction of each field from the payload is done in the ComputeFields method:
// The payload of AllocationSampled is not defined in TraceEvent yet
//
//  <data name="AllocationKind" inType="win:UInt32" map="GCAllocationKindMap" />
//  <data name="ClrInstanceID" inType="win:UInt16" />
//  <data name="TypeID" inType="win:Pointer" />
//  <data name="TypeName" inType="win:UnicodeString" />
//  <data name="HeapIndex" inType="win:UInt32" />
//  <data name="Address" inType="win:Pointer" />
//  <data name="ObjectSize" inType="win:UInt64" outType="win:HexInt64" />
//  <data name="SampledByteOffset" inType="win:UInt64" outType="win:HexInt64" />
//
    private void ComputeFields()
    {
        int offsetBeforeString = 4 + 2 + _pointerSize;

This offsetBeforeString value is computed based on the size of UInt32 (=4 bytes), UInt16 (=2 bytes) and a Pointer (depends on 32 bit=4 or 64 bit=8) fields before the string. As Span wraps the binary payload provided by TraceEvent:

1

Span<byte> data = _payload.EventData().AsSpan();

Since I know the size of each field from the payload definition in ClrEtwAll.man, the numeric fields are extracted thanks to the BitConverter methods:

1
2

AllocationKind = (GCAllocationKind)BitConverter.ToInt32(data.Slice(0, 4));
        ClrInstanceID = BitConverter.ToInt16(data.Slice(4, 2));

Things start to be more complicated when you need to get the value of an address. Its size is 4 bytes in 32 bit and 8 bytes in 64 bit:

1
2
3
4
5
6
7
8

if (_pointerSize == 4)
        {
            TypeID = BitConverter.ToUInt32(data.Slice(6, _pointerSize));
        }
        else
        {
            TypeID = BitConverter.ToUInt64(data.Slice(6, _pointerSize));
        }