In my previous posts, I explained how to use DIA and DbgHelp to map a method to its line in source code. I forgot to mention that it was correct for .NET Core but not for the “old” .NET Framework Windows PDB format. Instead of encoding the method token in the name, the symbol file contains the name of the methods. So, how to do the mapping for .NET Framework assemblies? You will find the answer (plus some tricks) in this article.

When I started to work on the support of the old Windows PDB format, I looked at what existed to parse the raw format and… I decided to try DbgHelp instead. With this first implementation, I realized that some token where missing and source code information was not retrieved for most of the methods.

So, I looked for another API to use and I found ISymUnmanagedReader. The usage philosophy is totally different from DIA or DbgHelp.

A little bit of magic

This interface is implemented in diasymreader.dll that comes with every .NET Framework installation. But you need to do COM magic to get it. After having called CoInitialize to setup COM, you ask for an instance of ISymUnmanagedBinder from CLSID_CorSymBinder_SxS:

1
2
3
4
5
6
7
8

CComPtr<ISymUnmanagedBinder> pBinder;
hr = CoCreateInstance(
    CLSID_CorSymBinder_SxS,
    NULL,
    CLSCTX_INPROC_SERVER,
    IID_ISymUnmanagedBinder,
    (void**)&pBinder
);

From the binder, you can get the ISymUnmanagedReader interface corresponding to the assembly you are interested in with GetReaderForFile. However, there are two tiny details to consider.

First, one parameter expects the path to the assembly, not to the .pdb file. That symbol file has to be stored in the same folder but note that the documentation states that you could have more flexible search with ISymUnmanagedBinder2::GetReaderForFile2 but I did not test it.

The second detail is the first parameter: an instance of IMetaDataImport for the same assembly. The steps to get it are… complicated.

Hosting the CLR

The idea is to host the .NET Framework and get the corresponding ICLRMetaHost interface:

1
2

CComPtr<ICLRMetaHost> pMetaHost;
HRESULT hr = CLRCreateInstance(CLSID_CLRMetaHost, IID_ICLRMetaHost, (void**)&pMetaHost);

Calling the CLRCreateInstance API allows you to get an instance of ICLRMetaHost from which you could enumerate installed version of .NET Framework. In my case, I know which version I want:

1
2
3

// Get the installed .NET Framework runtime (v4.0+)
CComPtr<ICLRRuntimeInfo> pRuntimeInfo;
hr = pMetaHost->GetRuntime(L"v4.0.30319", IID_ICLRRuntimeInfo, (void**)&pRuntimeInfo);

The ICLRRuntimeInfo interface allows you to get access to runtime services via GetInterface:

1
2

CComPtr<IMetaDataDispenser> pDispenser;
hr = pRuntimeInfo->GetInterface(CLSID_CorMetaDataDispenser, IID_IMetaDataDispenser, (void**)&pDispenser);

The service I’m interested in is the IMetadataDispenser interface that allows you to “open a scope” on the assembly you are interested in:

1
2
3
4
5
6

hr = pDispenser->OpenScope(
    wModulePath.c_str(),
    ofRead,
    IID_IMetaDataImport,
    (IUnknown**)&_pMetaDataImport
);

Note that the first parameter is the path to the assembly not to the .pdb file. The scope is abstracted by an IMetadataImport interface I have already described and that is needed to call GetReaderForFile: and get the ISymUnmanagedReader:

hr = pBinder->GetReaderForFile(_pMetaDataImport, wModulePath.c_str(), nullptr, &_pReader);

The road to get symbol details for a method

The ISymUnmanagedReader interface implements GetMethod to get details about a given method token via an ISymUnmanagedMethod interface. So, the next question is how to get these tokens. If you remember the previous article, these tokens are from the 06 MethodDef table in the assembly metadata; starting from 06000001 to the last one.

This means that you could write a simple loop starting from 1 up to a hardcoded maximum value, call TokenFromRid(index, mdtMethodDef) to get the corresponding token. However, since you are a professional developer, you would search for the exact number of tokens from IMetadataTables retrieved from IMetadataImport:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

ULONG cRows = 0;

// Get IMetaDataTables interface to query the MethodDef table
CComPtr<IMetaDataTables> pTables;
hr = _pMetaDataImport->QueryInterface(IID_IMetaDataTables, (void**)&pTables);
if (FAILED(hr) || pTables == nullptr)
{
    cRows = LAST_METHODDEF_TOKEN;
}
else
{
    // Get the number of rows in the MethodDef table (table index 0x06 = Method)
    hr = pTables->GetTableInfo(
        0x06,           // MethodDef table
        NULL,           // cbRow (not needed)
        &cRows,         // pcRows (number of methods)
        NULL,           // pcCols (not needed)
        NULL,           // piKey (not needed)
        NULL            // ppName (not needed)
    );

    if (FAILED(hr))
    {
        cRows = LAST_METHODDEF_TOKEN;
    }
}

Now that you have the number of rows (i.e. number of methods defined in the metadata), it is easy and safe to get method information from symbols:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

for (uint32_t i = 1; i <= cRows; i++)
{
    mdMethodDef token = TokenFromRid(i, mdtMethodDef);

    CComPtr<ISymUnmanagedMethod> pMethod;
    hr = _pReader->GetMethod(token, &pMethod);
    if (SUCCEEDED(hr) && pMethod != nullptr)
    {
        MethodInfo info;
        if (GetMethodInfoFromSymbol(pMethod, info))
        {
            _methods.push_back(info);
        }
    }
}

Note that GetMethod might fail (returning E_FAIL) for P/Invoked functions, abstract methods, or methods decorated with DebuggerHidden attribute.

Give me line and source code!

For the other methods with symbol information, you can get its token via the GetToken method. The ISymUnmanagedMethod interface allows low level access to line/column mapping that is beyond the scope of this article. At a high level, positions in source file are named sequence points. Call GetSequencePointCount to get… the number of sequence points for a given method.

The next step is to call GetSequencePoints with the number of points you want and the corresponding arrays of offsets, lines, columns, end lines, end columns and ISymUnmanagedDocument. In my case, I’m only interested in where the method starts so the first sequence point is good enough:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

// Get sequence points (source line information)
ULONG32 cPoints = 0;
hr = pMethod->GetSequencePointCount(&cPoints);
if (SUCCEEDED(hr) && (cPoints > 0))
{
    cPoints = 1; // We only need the first sequence point for start line
    std::vector<ULONG32> offsets(cPoints);
    std::vector<ULONG32> lines(cPoints);
    std::vector<ULONG32> columns(cPoints);
    std::vector<ULONG32> endLines(cPoints);
    std::vector<ULONG32> endColumns(cPoints);
    std::vector<ISymUnmanagedDocument*> documents(cPoints);

    ULONG32 actualCount = 0;
    hr = pMethod->GetSequencePoints(
        cPoints,
        &actualCount,
        &offsets[0],
        &documents[0],
        &lines[0],
        &columns[0],
        &endLines[0],
        &endColumns[0]
    );

    if (SUCCEEDED(hr) && (actualCount > 0))
    {

The source file is described by ISymUnmanagedDocument that provides its name when GetURL is called:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

// Get the first sequence point's document and line
        ISymUnmanagedDocument* pDoc = documents[0];
        if (pDoc != nullptr)
        {
            // Get document URL (file path)
            ULONG32 urlLen = 0;
            hr = pDoc->GetURL(0, &urlLen, NULL);
            if (SUCCEEDED(hr) && (urlLen > 0))
            {
                std::vector<WCHAR> url(urlLen);
                hr = pDoc->GetURL(urlLen, &urlLen, &url[0]);
                if (SUCCEEDED(hr))
                {
                    // Convert wide string to narrow string
                    int len = WideCharToMultiByte(CP_UTF8, 0, &url[0], urlLen, NULL, 0, NULL, NULL);
                    std::string narrowUrl(len, '\0');
                    WideCharToMultiByte(CP_UTF8, 0, &url[0], urlLen, &narrowUrl[0], len, NULL, NULL);
                    info.sourceFile = narrowUrl;
                }
            }

            // NOTE: 0xFEEFEE is a special value indicating hidden lines
            info.lineNumber = lines[0];
            pDoc->Release();
        }
    }
}

The final interesting trick is that the line number might have the special 0xFEEFEE value. It means that the line is hidden. I have seen it for methods generated by the C# compiler such as MoveNext for async state machines or anonymous methods:

The source code is available from my Github repository.

Happy coding!

References

• Archived Microsoft-pdb repository
• DIA implementation article
• DbgHelp implementation article

Introduction

In our Datatog continuous .NET profiler implementation, we collect the call stack of a thread when something interesting happens such as an exception is thrown for example. In addition to the method name we would like to figure out at what line in which source code file this method is implemented.

This information is usually stored in the program database (.pdb) file that is generated by the compiler when the assembly is generated from the source code. The type and the name of the method are stored in the metadata of the assembly itself but I already told this story before. The .NET compilers support two formats of .pdb: the Portable format for .NET Core and the Windows format for .NET Framework.

I’ve explained how to use the DIA API and it is now time to show how to leverage the DbgHelp API that is available on all Windows machines (even though it is recommended to always install the latest release via the Debugging Tools for Windows.

This time, my goal is to extract from a Windows .pdb file the source code and line information for a given managed method. You can find the corresponding source code of this DumpLine tool in my Github repository.

Starting with DbgHelp

There are two major ways to get access to symbols with DbgHelp: either from a running process (i.e. to map the currently loaded .dll to their associated .pdb files) or from a tool that would explicitly load a .dll or a .pdb file.

Before anything, you tell DbgHelp which options you want by calling SymSetOptions:

1
2
3
4
5
6
7
8

DWORD options = SymGetOptions();
    options |= SYMOPT_DEBUG;
    options |= SYMOPT_LOAD_LINES;           // Load line number information
    options |= SYMOPT_UNDNAME;              // Undecorate symbol names
    //options |= SYMOPT_DEFERRED_LOADS;       // Defer symbol loading
    options |= SYMOPT_EXACT_SYMBOLS;        // Require exact symbol match
    options |= SYMOPT_FAIL_CRITICAL_ERRORS; // Don't show error dialogs
    SymSetOptions(options);

This is where I ask that line number information should be collected.

The next step is to call SymInitialize to setup DbgHelp environment. The first parameter expects a process handle (returned by GetCurrentProcess in my case). You could pass a path where to find the .pdb files for your dlls as a second parameter. In my case, since I will provide a .pdb file path, I don’t need it, and NULL will be passed. It means that, if needed, DbgHelp will use the current folder and the path set in _NT_SYMBOL_PATH and _NT_ALTERNATE_SYMBOL_PATH environment variables.

The last boolean parameter tells DbgHelp if you want that SymLoadModule64 to be called for each and every loaded .dll in the given process. Definitively not what I want so I’m passing FALSE.

1
2
3
4
5

_hProcess = GetCurrentProcess();
    if (!SymInitialize(_hProcess, NULL, FALSE))
    {
        _hProcess = NULL;
    }

At that point, I’m ready to load a .pdb file.

Loading a .pdb file

The API is straightforward: just call SymLoadModuleEx :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

_baseAddress = SymLoadModuleEx(
        _hProcess,
        NULL,
        pdbFilePath.c_str(),
        NULL,
        0x10000000, // arbitrary base address
        0,
        NULL,
        0
    );

    if (_baseAddress == 0)
    {
        return false;
    }

The important parameters are the process handle (same as for SymInitialize) and the path of the .pdb file. I’ve lost some time trying to understand why my code was not working due to a weird behavior of this function. You know that it succeeds when the returned address is not 0. Well… This is not 100% correct. If the path you provide does not exist, you won’t get 0 but the base address that you also provide. Even worth, when you call the functions I’ll detail later on, no error will happen but nothing will work as expected. So, I simply check that the file exists:

1
2
3
4
5

// BUG? : dbghelp does not fail if the .pdb file does not exist...
    if (GetFileAttributesA(pdbFilePath.c_str()) == INVALID_FILE_ATTRIBUTES)
    {
        return false;
    }

Note that it is possible to unload the symbols of a given loaded module by calling SymUnloadModule with the same process handle and its base address: this will reduce the memory consumption if you don’t need the symbols anymore.

In case of deferred load symbols, it is needed to call SymGetModuleInfo64 before trying to access the symbols:

1
2
3
4
5
6

IMAGEHLP_MODULE64 moduleInfo = { 0 };
    moduleInfo.SizeOfStruct = sizeof(IMAGEHLP_MODULE64);
    if (!SymGetModuleInfo64(_hProcess, _baseAddress, &moduleInfo))
    {
        return false;
    }

In addition, this will fill up an IMAGEHLP_MODULE64 structure with possibly interesting details:

The .pdb signature and age could be useful to build urls to communicate with symbol servers; but this is another story:

1
2
3
4
5
6
7
8
9

_age = moduleInfo.PdbAge;
    GUID guid = moduleInfo.PdbSig70;
    char strGUID[80];
    sprintf_s(strGUID, 80, "%08x%04x%04x%02x%02x%02x%02x%02x%02x%02x%02x",
        guid.Data1, guid.Data2, guid.Data3,
        guid.Data4[0], guid.Data4[1], guid.Data4[2], guid.Data4[3],
        guid.Data4[4], guid.Data4[5], guid.Data4[6], guid.Data4[7]
        );
    _guid = strGUID;

You also know if line numbers are available or not thanks to the LineNumbers field.

Note that if you asked for deferred symbols option, you won’t get any interesting details:

Only the module name and path are provided but nothing else.

Enumerating the methods

It is now time to iterate on the symbols in the loaded .pdb thanks to SymEnumSymbols:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

if (!SymEnumSymbols(
            _hProcess,
            _baseAddress,
            "*!*",  // Mask (all symbols)
            EnumMethodSymbolsCallback,
            this    // User context to store the methods in _methods instance field
    ))
    {
        return false;
    }

In addition to the obvious parameters, this function expects a callback function that will be called for each symbol in the module specified by the process handle and the base address. Note that you can pass any context as the last parameter. In my case, the instance of my DbgHelpParser class is passed to be able to store the methods in a dedicated _methods field:

1
2
3
4
5
6
7
8
9

struct MethodInfo
{
    std::string name;
    uint64_t address;
    uint32_t size;
    std::string sourceFile;
    uint32_t lineNumber;
};
std::vector<MethodInfo> _methods;

The “*!*” mask tells DbgHelp to look for symbols in all modules. This might sound counter intuitive, but the syntax is similar to what you find in WinDBG or Visual Studio: !. This could be useful if you load more than one .pdb.

The job of the callback function is to detect the symbols you are interested in from the SYMBOL_INFO structure passed for each matching symbol:

1
2
3
4
5
6
7
8
9

BOOL CALLBACK DbgHelpParser::EnumMethodSymbolsCallback(PSYMBOL_INFO pSymInfo, ULONG SymbolSize, PVOID UserContext)
{
    DbgHelpParser* parser = reinterpret_cast<DbgHelpParser*>(UserContext);

    if (
        (pSymInfo->Tag == SymTagFunction) &&
        ((pSymInfo->Flags & (SYMFLAG_CLR_TOKEN | SYMFLAG_METADATA)) == (SYMFLAG_CLR_TOKEN | SYMFLAG_METADATA))
        )
    {

The Tag field contains a value from SymTagEnum but, for a managed .pdb file, you will only get SymTagFunction. Also, the Flags field should contain SYMFLAG_CLR_TOKEN and SYMFLAG_METADATA because we are only interested in managed methods.

Next, you get the name, address and size from other fields before looking for the source file and line details by calling SymGetLineFromAddr64:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

MethodInfo info;
        info.name = pSymInfo->Name;
        info.address = pSymInfo->Address;
        info.size = pSymInfo->Size;

        // Try to get source file and line information
        IMAGEHLP_LINE64 line = { 0 };
        line.SizeOfStruct = sizeof(IMAGEHLP_LINE64);
        DWORD displacement = 0;

        if (SymGetLineFromAddr64(parser->_hProcess, pSymInfo->Address, &displacement, &line))
        {
            info.sourceFile = line.FileName ? line.FileName : "";
            info.lineNumber = line.LineNumber;
        }
        else {
            info.sourceFile = "";
            info.lineNumber = 0;
        }

        parser->_methods.push_back(info);
    }

    return TRUE; // Continue enumeration
}

The callback returns TRUE to continue the enumeration. You could return FALSE if you would look for specific symbols and wanted to speed up the processing.

The managed side of the story

When the tool is run on a managed assembly, you get the following kind of output:

The name of the method does not match at all with the names in my test assembly! Why do all methods have this generic Method.# format?

Well… the number corresponds to the RID of the corresponding method in the metadata of the assembly. Let’s have a look at what the MethodDef metadata table of this assembly looks like in ILSpy:

The RID column corresponds to the number in the name in the tool output. So, the Method.#3 is the get_Records property getter in PDBFormatReaderTest.cs:28. And this is exactly what I can see in the test source code:

You can also check that the lines look correct compared to what is listed by the tool:

• FilePath getter at line 19
• FilePath setter at line 20
• Records getter at line 28
• Records setter at line 29

Feel free to look at the source code from my Github repository.

DbgHelp provides many more services to look into symbols but that’s all for today!

During this R&D week at Datadog, I wanted to implement a tool accepting a .pdb file and generate a .sym file listing functions symbols with their address, size, name with signature and if they are public or private. This post dig into the implementation details of using Microsoft Debug Interface Access (DIA) COM API to achieve these objectives. If you want to see what my vibe coding experience in Cursor was, read this other post instead.

One self-contained tool please!

I would like the tool to be self-contained but since DIA is based on a COM server, it would require registering msdia40.dll on the machine. Not a good idea. In case the dll is in the same folder as the tool, one could “emulate” the magic done by CoCreateInstance to get an instance of IDiaDataSource (more on this interface soon) by:

• Call LoadLibrary to load the dll in memory
• Call GetProcAddress to get the DllGetClassObject implementation
• Call this function to get the IClassFactory implementation
• Call its CreateInstance method to get an object implementing IDiaDataSource

Here is the corresponding code (without error checking for readability)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

// Create DIA data source without registration using DLL loading
HRESULT PdbSymbolExtractor::NoRegCoCreate(const std::wstring& dllPath, REFCLSID rclsid, REFIID riid, void** ppv) {
    HMODULE hDll = LoadLibraryW(dllPath.c_str());

    typedef HRESULT(__stdcall* DllGetClassObjectFunc)(REFCLSID, REFIID, LPVOID*);
    DllGetClassObjectFunc pDllGetClassObject = (DllGetClassObjectFunc)GetProcAddress(hDll, "DllGetClassObject");

    CComPtr<IClassFactory> pClassFactory;
    HRESULT hr = pDllGetClassObject(rclsid, IID_IClassFactory, (void**)&pClassFactory);

    hr = pClassFactory->CreateInstance(NULL, riid, ppv);

    // Note: We intentionally don't call FreeLibrary here because the DLL needs to stay loaded
    // The COM object references will keep it alive
    return S_OK;

This function is called with the path of msdia40.dll and the UUID of the expected IDiaDataSource:

1
2

// Create DIA data source without registration
hr = NoRegCoCreate(dllPath, CLSID_DiaSource, __uuidof(IDiaDataSource), (void**)&_pDiaDataSource);

But still, I don’t want to have two binaries!

The trick is to embed the msdia40.dll inside the tool as a Windows resource. In the .rc file, add an RCDATA entry that points to the dll:

1

IDR_MSDIA_DLL      RCDATA      "x64\\Release\\msdia140.dll"

You should see it in the Resource View in Visual Studio:

Here is the code that extracts it as a file on disk is straightforward (error checking has been removed for readability):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

// Extract embedded msdia140.dll from resources
bool PdbSymbolExtractor::ExtractEmbeddedDll(const std::wstring& outputPath) {
    // Find the resource
    HMODULE hModule = GetModuleHandle(NULL);
    HRSRC hResource = FindResource(hModule, MAKEINTRESOURCE(IDR_MSDIA_DLL), RT_RCDATA);

    // Load the resource
    HGLOBAL hLoadedResource = LoadResource(hModule, hResource);

    // Lock the resource to get a pointer to the data
    LPVOID pResourceData = LockResource(hLoadedResource);

    // Get the size of the resource
    DWORD resourceSize = SizeofResource(hModule, hResource);

    // Write the DLL to disk
    std::ofstream outFile(outputPath, std::ios::binary);
    outFile.write(static_cast<const char*>(pResourceData), resourceSize);
    outFile.close();

    return true;
}

Where are my function symbols?

After calling NoRegCoCreate(), _pDiaDataSource stores a reference to the entry point into the DIA APIs. Here are the steps to follow before being able to list the symbols:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

HRESULT PdbSymbolExtractor::ExtractSymbolsFromPdb(const std::wstring& pdbPath, std::vector<FunctionSymbol>& symbols) 
{
    // Load the PDB file
    HRESULT hr = _pDiaDataSource->loadDataFromPdb(pdbPath.c_str());

    // Open a session
    CComPtr<IDiaSession> pSession;
    hr = _pDiaDataSource->openSession(&pSession);

    // Get the global scope
    CComPtr<IDiaSymbol> pGlobal;
    hr = pSession->get_globalScope(&pGlobal);

Now, you have the global scope of the symbols, you can ask for an enumerator for the type of symbols you are interested in; SymTagFunction in my case:

1
2
3
4
5
6
7

// Enumerate all function symbols
    CComPtr<IDiaEnumSymbols> pEnumSymbols;
    hr = pGlobal->findChildren(SymTagFunction, NULL, nsNone, &pEnumSymbols);

    LONG count = 0;
    pEnumSymbols->get_Count(&count);
    std::wcout << L"Found " << count << L" function symbols" << std::endl;

The pEnumSymbols iterator allows you to loop on each SymTagFunction symbol and get its name:

1
2
3
4
5
6
7
8
9

while (SUCCEEDED(pEnumSymbols->Next(1, &pSymbol, &celt)) && celt == 1) {
        FunctionSymbol func;

        // Get function name
        BSTR bstrName;
        if (pSymbol->get_name(&bstrName) == S_OK) {
            func.name = bstrName;
            SysFreeString(bstrName);
        }

Note that each symbol details are stored in a FunctionSymbol instance:

1
2
3
4
5
6
7
8

struct FunctionSymbol {
    std::wstring name;
    ...
    std::wstring signature; // Function signature (parameters only, no return type)
    DWORD rva;              // Relative Virtual Address
    ULONGLONG length;
    bool isPublic;
};

with the rest of the code in the while() loop:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

// Get function signature (parameters only, no return type)
    func.signature = ExtractFunctionSignature(pSymbol);

    // Get relative virtual address
    DWORD rva;
    if (pSymbol->get_relativeVirtualAddress(&rva) == S_OK) {
        func.rva = rva;
    }

    // Get function length
    ULONGLONG length;
    if (pSymbol->get_length(&length) == S_OK) {
        func.length = length;
    }

    // Determine if function is public or private
    // Check access level - default to private
    func.isPublic = false;
    DWORD access;
    if (pSymbol->get_access(&access) == S_OK) {
        func.isPublic = (access == CV_public);
    }

    pSymbol.Release();

I did not have the time to do more trial for private/public state, but I should have tried by enumerating SymTagPublicSymbol or SymTagExport that could be considered as public.

Better with a signature

The final step is to figure out the signature of each function. This is where the genericity of DIA could be confusing because so many things are represented by IDiaSymbol: a symbol, a function, the type of a function, or the type of a parameter…

So, the type of the function is retrieved as an IDiaSymbol by calling getType() on the function symbol. From that IDiaSymbol, findChildren() lets you iterate on the parameters:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

// Extract function signature (parameters only, no return type)
std::wstring PdbSymbolExtractor::ExtractFunctionSignature(IDiaSymbol* pSymbol) {
    if (!pSymbol) {
        return L"()";
    }

    // Get function type
    CComPtr<IDiaSymbol> pFunctionType;
    if (pSymbol->get_type(&pFunctionType) != S_OK || !pFunctionType) {
        return L"()";
    }

    // Enumerate function arguments
    CComPtr<IDiaEnumSymbols> pEnumArgs;
    if (FAILED(pFunctionType->findChildren(SymTagFunctionArgType, NULL, nsNone, &pEnumArgs))) {
        return L"()";
    }

    LONG argCount = 0;
    pEnumArgs->get_Count(&argCount);

    if (argCount == 0) {
        return L"()";
    }

Now, the same Next() method is called on the enumerator to iterate on each parameter:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

// Build signature string
    std::wstring signature = L"(";
    CComPtr<IDiaSymbol> pArg;
    ULONG argCelt = 0;
    bool first = true;

    while (SUCCEEDED(pEnumArgs->Next(1, &pArg, &argCelt)) && argCelt == 1) {
        if (!first) {
            signature += L", ";
        }
        first = false;

        // Get the argument type
        CComPtr<IDiaSymbol> pArgType;
        if (pArg->get_type(&pArgType) == S_OK && pArgType) {
            signature += GetTypeName(pArgType);
        } else {
            signature += L"?";
        }

        pArg.Release();
    }

    signature += L")";
    return signature;
}

The final step is to get the name of the type from the IDiaSymbol returned by get_type(). If it is a custom type, call get_name() like any other symbol. Otherwise, for basic types, call get_baseType() and get_length() as shown by the code below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

std::wstring PdbSymbolExtractor::GetTypeName(IDiaSymbol* pType) {
    if (!pType) {
        return L"?";
    }

    // Try to get type name directly
    BSTR bstrTypeName;
    if (pType->get_name(&bstrTypeName) == S_OK && bstrTypeName && wcslen(bstrTypeName) > 0) {
        std::wstring typeName = bstrTypeName;
        SysFreeString(bstrTypeName);
        return typeName;
    }

    // For basic types or unnamed types, try getting basic type info
    DWORD baseType = 0;
    ULONGLONG length = 0;

    if (pType->get_baseType(&baseType) == S_OK) {
        pType->get_length(&length);

        // Map basic types to names
        switch (baseType) {
            case btVoid: return L"void";
            case btChar: return L"char";
            case btWChar: return L"wchar_t";
            case btBool: return L"bool";

            case btInt:
            case btLong:
                if (length == 1) return L"char";
                else if (length == 2) return L"short";
                else if (length == 4) return L"int";
                else if (length == 8) return L"__int64";
                else return L"int" + std::to_wstring(length * 8);

            case btUInt:
            case btULong:
                if (length == 1) return L"unsigned char";
                else if (length == 2) return L"unsigned short";
                else if (length == 4) return L"unsigned int";
                else if (length == 8) return L"unsigned __int64";
                else return L"uint" + std::to_wstring(length * 8);

            case btFloat:
                if (length == 4) return L"float";
                else if (length == 8) return L"double";
                else return L"float" + std::to_wstring(length * 8);

            default:
                return L"?";
        }
    }

    return L"?";
}

This is a “simple” implementation that does not take pointers, addresses, arrays, and more into account. For a more complete solution, I would recommend looking at the PrintType() implementation in the DIA2Dump code sample that is installed with Visual Studio.

I hope this will get your foot in the door of symbol parsing and make you want to dig further into DIA.

References

• Corresponding source code is available in my github repository.
• Archived Microsoft documentation/implementation of .pdb format including a symbol dumper code.
• DIA2Dump Visual Studio code sample.

In the previous article, I presented what is needed (i.e. listen to WaitHandleWait events) to compute lock/wait durations and call stacks for Mutex, Semaphore, SemaphoreSlim, Manual/AutoResetEvent, ManualResetEventSlim, ReaderWriterLockSlim .NET synchronization constructs for a running process.

However, since the application is already running, some JIT-related events are missing, and some frames of the call stacks cannot be symbolized. Also, it would be great to monitor an application’s startup to see if it could be faster.

This post will detail how to monitor a .NET application since the very beginning of its life and the issues you might face.

Preparing a new .NET process to be monitored

From .NET 5, the dotnet-trace CLI tool allows you to pass a command line to execute and trace it from startup. In a very interesting article, Olivier Coanet presented the gory details about how to tell the .NET runtime to start an application in a pseudo-suspended mode as shown in the following diagram:

The first step is to create a ReverseDiagnosticsServer instance with a specific port (i.e. dotnet-wait_1234 in the diagram). Next, the process to monitor is spawned with the DOTNET_DiagnosticPorts environment variable set to the same port (i.e. dotnet-wait_1234). Look at the Diagnostics documentation of the Diagnostic Ports with DOTNET_DiagnosticPorts environment variable for more details. The .NET runtime is the new process will listen to this port and… wait.

When the tool is ready, it sends a resume command via a DiagnosticsClient: from that point in time, the CLR executes the normal flow of actions to run the application and… you will receive all events without missing one!

Get my command line please

Following the dotnet-trace example, my dotnet-wait tool accepts the command line of the child process in its final arguments that follow the** — **trigger. For example, dotnet-wait — dotnet foo.dll will start the program in foo.dll by using dotnet.exe. I’m reusing the code in ReversedServerHelper.cs to deal with arguments containing spaces:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

else if (current == "--")  // this is supposed to be the last one
{
    i++;
    if (i < args.Length)
    {
        parameters.pathName = args[i];

        // use the remaining arguments as the arguments for the child app to spawn
        i++;
        if (i < args.Length)
        {
            parameters.arguments = "";
            for (int j = i; j < args.Length; j++)
            {
                if (args[j].Contains(' '))
                {
                    parameters.arguments += $"\"{args[j].Replace("\"", "\\\"")}\"";
                }
                else
                {
                    parameters.arguments += args[j];
                }

                if (j != args.Length)
                {
                    parameters.arguments += " ";
                }
            }
        }

        // no need to look for more arguments
        break;
    }
    else
    {
        throw new InvalidOperationException($"Missing path name value...");
    }
}

The code to spawn the child process is simple:

1
2
3
4
5
6
7
8
9

// start the monitored app
var psi = new ProcessStartInfo(pathName);
if (!string.IsNullOrEmpty(arguments))
{
    psi.Arguments = arguments;
}
psi.EnvironmentVariables["DOTNET_DiagnosticPorts"] = port;
psi.UseShellExecute = false;
var process = System.Diagnostics.Process.Start(psi);

Here is an example with the following prompt:

-- dotnet "C:\CommandLineTest.dll" one two "t h r e e" four 'five six'

that generates the output (the test application is just listing its arguments):

dotnet-wait v1.0.0.0 - List wait duration
by Christophe Nasarre

Press ENTER to exit...
6 arguments
   1 | one
   2 | two
   3 | t h r e e
   4 | four
   5 | 'five
   6 | six'

This test reminded me to never use simple quotes in prompts :^)

It’s my console!

Once I implemented these steps, I immediately faced a very simple problem: my dotnet-wait tool and the test application are console applications. It means that they will share the same console for both input and output. For example, both are waiting for the RETURN key to (1) stop for the tool and (2) start for the test application: too bad for me because the tool will stop as soon the application starts…

Going back in time in my Windows memories, I remembered that the Win32 CreateProcess API accepts CREATE_NEW_CONSOLE as creation flag to automagically start the child process into its own new console. Unfortunately, it is not possible to pass this flag in .NET; maybe a limitation due to Linux support.

One simple solution could be to redirect the output of the tool or the application to a file: that would avoid mixing them in the console. Note that, by default, dotnet-trace discards output from the child process (by setting RedirectStandardOutput, RedirectStandardError and RedirectStandardInput to false and by ignoring the error and output streams) except if you pass — show-child-io on the command line. In this case, no output for dotnet-trace.

I decided to do the opposite for dotnet-wait: by default, you also get the child output but you can redirect the output of the tool to a file with -o

. Still, this does not solve the input problem in case of common expected keys.

If you remember the interactions between the tool and the monitored application, the latter is suspended until DiagnosticsClient::ResumeRuntime is called. So, why not starting the tool that spawns the application in one console and another instance of the tool in a new console that will resume the application? This is exactly what my friend Kevin Gosse imagined and how dotnet-wait works.

After the timeout that you give to diagnosticsServer.AcceptAsync(cancellation.Token) has elapsed, the runtime in the child process will display the following message:

The runtime has been configured to pause during startup and is awaiting a Diagnostics IPC ResumeStartup command from a Diagnostic Port.
DOTNET_DiagnosticPorts="dotnet-wait_34296"
DOTNET_DefaultDiagnosticPortSuspend=0

And this is exactly what the -r 34296 parameter will do!

You can now install dotnet wait and monitor the lock and wait contentions of your .NET9+ applications.

Introduction

In an old post, I detailed how to use ContentionStart and ContentionStop events to measure the lock contentions duration for a .NET application. In a .NET 9 pull request, a former Criteo’s colleague Grégoire Verdier has added new events to be notified when wait time similar to lock contention is happening for Mutex, Semaphore, Manual/AutoResetEvent. Read his post for more details about what he was trying to investigate.

With asynchronous and multi-threaded algorithms, it is essential to detect unexpected wait/locks in our applications. This post shows you how to leverage these events to measure the duration of these waits and get the call stack when the wait started:

New WaitHandleWait events

These new events are emitted by the Microsoft-Windows-DotNETRuntime CLR provider when you enable the WaitHandle (= 0x40000000000) keyword with Verbose verbosity. Each time WaitOne is called on a waitable object and this object is already owned, a WaitHandleWaitStart event is emitted. When the object is released, a WaitHandleWaitStop event is emitted.

For example, the following code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

static Mutex mutex = new Mutex();

static void Main()
{
    var owningThread = new Thread(OwningThread);
    owningThread.Start();
    var mutexThread = new Thread(MutexThread);
    mutexThread.Start();

    owningThread.Join();
    mutexThread.Join();
}

static void OwningThread()
{
    Console.WriteLine($"    [{GetCurrentThreadId(), 8}] Start to hold resources");
    Console.WriteLine("___________________________________________");
    mutex.WaitOne();

    Thread.Sleep(3000);  // the wait should last ~3 seconds

    Console.WriteLine("    Release resources");
    mutex.ReleaseMutex();
}

static void MutexThread()
{
    Console.WriteLine($"    [{GetCurrentThreadId(), 8}] waiting for Mutex...");
    mutex.WaitOne();  // events are emitted in the implementation when a contention happens
    mutex.ReleaseMutex();
    Console.WriteLine("    <-- Mutex");
}

generates a Start and Stop events pair:

1
2

125980 | 00000000-0000-0000-0000-000000000000 > event 301 __ [ 1| Start] WaitHandleWait/Start
125980 | 00000000-0000-0000-0000-000000000000 > event 302 __ [ 2|  Stop] WaitHandleWait/Stop

There is no associated activity ID so you rely on the fact that the same waiter thread (125980 in the previous example) is emitting for both events.

Listening to the new Wait events

As usual, you should rely on the TraceEvent nuget to start an EventPipe session with an already running .NET application. The last version already contains the definition of the keyword:

1

keywords |= ClrTraceEventParser.Keywords.WaitHandle; // .NET 9 WaitHandle kind of contention

and the C# events for Start and Stop:

1
2

source.Clr.WaitHandleWaitStart += OnWaitHandleWaitStart;
source.Clr.WaitHandleWaitStop += OnWaitHandleWaitStop;

The handler’s implementation is straightforward. The start of the wait is recorded for the current thread:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

private void OnWaitHandleWaitStart(WaitHandleWaitStartTraceData data)
{
    // get the contention info for the current thread
    ContentionInfo info = _contentionStore.GetContentionInfo(data.ProcessID, data.ThreadID);
    if (info == null)
        return;

    // keep track of the wait start
    info.ContentionStartRelativeMSec = data.TimeStampRelativeMSec;
}

When the wait ends, the duration is computed based on the recorded wait start because it is not provided in the payload like for ContentionStop:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

private void OnWaitHandleWaitStop(WaitHandleWaitStopTraceData data)
{
    ContentionInfo info = _contentionStore.GetContentionInfo(data.ProcessID, data.ThreadID);
    if (info == null)
        return;
    // unlucky case when we start to listen just after the WaitHandleStart event
    if (info.ContentionStartRelativeMSec == 0)
    {
        return;
    }

    // Too bad the duration is not provided in the payload like in ContentionStop...
    var contentionDurationMSec = data.TimeStampRelativeMSec - info.ContentionStartRelativeMSec;
    info.ContentionStartRelativeMSec = 0;
    var duration = TimeSpan.FromMilliseconds(contentionDurationMSec);
    Console.WriteLine($"{e.ThreadId,7} | {e.Duration.TotalMilliseconds} ms");
}

This is nice but it would be more useful if we could get the call stack of long waits.

Call stacks with EventPipe

In a previous post, I explained that it is possible to get the call stack when an event is emitted thanks to the ClrStackWalk event that follows the event you are interested in. Unfortunately, this is not more the case for .NET 5+ that is using EventPipe instead of ETW.

As Olivier Coanet presents in his post, you can get the call stack as an array of addresses from the hidden event record that is mapped by the TraceEvent parameter passed to each event handlers. This EVENT_RECORD structure contains a ExtendedData field that is an array of EVENT_HEADER_EXTENDED_DATA_ITEM:

1
2
3
4
5
6
7
8

public struct EVENT_HEADER_EXTENDED_DATA_ITEM
{
    public ushort Reserved1;
    public ushort ExtType;
    public ushort Reserved2;
    public ushort DataSize;
    public ulong DataPtr;
}

If the ExtType value is EVENT_HEADER_EXT_TYPE_STACK_TRACE64 (=6) then DataPtr points to a EVENT_EXTENDED_ITEM_STACK_TRACE64 structure:

1
2
3
4
5

public struct EVENT_EXTENDED_ITEM_STACK_TRACE64
{
    public ulong MatchId;
    public unsafe fixed ulong Address[1];
}

that contains an array of 64-bit addresses. The size of this array is given by DataSize — sizeof(ulong).

For 32-bit applications, you will get EVENT_HEADER_EXT_TYPE_STACK_TRACE32 (=5) as ExtType value and DataPtr will point to EVENT_EXTENDED_ITEM_STACK_TRACE32:

1
2
3
4
5

public struct EVENT_EXTENDED_ITEM_STACK_TRACE32
{
    public ulong MatchId;
    public unsafe fixed uint Address[1];
}

that stores an array of 32-bit addresses.

Knowing that makes writing the code to get the call stacks as an array of 64-bit addresses (same with 32-bit applications for simplicity sake) pretty straightforward:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

public static EventPipeUnresolvedStack ReadStackUsingUnsafeAccessor(TraceEvent traceEvent)
{
    return GetFromEventRecord(traceEvent.eventRecord);
}

private static EventPipeUnresolvedStack GetFromEventRecord(TraceEventNativeMethods.EVENT_RECORD* eventRecord)
{
    if (eventRecord == null)
        return null;

    var extendedDataCount = eventRecord->ExtendedDataCount;

    for (var dataIndex = 0; dataIndex < extendedDataCount; dataIndex++)
    {
        var extendedData = eventRecord->ExtendedData[dataIndex];
        if (extendedData.ExtType == TraceEventNativeMethods.EVENT_HEADER_EXT_TYPE_STACK_TRACE64)
        {
            var stackRecord = (TraceEventNativeMethods.EVENT_EXTENDED_ITEM_STACK_TRACE64*)extendedData.DataPtr;
            var addresses = &stackRecord->Address[0];
            var addressCount = (extendedData.DataSize - sizeof(UInt64)) / sizeof(UInt64);
            if (addressCount == 0)
                return null;

            var callStackAddresses = new ulong[addressCount];
            for (var index = 0; index < addressCount; index++)
            {
                callStackAddresses[index] = addresses[index];
            }
            return new EventPipeUnresolvedStack(callStackAddresses);
        }
        else if (extendedData.ExtType == TraceEventNativeMethods.EVENT_HEADER_EXT_TYPE_STACK_TRACE32)
        {
            var stackRecord = (TraceEventNativeMethods.EVENT_EXTENDED_ITEM_STACK_TRACE32*)extendedData.DataPtr;
            var addresses = &stackRecord->Address[0];
            var addressCount = (extendedData.DataSize - sizeof(UInt32)) / sizeof(UInt32);
            if (addressCount == 0)
                return null;

            var callStackAddresses = new ulong[addressCount];  // store the 32 addresses as 64 bit addresses
            for (var index = 0; index < addressCount; index++)
            {
                callStackAddresses[index] = addresses[index];
            }

            return new EventPipeUnresolvedStack(callStackAddresses);
        }
    }

    return null;
}

Note that the last version of TraceEvent nuget provides a public access to the eventRecord field so it is no more needed to use the UnsafeAccessor attribute used by Olivier.

Symbolize the call stack addresses

Address is good but the corresponding method name is better. I won’t repeat what I’ve already detailed in an older post that shows how to get the name of a native and managed name from an instruction pointer address. Instead, I want to pinpoint a big limitation of this solution to listen to CLR provider MethodLoadVerbose/MethodDCStartVerboseV2 events. If the methods you are interested in are jitted BEFORE your tool attaches to the application, you will never get these events.

You could get the same mapping address span/method name via the other “Microsoft-Windows-DotNETRuntimeRundown” provider and its MethodDCEndVerbose event that contains the expected MethodStartAddress, MethodSize and MethodName in its payload. But I need this information before the end of the application…

Looking at the documentation, it seems that the rundown provider accepts the StartRundownKeyword value to emit the DCStart events when the provider is enabled! Since .NET 9, it is possible to pass the keywords you want (before, the default value did not contain StartRundownKeyword) when creating the EventPipe session

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

//                V-- this is the default rundown keyword
rundownKeywords = 0x80020139 | (long)ClrTraceEventParser.Keywords.StartEnumeration;
var config = new EventPipeSessionConfiguration(GetProviders(), 256, rundownKeywords, true);
using (var session = client.StartEventPipeSession(config))
{
    var source = new EventPipeEventSource(session.EventStream);
    RegisterListeners(source);

    // this is a blocking call
    source.Process();
}

Note that you should not add the rundown provider to the list passed as parameter.

Unfortunately, there is currently an issue in the runtime since September 2020 that pinpoints this exact problem. I even tried to create and close a session to get the DCStop events before recreating a new one, but I failed.

The next episode will talk about how it is possible to start a .NET application and get the events since its startup… with the problems that are happening.

In the previous post, I detailed how I used the undocumented events from the BCL to create the dotnet-http CLI tool to monitor your outgoing HTTP requests. After testing with older versions of .NET, I realized that the code needed to be updated and I’m sharing my findings in this post.

The main point is that url redirections could have a major impact on requests latency:

Always test supported versions…

When I wrote the initial version of dotnet-http, I only tested it with .NET 8 and .NET 9 with limited formats of urls. Unfortunately, things went bad when I tried to monitor applications running on .NET 5 and .NET 6: no events are emitted by these versions of the BCL.

So, the next step was to test .NET 7 and the result was simple: crash! After investigating, I realized that some events I looked at in .NET 8 source code did not have the same payload in .NET 7; even no payload at all:

Even though there is no version field in the events payload, it is easy to check its size such as the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

private void OnConnectionEstablished(
     DateTime timestamp,
     int threadId,
     Guid activityId,
     Guid relatedActivityId,
     byte[] eventData
     )
 {
     ...
     EventSourcePayload payload = new EventSourcePayload(eventData);
     var versionMajor = payload.GetByte();
     var versionMinor = payload.GetByte();

     // in .NET 7, nothing else is available
     Int64 connectionId = 0;
     var scheme = "";
     var host = "";
     UInt32 port = 0;
     var path = "";

     if (eventData.Length > 2)
     {
         connectionId = payload.GetInt64();
         scheme = payload.GetString();
         host = payload.GetString();
         port = payload.GetUInt32();
         path = payload.GetString();
     }
     ...

Another difference is that one even is not even emitted in .NET 7:

Explain what a redirection is please!

Because I based the code on this Redirect event, I needed to find another way to support .NET 7 even though I would not have a redirected url to display. But first, let’s see what I’m talking about in terms of HTTP communication.

When you try to get the content / status code behind a url, the code is following different phases with the related events:

• **Start **RequestStart
• **DNS resolution **ResolutionStart ResolutionStop/Fail
• **Socket connection **ConnectStart ConnectStop
• **Security hand check (HTTPS only) **HandshakeStart HandshakeStop/Failed
• **Request/response **RequestHeadersStart RequestHeadersStop ResponseHeadersStart ResponseHeadersStop Redirect (.NET 8+) ResponseContentStart ResponseContentStop
• **Request stop **RequestStop/Failed

Based on the received url, a server can decide to answer that another url should be used instead. For example, if you call github with http:// instead of https://, such a redirection will happen. Without invasive tools such as Wireshark, these redirections are impossible to detect and could cause unnecessary delay.

From the client perspective, this can be detected in the ResponseHeadersStop event payload that provides a status code. If its value is 301, then it is a redirection. The other effect of a redirection is that the BCL code will change the flow of events because it needs to start over with the new url from step 2. to step 6. As you can see, instead of paying the cost of just one request, two are actually emitted and processed.

Impact on the implementation

In addition to the payload size checks addition, my initial implementation was not properly handling the redirection because the values (timestamps and durations) where overridden by the events related to the second redirected url.

The new implementation is splitting the request details into two classes. A base class that contains common fields to both parts of the request in case of redirection:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

private class HttpRequestInfoBase
   {
       public HttpRequestInfoBase(DateTime timestamp, string scheme, string host, uint port, string path)
       {
           StartTime = timestamp;
           if (scheme == string.Empty)
           {
               Url = string.Empty;
           }
           else
           {
               if (port != 0)
               {
                   Url = $"{scheme}://{host}:{port}{path}";
               }
               else
               {
                   Url = $"{scheme}://{host}:{path}";
               }
           }
       }

       public string Url { get; set; }
       public DateTime StartTime { get; set; }
       public DateTime ReqRespStartTime { get; set; }
       public double ReqRespDuration { get; set; }

       // DNS
       public double DnsWait { get; set; }
       public DateTime DnsStartTime { get; set; }
       public double DnsDuration { get; set; }

       // HTTPS
       public double HandshakeWait { get; set; }
       public DateTime HandshakeStartTime { get; set; }
       public double HandshakeDuration { get; set; }

       // socket connection
       public DateTime SocketConnectionStartTime { get; set; }
       public double SocketWait { get; set; }
       public double SocketDuration { get; set; }

       public DateTime QueueuingEndTime { get; set; }
       public double QueueingDuration { get; set; }
   }

The second one inherits from the base class and contains addition details; including the details of the redirected url if any:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

private class HttpRequestInfo : HttpRequestInfoBase
   {
       public HttpRequestInfo(DateTime timestamp, string scheme, string host, uint port, string path)
           :
           base(timestamp, scheme, host, port, path)
       {
       }

       public HttpRequestInfoBase Redirect { get; set; }

       public UInt32 StatusCode { get; set; }

       // HTTPS
       public string HandshakeErrorMessage { get; set; }

       public string Error { get; set; }
   }

A new instance of HttpRequestInfoBase is created when a 301 status code is received in HttpResponseHeaderStop handler:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

private void OnHttpResponseHeaderStop(object sender, HttpRequestStatusEventArgs e)
    {
        // used to detect redirection in .NET 8+
        if (e.StatusCode != 301)
        {
            return;
        }

        // create a new request info for the redirected request
        // because .NET 7 does not emit a Redirect event, we need to create a new request info here
        // --> it means that the redirect url will be empty in .NET 7
        var root = GetRoot(e.ActivityId);
        if (_requests.TryGetValue(root, out HttpRequestInfo info))
        {
            info.Redirect = new HttpRequestInfoBase(e.Timestamp, "", "", 0, "");

            // if you really want to have the duration of both original request + redirected request,
            // then do the following:
            //    info.ReqRespDuration = (e.Timestamp - info.ReqRespStartTime).TotalMilliseconds;
            // However, I prefer to show the duration of the redirected request only to more easily
            // compute the cost of the initial redirected request = total duration - other durations
        }
    }

For .NET 8+, the redirected url is provided to the Redirect handler and stored in the Url field of the instance created in the previous handler:

1
2
3
4
5
6
7
8
9

private void OnHttpRedirect(object sender, HttpRedirectEventArgs e)
    {
        // since this is an Info event, the activityID is the root
        var root = ActivityHelpers.ActivityPathString(e.ActivityId);
        if (_requests.TryGetValue(root, out HttpRequestInfo info))
        {
            info.Redirect.Url = e.RedirectUrl;
        }
    }

In each handler of events that could be received for both initial and redirected requests,

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

private void OnHttpResponseContentStop(object sender, EventPipeBaseArgs e)
   {
       var root = GetRoot(e.ActivityId);
       if (!_requests.TryGetValue(root, out HttpRequestInfo info))
       {
           return;
       }

       if (info.Redirect == null)
       {
           info.ReqRespDuration = (e.Timestamp - info.ReqRespStartTime).TotalMilliseconds;
       }
       else
       {
           info.Redirect.ReqRespDuration = (e.Timestamp - info.Redirect.ReqRespStartTime).TotalMilliseconds;
       }
   }

The wait time and durations are now computed as the events are received and aggregated at the end of the request by adding the value of both parts (initial and redirected if any:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

double dnsDuration = info.DnsDuration + ((info.Redirect != null) ? info.Redirect.DnsDuration : 0);
    if (dnsDuration > 0)
    {
        double dnsWait = info.DnsWait + ((info.Redirect != null) ? info.Redirect.DnsWait : 0);
        Console.Write($"{dnsWait,9:F3} | {dnsDuration,9:F3} | ");
    }
    else
    {
        Console.Write($"          |           | ");
    }

As a conclusion, you should try to monitor these redirections by using my dotnet-http CLI tool. Feel free to download it or install it with the following command line: dotnet tool install -g dotnet-http or update to the latest version with: dotnet tool update -g dotnet-http

You could also integrate some event listening code into your framework that simply handles the ResponseHeadersStop/Redirect events.

I’ve presented in depth the events emitted by the CLR in many posts to get insightful details about how the .NET runtime is working (lock contention, GC, allocations, …). Some .NET features are not implemented at the runtime level but at the Base Class Library (a.k.a. BCL) level. For example, if you are using HttpClient, you might want to measure how long it takes to get the response to your HTTP requests.

In this new series, I will describe how the BCL is using EventSource to emit the events, how you can listen to them with TraceEvent; focusing on HTTP requests.

Where do these events come from?

When you look for these BCL events in the documentation, you end up to this well-known event provides in .NET page and in the Framework libraries section. It lists the providers with their name and the emitted events with their keyword and verbosity. However, there is no detail about their payload! For example, for the “System.Net.Http” provider, you know that the RequestStart informal event is emitted each time “an HTTP request has started”. But you don’t know the corresponding url… Let me tell you how to get these tiny details.

Within the BCL, the classes responsible for emitting events derive from EventSource with the “Telemetry” suffix as a naming convention. They are decorated with an EventSource attribute to define their name (used as provider name by a listener) and their Guid that will be provided with each event. For example, here is the declaration of the class responsible for events related to sending HTTP requests:

1
2
3

[EventSource(Name = "System.Net.Http")]
internal sealed partial class HttpTelemetry : EventSource
{

If a Guid is not provided, one is automatically computed (see the corresponding code in Perfview).

Next, public helper methods decorated with a NonEvent attribute are provided to be used in the BCL code when it is needed to emit events. These methods are calling private methods decorated with an Event attribute to define their unique ID and their verbosity level:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

[NonEvent]
public void RequestStart(HttpRequestMessage request)
{
    ...
    RequestStart(
        request.RequestUri.Scheme,
        request.RequestUri.IdnHost,
        request.RequestUri.Port,
        request.RequestUri.PathAndQuery,
        (byte)request.Version.Major,
        (byte)request.Version.Minor,
        request.VersionPolicy);
} 

[Event(1, Level = EventLevel.Informational)]
private void RequestStart(string scheme, string host, int port, string pathAndQuery, byte versionMajor, byte versionMinor, HttpVersionPolicy versionPolicy)
{
    ...
    WriteEvent(eventId: 1, scheme, host, port, pathAndQuery, versionMajor, versionMinor, versionPolicy);
}

The different WriteEvent overloads are responsible for filling an array of EventData elements (each one contains a pointer to the data and its size) that is passed to EventSource.WriteEventCore. This helper methods dispatches the event to EventPipe/ETW pipelines.

Document the undocumented

Unlike CLR events, their payload is not explicitly described in a text file but, instead, you have to look at the implementation of the different WriteEvent overloads. The rest of this section provides the details of the sources I’ve looked at and the corresponding events payload.

HTTP

Sources: https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.Http/src/System/Net/Http/HttpTelemetry.cs Name: System.Net.Http Guid: d30b5633–7ef1–5485-b4e0–94979b102068

Sockets

Sources: https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketsTelemetry.cs

Name: System.Net.Sockets Guid: d5b2e7d4-b6ec-50ae-7cde-af89427ad21f

DNS

Sources: https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.NameResolution/src/System/Net/NameResolutionTelemetry.cs

Name: System.Net.NameResolution Guid: 4b326142-bfb5–5ed3–8585–7714181d14b0

Network Security

Sources: https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.Security/src/System/Net/Security/NetSecurityTelemetry.cs

Name: System.Net.Security  Guid: 7beee6b1-e3fa-5ddb-34be-1404ad0e2520

The next episode will describe how to listen to these events and extract their payload in C#.

Testing the statistical results

In parallel of the performance impact, it is important to validate the expected statistical distribution of the sampled allocations. Basically, I need to execute the same run of allocations multiple times in a row. Each run allocates the same number of instances of different types. For example, it is interesting to know if sampling instances of types with sizes proportional to a base value gives good results. Same question for totally different sized types or with Finalizers.

I would like to pass the number of runs to execute and a given scenario to a C# runner program and listen to the emitted events in another C# listener.

I’m facing 3 issues here:

• How many instances are allocated to validate the upscaling algorithm (sampled vs real count)
• What are the types I want to focus on because I don’t want to hard code them in the listener application.
• When does each run start?

It would be great if I could send the answer to these questions via events so the listener would know at runtime. Well… This is exactly what a class inherited from EventSource allows you to do!

In the runner application, I’ve defined the AllocationsRunEventSource that is decorated with the EventSource attribute to set its name that will be used as a provider name like Microsoft-Windows-DotNETRuntime for the .NET runtime provider.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

[EventSource(Name = "Allocations-Run")]
public class AllocationsRunEventSource : EventSource
{
    public static readonly AllocationsRunEventSource Log = new AllocationsRunEventSource();

The four implemented methods are defining which event ID for which verbosity will be emitted with which payload:
    [Event(600, Level = EventLevel.Informational)]
    public void StartRun(int iterationsCount, int allocationCount, string listOfTypes)
    {
        WriteEvent(eventId: 600, iterationsCount, allocationCount, listOfTypes);
    }

    [Event(601, Level = EventLevel.Informational)]
    public void StopRun()
    {
        WriteEvent(eventId: 601);
    }

    [Event(602, Level = EventLevel.Informational)]
    public void StartIteration(int iteration)
    {
        WriteEvent(eventId: 602, iteration);
    }

    [Event(603, Level = EventLevel.Informational)]
    public void StopIteration(int iteration)
    {
        WriteEvent(eventId: 603, iteration);
    }
}

To make the payload serialization and parsing easy, the list of types that will be allocated is passed as a string with the following format allocatedTypes = “Object24;Object48;Object72;Object32;Object64;Object96”.

The code of the runner calls these methods as expected at different moment of the execution:

1
2
3
4
5
6
7
8

AllocationsRunEventSource.Log.StartRun(iterations, allocationsCount, allocatedTypes);
 for (int i = 0; i < iterations; i++)
 {
     AllocationsRunEventSource.Log.StartIteration(i);
     allocationsRun.Allocate(allocationsCount);
     AllocationsRunEventSource.Log.StopIteration(i);
 }
 AllocationsRunEventSource.Log.StopRun();

Instead of recording the events with dotnet-trace, this time I’m using TraceEvent and Microsoft.Diagnostics.NETCore.Client to code a listener application. The code is very similar to what was presented for my dotnet-fullgc CLI tool except that I’m enabling the AllocationsRun provider corresponding to the event source of the runner in addition to the .NET runtime one:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

public static void PrintEventsLive(int processId)
{
    var providers = new List<EventPipeProvider>()
    {
        new EventPipeProvider(
                "Microsoft-Windows-DotNETRuntime",
                EventLevel.Verbose, // verbose is required for AllocationTick
                (long)0x80000000001 // new AllocationSamplingKeyword + GCKeyword
                ),
        new EventPipeProvider(
                "Allocations-Run",
                EventLevel.Informational
                ),
    };

The custom events from that provider are received via the source.Dynamic.All C# event:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

var client = new DiagnosticsClient(processId);
    using (var session = client.StartEventPipeSession(providers, false))
    {
        Console.WriteLine();

        Task streamTask = Task.Run(() =>
        {
            var source = new EventPipeEventSource(session.EventStream);
            _source = source;
            ClrTraceEventParser clrParser = new ClrTraceEventParser(source);
            clrParser.GCAllocationTick += OnAllocationTick;
            source.Dynamic.All += OnEvents;
            ...

Because TraceEvent is not already aware of the new AllocationSampled event emitted by the PR code, it will also be received via the same OnEvent handler:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

private static void OnEvents(TraceEvent eventData)
{
    if (eventData.ID == (TraceEventID)303)
    {
        // AllocationSampled parsing 
        ...
        return;
    }

    if (eventData.ID == (TraceEventID)600)  // Start run
    {
        // keep track of the expected types and the number of allocated instances 
        ...
        return;
    }

    if (eventData.ID == (TraceEventID)601)  // Stop run
    {
        // show the results of the run
        ...
        return;
    }

    if (eventData.ID == (TraceEventID)602)  // Start an iteration in a run
    {
        // reset for a new iteration
        ...
        return;
    }

    if (eventData.ID == (TraceEventID)603)  // Stop an iteration in a run
    {
        // Show iteration results
        ...
        return;
    }
}