DbgHelp on Welcome to Christophe Nasarre's Blog

How to support .NET Framework PDB format and source line with ISymUnmanagedReader

Wed, 11 Feb 2026 09:16:11 +0000

In my previous posts, I explained how to use DIA and DbgHelp to map a method to its line in source code. I forgot to mention that it was correct for .NET Core but not for the “old” .NET Framework Windows PDB format. Instead of encoding the method token in the name, the symbol file contains the name of the methods. So, how to do the mapping for .NET Framework assemblies? You will find the answer (plus some tricks) in this article.

When I started to work on the support of the old Windows PDB format, I looked at what existed to parse the raw format and… I decided to try DbgHelp instead. With this first implementation, I realized that some token where missing and source code information was not retrieved for most of the methods.

So, I looked for another API to use and I found ISymUnmanagedReader. The usage philosophy is totally different from DIA or DbgHelp.

A little bit of magic

This interface is implemented in diasymreader.dll that comes with every .NET Framework installation. But you need to do COM magic to get it. After having called CoInitialize to setup COM, you ask for an instance of ISymUnmanagedBinder from CLSID_CorSymBinder_SxS:

1
2
3
4
5
6
7
8


CComPtr<ISymUnmanagedBinder> pBinder;
hr = CoCreateInstance(
    CLSID_CorSymBinder_SxS,
    NULL,
    CLSCTX_INPROC_SERVER,
    IID_ISymUnmanagedBinder,
    (void**)&pBinder
);

From the binder, you can get the ISymUnmanagedReader interface corresponding to the assembly you are interested in with GetReaderForFile. However, there are two tiny details to consider.

First, one parameter expects the path to the assembly, not to the .pdb file. That symbol file has to be stored in the same folder but note that the documentation states that you could have more flexible search with ISymUnmanagedBinder2::GetReaderForFile2 but I did not test it.

The second detail is the first parameter: an instance of IMetaDataImport for the same assembly. The steps to get it are… complicated.

Hosting the CLR

The idea is to host the .NET Framework and get the corresponding ICLRMetaHost interface:

1
2


CComPtr<ICLRMetaHost> pMetaHost;
HRESULT hr = CLRCreateInstance(CLSID_CLRMetaHost, IID_ICLRMetaHost, (void**)&pMetaHost);

Calling the CLRCreateInstance API allows you to get an instance of ICLRMetaHost from which you could enumerate installed version of .NET Framework. In my case, I know which version I want:

1
2
3


// Get the installed .NET Framework runtime (v4.0+)
CComPtr<ICLRRuntimeInfo> pRuntimeInfo;
hr = pMetaHost->GetRuntime(L"v4.0.30319", IID_ICLRRuntimeInfo, (void**)&pRuntimeInfo);

The ICLRRuntimeInfo interface allows you to get access to runtime services via GetInterface:

1
2


CComPtr<IMetaDataDispenser> pDispenser;
hr = pRuntimeInfo->GetInterface(CLSID_CorMetaDataDispenser, IID_IMetaDataDispenser, (void**)&pDispenser);

The service I’m interested in is the IMetadataDispenser interface that allows you to “open a scope” on the assembly you are interested in:

1
2
3
4
5
6


hr = pDispenser->OpenScope(
    wModulePath.c_str(),
    ofRead,
    IID_IMetaDataImport,
    (IUnknown**)&_pMetaDataImport
);

Note that the first parameter is the path to the assembly not to the .pdb file. The scope is abstracted by an IMetadataImport interface I have already described and that is needed to call GetReaderForFile: and get the ISymUnmanagedReader:

hr = pBinder->GetReaderForFile(_pMetaDataImport, wModulePath.c_str(), nullptr, &_pReader);

The road to get symbol details for a method

The ISymUnmanagedReader interface implements GetMethod to get details about a given method token via an ISymUnmanagedMethod interface. So, the next question is how to get these tokens. If you remember the previous article, these tokens are from the 06 MethodDef table in the assembly metadata; starting from 06000001 to the last one.

This means that you could write a simple loop starting from 1 up to a hardcoded maximum value, call TokenFromRid(index, mdtMethodDef) to get the corresponding token. However, since you are a professional developer, you would search for the exact number of tokens from IMetadataTables retrieved from IMetadataImport:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


ULONG cRows = 0;

// Get IMetaDataTables interface to query the MethodDef table
CComPtr<IMetaDataTables> pTables;
hr = _pMetaDataImport->QueryInterface(IID_IMetaDataTables, (void**)&pTables);
if (FAILED(hr) || pTables == nullptr)
{
    cRows = LAST_METHODDEF_TOKEN;
}
else
{
    // Get the number of rows in the MethodDef table (table index 0x06 = Method)
    hr = pTables->GetTableInfo(
        0x06,           // MethodDef table
        NULL,           // cbRow (not needed)
        &cRows,         // pcRows (number of methods)
        NULL,           // pcCols (not needed)
        NULL,           // piKey (not needed)
        NULL            // ppName (not needed)
    );

    if (FAILED(hr))
    {
        cRows = LAST_METHODDEF_TOKEN;
    }
}

Now that you have the number of rows (i.e. number of methods defined in the metadata), it is easy and safe to get method information from symbols:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


for (uint32_t i = 1; i <= cRows; i++)
{
    mdMethodDef token = TokenFromRid(i, mdtMethodDef);

    CComPtr<ISymUnmanagedMethod> pMethod;
    hr = _pReader->GetMethod(token, &pMethod);
    if (SUCCEEDED(hr) && pMethod != nullptr)
    {
        MethodInfo info;
        if (GetMethodInfoFromSymbol(pMethod, info))
        {
            _methods.push_back(info);
        }
    }
}

Note that GetMethod might fail (returning E_FAIL) for P/Invoked functions, abstract methods, or methods decorated with DebuggerHidden attribute.

Give me line and source code!

For the other methods with symbol information, you can get its token via the GetToken method. The ISymUnmanagedMethod interface allows low level access to line/column mapping that is beyond the scope of this article. At a high level, positions in source file are named sequence points. Call GetSequencePointCount to get… the number of sequence points for a given method.

The next step is to call GetSequencePoints with the number of points you want and the corresponding arrays of offsets, lines, columns, end lines, end columns and ISymUnmanagedDocument. In my case, I’m only interested in where the method starts so the first sequence point is good enough:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


// Get sequence points (source line information)
ULONG32 cPoints = 0;
hr = pMethod->GetSequencePointCount(&cPoints);
if (SUCCEEDED(hr) && (cPoints > 0))
{
    cPoints = 1; // We only need the first sequence point for start line
    std::vector<ULONG32> offsets(cPoints);
    std::vector<ULONG32> lines(cPoints);
    std::vector<ULONG32> columns(cPoints);
    std::vector<ULONG32> endLines(cPoints);
    std::vector<ULONG32> endColumns(cPoints);
    std::vector<ISymUnmanagedDocument*> documents(cPoints);

    ULONG32 actualCount = 0;
    hr = pMethod->GetSequencePoints(
        cPoints,
        &actualCount,
        &offsets[0],
        &documents[0],
        &lines[0],
        &columns[0],
        &endLines[0],
        &endColumns[0]
    );

    if (SUCCEEDED(hr) && (actualCount > 0))
    {

The source file is described by ISymUnmanagedDocument that provides its name when GetURL is called:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


// Get the first sequence point's document and line
        ISymUnmanagedDocument* pDoc = documents[0];
        if (pDoc != nullptr)
        {
            // Get document URL (file path)
            ULONG32 urlLen = 0;
            hr = pDoc->GetURL(0, &urlLen, NULL);
            if (SUCCEEDED(hr) && (urlLen > 0))
            {
                std::vector<WCHAR> url(urlLen);
                hr = pDoc->GetURL(urlLen, &urlLen, &url[0]);
                if (SUCCEEDED(hr))
                {
                    // Convert wide string to narrow string
                    int len = WideCharToMultiByte(CP_UTF8, 0, &url[0], urlLen, NULL, 0, NULL, NULL);
                    std::string narrowUrl(len, '\0');
                    WideCharToMultiByte(CP_UTF8, 0, &url[0], urlLen, &narrowUrl[0], len, NULL, NULL);
                    info.sourceFile = narrowUrl;
                }
            }

            // NOTE: 0xFEEFEE is a special value indicating hidden lines
            info.lineNumber = lines[0];
            pDoc->Release();
        }
    }
}

The final interesting trick is that the line number might have the special 0xFEEFEE value. It means that the line is hidden. I have seen it for methods generated by the C# compiler such as MoveNext for async state machines or anonymous methods:

The source code is available from my Github repository.

Happy coding!

References

But where is my method code? DbgHelp comes to the rescue

Fri, 16 Jan 2026 08:21:30 +0000

Introduction

In our Datatog continuous .NET profiler implementation, we collect the call stack of a thread when something interesting happens such as an exception is thrown for example. In addition to the method name we would like to figure out at what line in which source code file this method is implemented.

This information is usually stored in the program database (.pdb) file that is generated by the compiler when the assembly is generated from the source code. The type and the name of the method are stored in the metadata of the assembly itself but I already told this story before. The .NET compilers support two formats of .pdb: the Portable format for .NET Core and the Windows format for .NET Framework.

I’ve explained how to use the DIA API and it is now time to show how to leverage the DbgHelp API that is available on all Windows machines (even though it is recommended to always install the latest release via the Debugging Tools for Windows.

This time, my goal is to extract from a Windows .pdb file the source code and line information for a given managed method. You can find the corresponding source code of this DumpLine tool in my Github repository.

Starting with DbgHelp

There are two major ways to get access to symbols with DbgHelp: either from a running process (i.e. to map the currently loaded .dll to their associated .pdb files) or from a tool that would explicitly load a .dll or a .pdb file.

Before anything, you tell DbgHelp which options you want by calling SymSetOptions:

1
2
3
4
5
6
7
8


DWORD options = SymGetOptions();
    options |= SYMOPT_DEBUG;
    options |= SYMOPT_LOAD_LINES;           // Load line number information
    options |= SYMOPT_UNDNAME;              // Undecorate symbol names
    //options |= SYMOPT_DEFERRED_LOADS;       // Defer symbol loading
    options |= SYMOPT_EXACT_SYMBOLS;        // Require exact symbol match
    options |= SYMOPT_FAIL_CRITICAL_ERRORS; // Don't show error dialogs
    SymSetOptions(options);

This is where I ask that line number information should be collected.

The next step is to call SymInitialize to setup DbgHelp environment. The first parameter expects a process handle (returned by GetCurrentProcess in my case). You could pass a path where to find the .pdb files for your dlls as a second parameter. In my case, since I will provide a .pdb file path, I don’t need it, and NULL will be passed. It means that, if needed, DbgHelp will use the current folder and the path set in _NT_SYMBOL_PATH and _NT_ALTERNATE_SYMBOL_PATH environment variables.

The last boolean parameter tells DbgHelp if you want that SymLoadModule64 to be called for each and every loaded .dll in the given process. Definitively not what I want so I’m passing FALSE.

1
2
3
4
5


_hProcess = GetCurrentProcess();
    if (!SymInitialize(_hProcess, NULL, FALSE))
    {
        _hProcess = NULL;
    }

At that point, I’m ready to load a .pdb file.

Loading a .pdb file

The API is straightforward: just call SymLoadModuleEx :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


_baseAddress = SymLoadModuleEx(
        _hProcess,
        NULL,
        pdbFilePath.c_str(),
        NULL,
        0x10000000, // arbitrary base address
        0,
        NULL,
        0
    );

    if (_baseAddress == 0)
    {
        return false;
    }

The important parameters are the process handle (same as for SymInitialize) and the path of the .pdb file. I’ve lost some time trying to understand why my code was not working due to a weird behavior of this function. You know that it succeeds when the returned address is not 0. Well… This is not 100% correct. If the path you provide does not exist, you won’t get 0 but the base address that you also provide. Even worth, when you call the functions I’ll detail later on, no error will happen but nothing will work as expected. So, I simply check that the file exists:

1
2
3
4
5


// BUG? : dbghelp does not fail if the .pdb file does not exist...
    if (GetFileAttributesA(pdbFilePath.c_str()) == INVALID_FILE_ATTRIBUTES)
    {
        return false;
    }

Note that it is possible to unload the symbols of a given loaded module by calling SymUnloadModule with the same process handle and its base address: this will reduce the memory consumption if you don’t need the symbols anymore.

In case of deferred load symbols, it is needed to call SymGetModuleInfo64 before trying to access the symbols:

1
2
3
4
5
6


IMAGEHLP_MODULE64 moduleInfo = { 0 };
    moduleInfo.SizeOfStruct = sizeof(IMAGEHLP_MODULE64);
    if (!SymGetModuleInfo64(_hProcess, _baseAddress, &moduleInfo))
    {
        return false;
    }

In addition, this will fill up an IMAGEHLP_MODULE64 structure with possibly interesting details:

The .pdb signature and age could be useful to build urls to communicate with symbol servers; but this is another story:

1
2
3
4
5
6
7
8
9


_age = moduleInfo.PdbAge;
    GUID guid = moduleInfo.PdbSig70;
    char strGUID[80];
    sprintf_s(strGUID, 80, "%08x%04x%04x%02x%02x%02x%02x%02x%02x%02x%02x",
        guid.Data1, guid.Data2, guid.Data3,
        guid.Data4[0], guid.Data4[1], guid.Data4[2], guid.Data4[3],
        guid.Data4[4], guid.Data4[5], guid.Data4[6], guid.Data4[7]
        );
    _guid = strGUID;

You also know if line numbers are available or not thanks to the LineNumbers field.

Note that if you asked for deferred symbols option, you won’t get any interesting details:

Only the module name and path are provided but nothing else.

Enumerating the methods

It is now time to iterate on the symbols in the loaded .pdb thanks to SymEnumSymbols:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


if (!SymEnumSymbols(
            _hProcess,
            _baseAddress,
            "*!*",  // Mask (all symbols)
            EnumMethodSymbolsCallback,
            this    // User context to store the methods in _methods instance field
    ))
    {
        return false;
    }

In addition to the obvious parameters, this function expects a callback function that will be called for each symbol in the module specified by the process handle and the base address. Note that you can pass any context as the last parameter. In my case, the instance of my DbgHelpParser class is passed to be able to store the methods in a dedicated _methods field:

1
2
3
4
5
6
7
8
9


struct MethodInfo
{
    std::string name;
    uint64_t address;
    uint32_t size;
    std::string sourceFile;
    uint32_t lineNumber;
};
std::vector<MethodInfo> _methods;

The “*!*” mask tells DbgHelp to look for symbols in all modules. This might sound counter intuitive, but the syntax is similar to what you find in WinDBG or Visual Studio: !. This could be useful if you load more than one .pdb.

The job of the callback function is to detect the symbols you are interested in from the SYMBOL_INFO structure passed for each matching symbol:

1
2
3
4
5
6
7
8
9


BOOL CALLBACK DbgHelpParser::EnumMethodSymbolsCallback(PSYMBOL_INFO pSymInfo, ULONG SymbolSize, PVOID UserContext)
{
    DbgHelpParser* parser = reinterpret_cast<DbgHelpParser*>(UserContext);

    if (
        (pSymInfo->Tag == SymTagFunction) &&
        ((pSymInfo->Flags & (SYMFLAG_CLR_TOKEN | SYMFLAG_METADATA)) == (SYMFLAG_CLR_TOKEN | SYMFLAG_METADATA))
        )
    {

The Tag field contains a value from SymTagEnum but, for a managed .pdb file, you will only get SymTagFunction. Also, the Flags field should contain SYMFLAG_CLR_TOKEN and SYMFLAG_METADATA because we are only interested in managed methods.

Next, you get the name, address and size from other fields before looking for the source file and line details by calling SymGetLineFromAddr64:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


MethodInfo info;
        info.name = pSymInfo->Name;
        info.address = pSymInfo->Address;
        info.size = pSymInfo->Size;

        // Try to get source file and line information
        IMAGEHLP_LINE64 line = { 0 };
        line.SizeOfStruct = sizeof(IMAGEHLP_LINE64);
        DWORD displacement = 0;

        if (SymGetLineFromAddr64(parser->_hProcess, pSymInfo->Address, &displacement, &line))
        {
            info.sourceFile = line.FileName ? line.FileName : "";
            info.lineNumber = line.LineNumber;
        }
        else {
            info.sourceFile = "";
            info.lineNumber = 0;
        }

        parser->_methods.push_back(info);
    }

    return TRUE; // Continue enumeration
}

The callback returns TRUE to continue the enumeration. You could return FALSE if you would look for specific symbols and wanted to speed up the processing.

The managed side of the story

When the tool is run on a managed assembly, you get the following kind of output:

The name of the method does not match at all with the names in my test assembly! Why do all methods have this generic Method.# format?

Well… the number corresponds to the RID of the corresponding method in the metadata of the assembly. Let’s have a look at what the MethodDef metadata table of this assembly looks like in ILSpy:

The RID column corresponds to the number in the name in the tool output. So, the Method.#3 is the get_Records property getter in PDBFormatReaderTest.cs:28. And this is exactly what I can see in the test source code:

You can also check that the lines look correct compared to what is listed by the tool:

FilePath getter at line 19
FilePath setter at line 20
Records getter at line 28
Records setter at line 29

Feel free to look at the source code from my Github repository.

DbgHelp provides many more services to look into symbols but that’s all for today!

Vibe coding a .pdb dumper or how I became a Product Manager

Mon, 08 Dec 2025 10:12:16 +0000

During this R&D week at Datadog, I wanted to implement a tool accepting a .pdb file and generate a .sym file listing functions symbols with their address, size, name with signature and if they are public or private. This post dig into the implementation details of using Microsoft Debug Interface Access (DIA) COM API to achieve these objectives. If you want to see what my vibe coding experience in Cursor was, read this other post instead.

One self-contained tool please!

I would like the tool to be self-contained but since DIA is based on a COM server, it would require registering msdia40.dll on the machine. Not a good idea. In case the dll is in the same folder as the tool, one could “emulate” the magic done by CoCreateInstance to get an instance of IDiaDataSource (more on this interface soon) by:

Call LoadLibrary to load the dll in memory
Call GetProcAddress to get the DllGetClassObject implementation
Call this function to get the IClassFactory implementation
Call its CreateInstance method to get an object implementing IDiaDataSource

Here is the corresponding code (without error checking for readability)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


// Create DIA data source without registration using DLL loading
HRESULT PdbSymbolExtractor::NoRegCoCreate(const std::wstring& dllPath, REFCLSID rclsid, REFIID riid, void** ppv) {
    HMODULE hDll = LoadLibraryW(dllPath.c_str());

    typedef HRESULT(__stdcall* DllGetClassObjectFunc)(REFCLSID, REFIID, LPVOID*);
    DllGetClassObjectFunc pDllGetClassObject = (DllGetClassObjectFunc)GetProcAddress(hDll, "DllGetClassObject");

    CComPtr<IClassFactory> pClassFactory;
    HRESULT hr = pDllGetClassObject(rclsid, IID_IClassFactory, (void**)&pClassFactory);

    hr = pClassFactory->CreateInstance(NULL, riid, ppv);

    // Note: We intentionally don't call FreeLibrary here because the DLL needs to stay loaded
    // The COM object references will keep it alive
    return S_OK;

This function is called with the path of msdia40.dll and the UUID of the expected IDiaDataSource:

1
2


// Create DIA data source without registration
hr = NoRegCoCreate(dllPath, CLSID_DiaSource, __uuidof(IDiaDataSource), (void**)&_pDiaDataSource);

But still, I don’t want to have two binaries!

The trick is to embed the msdia40.dll inside the tool as a Windows resource. In the .rc file, add an RCDATA entry that points to the dll:

1

IDR_MSDIA_DLL      RCDATA      "x64\\Release\\msdia140.dll"

You should see it in the Resource View in Visual Studio:

Here is the code that extracts it as a file on disk is straightforward (error checking has been removed for readability):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


// Extract embedded msdia140.dll from resources
bool PdbSymbolExtractor::ExtractEmbeddedDll(const std::wstring& outputPath) {
    // Find the resource
    HMODULE hModule = GetModuleHandle(NULL);
    HRSRC hResource = FindResource(hModule, MAKEINTRESOURCE(IDR_MSDIA_DLL), RT_RCDATA);

    // Load the resource
    HGLOBAL hLoadedResource = LoadResource(hModule, hResource);

    // Lock the resource to get a pointer to the data
    LPVOID pResourceData = LockResource(hLoadedResource);

    // Get the size of the resource
    DWORD resourceSize = SizeofResource(hModule, hResource);

    // Write the DLL to disk
    std::ofstream outFile(outputPath, std::ios::binary);
    outFile.write(static_cast<const char*>(pResourceData), resourceSize);
    outFile.close();

    return true;
}

Where are my function symbols?

After calling NoRegCoCreate(), _pDiaDataSource stores a reference to the entry point into the DIA APIs. Here are the steps to follow before being able to list the symbols:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


HRESULT PdbSymbolExtractor::ExtractSymbolsFromPdb(const std::wstring& pdbPath, std::vector<FunctionSymbol>& symbols) 
{
    // Load the PDB file
    HRESULT hr = _pDiaDataSource->loadDataFromPdb(pdbPath.c_str());

    // Open a session
    CComPtr<IDiaSession> pSession;
    hr = _pDiaDataSource->openSession(&pSession);

    // Get the global scope
    CComPtr<IDiaSymbol> pGlobal;
    hr = pSession->get_globalScope(&pGlobal);

Now, you have the global scope of the symbols, you can ask for an enumerator for the type of symbols you are interested in; SymTagFunction in my case:

1
2
3
4
5
6
7


// Enumerate all function symbols
    CComPtr<IDiaEnumSymbols> pEnumSymbols;
    hr = pGlobal->findChildren(SymTagFunction, NULL, nsNone, &pEnumSymbols);

    LONG count = 0;
    pEnumSymbols->get_Count(&count);
    std::wcout << L"Found " << count << L" function symbols" << std::endl;

The pEnumSymbols iterator allows you to loop on each SymTagFunction symbol and get its name:

1
2
3
4
5
6
7
8
9


while (SUCCEEDED(pEnumSymbols->Next(1, &pSymbol, &celt)) && celt == 1) {
        FunctionSymbol func;

        // Get function name
        BSTR bstrName;
        if (pSymbol->get_name(&bstrName) == S_OK) {
            func.name = bstrName;
            SysFreeString(bstrName);
        }

Note that each symbol details are stored in a FunctionSymbol instance:

1
2
3
4
5
6
7
8


struct FunctionSymbol {
    std::wstring name;
    ...
    std::wstring signature; // Function signature (parameters only, no return type)
    DWORD rva;              // Relative Virtual Address
    ULONGLONG length;
    bool isPublic;
};

with the rest of the code in the while() loop:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


// Get function signature (parameters only, no return type)
    func.signature = ExtractFunctionSignature(pSymbol);

    // Get relative virtual address
    DWORD rva;
    if (pSymbol->get_relativeVirtualAddress(&rva) == S_OK) {
        func.rva = rva;
    }

    // Get function length
    ULONGLONG length;
    if (pSymbol->get_length(&length) == S_OK) {
        func.length = length;
    }

    // Determine if function is public or private
    // Check access level - default to private
    func.isPublic = false;
    DWORD access;
    if (pSymbol->get_access(&access) == S_OK) {
        func.isPublic = (access == CV_public);
    }

    pSymbol.Release();

I did not have the time to do more trial for private/public state, but I should have tried by enumerating SymTagPublicSymbol or SymTagExport that could be considered as public.

Better with a signature

The final step is to figure out the signature of each function. This is where the genericity of DIA could be confusing because so many things are represented by IDiaSymbol: a symbol, a function, the type of a function, or the type of a parameter…

So, the type of the function is retrieved as an IDiaSymbol by calling getType() on the function symbol. From that IDiaSymbol, findChildren() lets you iterate on the parameters:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


// Extract function signature (parameters only, no return type)
std::wstring PdbSymbolExtractor::ExtractFunctionSignature(IDiaSymbol* pSymbol) {
    if (!pSymbol) {
        return L"()";
    }

    // Get function type
    CComPtr<IDiaSymbol> pFunctionType;
    if (pSymbol->get_type(&pFunctionType) != S_OK || !pFunctionType) {
        return L"()";
    }

    // Enumerate function arguments
    CComPtr<IDiaEnumSymbols> pEnumArgs;
    if (FAILED(pFunctionType->findChildren(SymTagFunctionArgType, NULL, nsNone, &pEnumArgs))) {
        return L"()";
    }

    LONG argCount = 0;
    pEnumArgs->get_Count(&argCount);

    if (argCount == 0) {
        return L"()";
    }

Now, the same Next() method is called on the enumerator to iterate on each parameter:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


// Build signature string
    std::wstring signature = L"(";
    CComPtr<IDiaSymbol> pArg;
    ULONG argCelt = 0;
    bool first = true;

    while (SUCCEEDED(pEnumArgs->Next(1, &pArg, &argCelt)) && argCelt == 1) {
        if (!first) {
            signature += L", ";
        }
        first = false;

        // Get the argument type
        CComPtr<IDiaSymbol> pArgType;
        if (pArg->get_type(&pArgType) == S_OK && pArgType) {
            signature += GetTypeName(pArgType);
        } else {
            signature += L"?";
        }

        pArg.Release();
    }

    signature += L")";
    return signature;
}

The final step is to get the name of the type from the IDiaSymbol returned by get_type(). If it is a custom type, call get_name() like any other symbol. Otherwise, for basic types, call get_baseType() and get_length() as shown by the code below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55


std::wstring PdbSymbolExtractor::GetTypeName(IDiaSymbol* pType) {
    if (!pType) {
        return L"?";
    }

    // Try to get type name directly
    BSTR bstrTypeName;
    if (pType->get_name(&bstrTypeName) == S_OK && bstrTypeName && wcslen(bstrTypeName) > 0) {
        std::wstring typeName = bstrTypeName;
        SysFreeString(bstrTypeName);
        return typeName;
    }

    // For basic types or unnamed types, try getting basic type info
    DWORD baseType = 0;
    ULONGLONG length = 0;

    if (pType->get_baseType(&baseType) == S_OK) {
        pType->get_length(&length);

        // Map basic types to names
        switch (baseType) {
            case btVoid: return L"void";
            case btChar: return L"char";
            case btWChar: return L"wchar_t";
            case btBool: return L"bool";

            case btInt:
            case btLong:
                if (length == 1) return L"char";
                else if (length == 2) return L"short";
                else if (length == 4) return L"int";
                else if (length == 8) return L"__int64";
                else return L"int" + std::to_wstring(length * 8);

            case btUInt:
            case btULong:
                if (length == 1) return L"unsigned char";
                else if (length == 2) return L"unsigned short";
                else if (length == 4) return L"unsigned int";
                else if (length == 8) return L"unsigned __int64";
                else return L"uint" + std::to_wstring(length * 8);

            case btFloat:
                if (length == 4) return L"float";
                else if (length == 8) return L"double";
                else return L"float" + std::to_wstring(length * 8);

            default:
                return L"?";
        }
    }

    return L"?";
}

This is a “simple” implementation that does not take pointers, addresses, arrays, and more into account. For a more complete solution, I would recommend looking at the PrintType() implementation in the DIA2Dump code sample that is installed with Visual Studio.

I hope this will get your foot in the door of symbol parsing and make you want to dig further into DIA.

References

Corresponding source code is available in my github repository.
Archived Microsoft documentation/implementation of .pdb format including a symbol dumper code.
DIA2Dump Visual Studio code sample.

Build your own .NET memory profiler in C# — call stacks (2/2–2)

Fri, 19 Jun 2020 09:32:16 +0000

In the past two episodes of this series I have explained how to get a sampling of .NET application allocations and one way to get the call stack corresponding to the allocations; all with CLR events. In this last episode, I will detail how to transform addresses from the stack into methods name and possibly signature.

From managed address to method signature

In order to transform an address on the stack into a managed method name, you need to know where in memory (i.e. at which address) is stored the method JITted assembly code and what is its size:

For each JITted method, the MethodLoadVerbose/MethodDCStartVerboseV2 events are providing this information in addition to 3 properties to rebuild the full method name and signature (more on this later). I’m storing each method description as a MethodInfo into a MethodStore per process.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


public class PerProcessProfilingState : IDisposable
{
    ...
    private readonly Dictionary<int, MethodStore> _methods = new Dictionary<int, MethodStore>();

public class MethodStore : IDisposable
{
    // JITed methods information (start address + size + signature)
    private readonly List<MethodInfo> _methods;
    ...

The only interesting part of the MethodInfo class is the computation of the full method name stored in the _fullName field:

1
2
3
4
5
6


public class MethodInfo
{
    private readonly ulong _startAddress;
    private readonly int _size;
    private readonly string _fullName;
    ...

The ComputeFullName helper merges together the 3 properties given by the MethodxxxVerbose events including special processing for constructors:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


private string ComputeFullName(ulong startAddress, string namespaceAndTypeName, string name, string signature)
{
    var fullName = signature;

    // constructor case: name = .ctor | namespaceAndTypeName = A.B.typeName | signature = ...  (parameters)
    // --> A.B.typeName(parameters)
    if (name == ".ctor")
    {
        return $"{namespaceAndTypeName}{ExtractParameters(signature)}";
    }

    // general case: name = Foo | namespaceAndTypeName = A.B.typeName | signature = ...  (parameters)
    // --> A.B.Foo(parameters)
    fullName = $"{namespaceAndTypeName}.{name}{ExtractParameters(signature)}";
    return fullName;
}

private string ExtractTypeName(string namespaceAndTypeName)
{
    var pos = namespaceAndTypeName.LastIndexOf(".", StringComparison.Ordinal);
    if (pos == -1)
    {
        return namespaceAndTypeName;
    }
    
    // skip the .
    pos++;

    return namespaceAndTypeName.Substring(pos);
}

Only the parameters (not the return type) are extracted from the “return type SPACE SPACE (parameters)” signature format:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


private string ExtractParameters(string signature)
{
    var pos = signature.IndexOf("  (");
    if (pos == -1)
    {
        return "(???)";
    }

    // skip double space
    pos += 2;

    var parameters = signature.Substring(pos);
    return parameters;
}

With the starting address and the size of each JITted methods, it is easy to find the one corresponding to a given address on the stack: look for the MethodInfo where this address could be between the start address and the start address + the code size:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


public string GetFullName(ulong address)
{
    if (_cache.TryGetValue(address, out var fullName))
        return fullName;
    
    // look for managed methods
    for (int i = 0; i < _methods.Count; i++)
    {
        var method = _methods[i];
        
        if ((address >= method.StartAddress) && (address < method.StartAddress + (ulong)method.Size))
        {
            fullName = method.FullName;
            _cache[address] = fullName;
            return fullName;
        }
    }

    // look for native methods
    fullName = GetNativeMethodName(address);
    _cache[address] = fullName;

    return fullName;
}

For performance sake, the _cache dictionary property speeds up the process by keeping track of the address/full name mappings.

It is now time to look at the details of the GetNativeMethodName helper that takes care of the native functions scenario.

The native part of the symbols story

Unlike for JITted methods, the CLR does not send events to describe native functions even for the CLR itself. Instead, you need to find a way to map a call stack address to a native function by yourself. Unlike Perfview, I will be using the dbghelp native API instead of DIA mostly because my scenario is to get the stacks while the applications are still running:

After reading the march 2002 MSDN article about DBGHELP by Matt Pietrek, the updated symbols related Microsoft Docs and the dbghelp.h include a file from the Windows SDK, I wrote a C# wrapper around the dbghelp function needed to get a method name from an address in a process address space:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


internal static class NativeDbgHelp
{
    // from C:\Program Files (x86)\Windows Kits\10\Debuggers\inc\dbghelp.h
    public const uint SYMOPT_UNDNAME = 0x00000002;
    public const uint SYMOPT_DEFERRED_LOADS = 0x00000004;

    [StructLayout(LayoutKind.Sequential)]
    public struct SYMBOL_INFO
    {
        public uint SizeOfStruct;
        public uint TypeIndex;      // Type Index of symbol
        private ulong Reserved1;
        private ulong Reserved2;
        public uint Index;
        public uint Size;
        public ulong ModBase;       // Base Address of module containing this symbol
        public uint Flags;
        public ulong Value;         // Value of symbol, ValuePresent should be 1
        public ulong Address;       // Address of symbol including base address of module
        public uint Register;       // register holding value or pointer to value
        public uint Scope;          // scope of the symbol
        public uint Tag;            // pdb classification
        public uint NameLen;        // Actual length of name
        public uint MaxNameLen;
        [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 1024)]
        public string Name;
    }

    [DllImport("dbghelp.dll", SetLastError = true)]
    public static extern bool SymInitialize(IntPtr hProcess, string userSearchPath, bool invadeProcess);

    [DllImport("dbghelp.dll", SetLastError = true)]
    public static extern uint SymSetOptions(uint symOptions);

    [DllImport("dbghelp.dll", SetLastError = true, CharSet = CharSet.Ansi)]
    public static extern ulong SymLoadModule64(IntPtr hProcess, IntPtr hFile, string imageName, string moduleName, ulong baseOfDll, uint sizeOfDll);

    // use ANSI version to ensure the right size of the structure 
    // read https://docs.microsoft.com/en-us/windows/win32/api/dbghelp/ns-dbghelp-symbol_info
    [DllImport("dbghelp.dll", SetLastError = true, CharSet = CharSet.Ansi)]
    public static extern bool SymFromAddr(IntPtr hProcess, ulong address, out ulong displacement, ref SYMBOL_INFO symbol);

    [DllImport("dbghelp.dll", SetLastError = true)]
    public static extern bool SymCleanup(IntPtr hProcess);
}

Note that you will need to download the dbghelp.dll (SymSrv.dll if needed) from the Windows SDK and copy it next to your memory profiler binaries.

The usage of the dbghelp API is straightforward. First, for each new process, call SymSetOptions/SymInitialize** **with a handle of the process:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


private bool SymInitialize(IntPtr hProcess)
{
    // read https://docs.microsoft.com/en-us/windows/win32/api/dbghelp/nf-dbghelp-symsetoptions for more details
    // maybe SYMOPT_NO_PROMPTS and SYMOPT_FAIL_CRITICAL_ERRORS could be used
    NativeDbgHelp.SymSetOptions(
        NativeDbgHelp.SYMOPT_DEFERRED_LOADS |   // performance optimization
        NativeDbgHelp.SYMOPT_UNDNAME            // C++ names are not mangled
        );

    // https://docs.microsoft.com/en-us/windows/win32/api/dbghelp/nf-dbghelp-syminitialize
    // search path for symbols:
    //   - The current working directory of the application
    //   - The _NT_SYMBOL_PATH environment variable
    //   - The _NT_ALTERNATE_SYMBOL_PATH environment variable
    //
    // passing false as last parameter means that we will need to call SymLoadModule64 
    // each time a module is loaded in the process
    return NativeDbgHelp.SymInitialize(hProcess, null, false);
}
private IntPtr BindToProcess(int pid)
{
    try
    {
        _process = Process.GetProcessById(pid);

        if (!SymInitialize(_process.Handle))
            return IntPtr.Zero;

        return _process.Handle;
    }
    catch (Exception x)
    {
        Console.WriteLine($"Error while binding pid #{pid} to DbgHelp:");
        Console.WriteLine(x.Message);
        return IntPtr.Zero;
    }
}

In the case of protected processes, Process.GetProcessById might throw an exception. The _hProcess field storing the process handle will be cleaned up in the IDisposible.Dispose implementation of the MethodStore:

1
2
3
4
5
6
7
8


public void Dispose()
{
    if (_hProcess == IntPtr.Zero)
        return;
    _hProcess = IntPtr.Zero;

    _process.Dispose();
}

After a process has been bound, each time one of its modules is loaded, SymLoadModule64 must be called. You can be notified of such a loaded module by enabling the Kernel provider with the ImageLoad keyword.

1
2
3
4
5


session.EnableKernelProvider(
    KernelTraceEventParser.Keywords.ImageLoad |
    KernelTraceEventParser.Keywords.Process,
    KernelTraceEventParser.Keywords.None
);

The handler attached to the ImageLoaded event will be called each time a dll gets loaded.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


private void SetupListeners(ETWTraceEventSource source)
{
    ...

    // get notified when a module is load to map the corresponding symbols
    source.Kernel.ImageLoad += OnImageLoad;
}

const int ERROR_SUCCESS = 0;
private void OnImageLoad(ImageLoadTraceData data)
{
    if (FilterOutEvent(data)) return;

    GetProcessMethods(data.ProcessID).AddModule(data.FileName, data.ImageBase, data.ImageSize);
}
public void AddModule(string filename, ulong baseOfDll, int sizeOfDll)
{
    var baseAddress = NativeDbgHelp.SymLoadModule64(_hProcess, IntPtr.Zero, filename, null, baseOfDll, (uint)sizeOfDll);
    if (baseAddress == 0)
    {
        // should work if the same module is added more than once
        if (Marshal.GetLastWin32Error() == ERROR_SUCCESS) return;

        Console.WriteLine($"SymLoadModule64 failed for {filename}");
    }
}

Now everything is in place to get a native function name from an address on the stack:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


private string GetNativeMethodName(ulong address)
{
    var symbol = new NativeDbgHelp.SYMBOL_INFO();
    symbol.MaxNameLen = 1024;
    symbol.SizeOfStruct = (uint)Marshal.SizeOf(symbol) - 1024;   // char buffer is not counted
    // the ANSI version of SymFromAddr is called so each character is 1 byte long

    if (NativeDbgHelp.SymFromAddr(_hProcess, address, out var displacement, ref symbol))
    {
        var buffer = new StringBuilder(symbol.Name.Length);

        // remove weird "$##" at the end of some symbols
        var pos = symbol.Name.LastIndexOf("$##");
        if (pos == -1)
            buffer.Append(symbol.Name);
        else
            buffer.Append(symbol.Name, 0, pos);

        // add offset if any
        if (displacement != 0)
            buffer.Append($"+0x{displacement}");

        return buffer.ToString();
    }

    // default value is the just the address in HEX
    return $"0x{address:x}";
}

Note that I needed to remove some unexpected $## strings are the end of some symbols.

This is the last episode of the series about building your own memory profiler in C#. In case you missed the first episodes, check them out on Medium:

Build your own .NET memory profiler in C# — call stacks (2/2–1) *This post explains how to get the call stack corresponding to the allocations with CLR events.*medium.com Build your own .NET memory profiler in C# *This post explains how to collect allocation details by writing your own memory profiler in C#.*medium.com

Resources

Source code available on Github.
Download Debugging Tools for Windows for dbghelp.dll and SymSrv.dll
Matt Pietrek article in MSDN Magazine about DBGHELP
Dbghelp samples -http://www.debuginfo.com/examples/dbghelpexamples.html

Join the crowd!

Careers at Criteo | Criteo jobs *Find opportunities everywhere. Choose your next challenge. Find the job opportunities at Criteo in Product, research &…*careers.criteo.com

Build your own .NET memory profiler in C# — call stacks (2/2–1)

Mon, 18 May 2020 12:07:01 +0000

In the previous episode of this series, you have seen how to get a sampling of .NET application allocations thanks to the AllocationTick and GCSampleObjectAllocation(High/Low) CLR events. However, this is often not enough to investigate unexpected memory consumption: you would need to know which part of the code is triggering the allocations. This post explains how to get the call stack corresponding to the allocations, again with CLR events.

Introduction

If you look carefully at the payload of the TraceEvent object mapped by Microsoft TraceEvent library (not my fault if they have the same name) for each CLR event, you won’t see anything related to a call stack. However, in the TraceEvent sample 41, the following line looks promising:

var callStack = data.CallStack();

with data being a TraceEvent object received for each CLR event!

This CallStack method is an extension method provided by the TraceLog special kind of event source. You might not have noticed but I have used it in the AllocationTick code sample from the previous post. This class (and many more helper classes) is doing a lot of work to :

“attach” a call stack to each CLR event; i.e. a list of addresses of assembly code
to translate addresses into string symbols (method names or full signatures), listen to a bunch of JIT related events for managed methods (more on this later), using COM-based Debug Interface Access (a.k.a. DIA) and MetadataReaderProvider** **for native functions

Notice that since events from all managed processes on the machine are handled by TraceLog, the internal cache for JITted methods description could consume a lot of memory. During my tests with two Visual Studio running, my test profiler consumed more than 500 MB before even handling call stacks. If you are in such an environment with multiple .NET processes, I will show how to “manually” get the same stacks (+ symbols in the next episode) with CLR events and a few methods from dbghelp.dll in a cheaper way.

The new provider (more on ClrRundown later), keywords and events need to be received to make all this work:

TraceLog: the easy way

As you have seen in the previous posts, the TraceEventSession class exposes a Source property of ETWTraceEventSource type. This source has event parsers properties from which you register handler methods that will be called when CLR events are received. Instead of directly using this source, you should wrap it with a TraceLogEventSource object that provides the same event parsers.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


await Task.Factory.StartNew(() =>
{
    using (_session)
    {
        SetupProviders(_session);

        using (TraceLogEventSource source = TraceLog.CreateFromTraceEventSession(_session))
        {
            SetupListeners(source);

            source.Process();
        }
    }
});

What’s new with providers?

The code for mySetupProviders method is a little bit different from the previous post even though no new event listeners are needed:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


private void SetupProviders(TraceEventSession session)
{
    // Note: the kernel provider MUST be the first provider to be enabled
    // If the kernel provider is not enabled, the callstacks for CLR events are still received 
    // but the symbols are not found (except for the application itself)
    // TraceEvent implementation details triggered when a module (image) is loaded
    session.EnableKernelProvider(
        KernelTraceEventParser.Keywords.ImageLoad |
        KernelTraceEventParser.Keywords.Process,
        KernelTraceEventParser.Keywords.None
    );

    session.EnableProvider(
        ClrTraceEventParser.ProviderGuid,
        TraceEventLevel.Verbose,    // this is needed in order to receive AllocationTick_V2 event
        (ulong)(
        // required to receive AllocationTick events
        ClrTraceEventParser.Keywords.GC |
        ClrTraceEventParser.Keywords.Jit |                      // Turning on JIT events is necessary to resolve JIT compiled code 
        ClrTraceEventParser.Keywords.JittedMethodILToNativeMap |// This is needed if you want line number information in the stacks
        ClrTraceEventParser.Keywords.Loader |                   // You must include loader events as well to resolve JIT compiled code.
        ClrTraceEventParser.Keywords.Stack
        )
    );

    // this provider will send events of already JITed methods
    session.EnableProvider(ClrRundownTraceEventParser.ProviderGuid, TraceEventLevel.Informational,
    (ulong)(
        ClrTraceEventParser.Keywords.Jit |              // We need JIT events to be rundown to resolve method names
        ClrTraceEventParser.Keywords.JittedMethodILToNativeMap | // This is needed if you want line number information in the stacks
        ClrTraceEventParser.Keywords.Loader |           // As well as the module load events.  
        ClrTraceEventParser.Keywords.StartEnumeration   // This indicates to do the rundown now (at enable time)
        ));

}

The kernel provider needs to be enabled with the ImageLoad and Process keywords in order to let TraceEvent detect when a process loads “images” (i.e. dlls) and at which address (needed to convert Relative Virtual Addresses (RVA) to addresses in the address space). Note that this provider must be enabled before any other provider or your code will trigger an exception.
The CLR provider needs to be enabled with Jit, JittedMethodILToNativeMap, and Loader (in addition to the usual GC one).
The Stack keyword has to be set on the same CLR provider to receive call stacks events for “normal” CLR event (more on this later)
The CLR Rundown provider is enabled with the same Jit, JittedMethodILToNativeMap, and Loader keywords. That way, JIT events corresponding to already JITted methods will be received (not only the new ones). This is important because otherwise, you won’t be able to map these methods with the address in memory of their JITted native code in the case of processes that have been started before the profiler. This is the case for my AllocationTickProfiler sample.

Callstacks and symbols

Now, when an AllocationTick event is received, calling the CallStack extension method on the GCAllocationTickTraceData argument returns a TraceCallStack object. This class is a linked list of TraceCodeAddress representing each stack frame (i.e. address in assembly code). These classes are at the heart of TraceEvent and Perfview callstack management. The method names and signatures are retrieved behind the scene thanks to JIT events and the SymbolReader class that digs into .pdb files.

You first need to initialize a SymbolReader instance:

Set the path to find the .pdb; including the Microsoft HTTP endpoint for public .NET versions symbols,
Allow pdb next to the executable to be loaded.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


// By default a symbol Reader uses whatever is in the _NT_SYMBOL_PATH variable.  However you can override
// if you wish by passing it to the SymbolReader constructor.  Since we want this to work even if you 
// have not set an _NT_SYMBOL_PATH, so we add the Microsoft default symbol server path to be sure/
var symbolPath = new SymbolPath(SymbolPath.SymbolPathFromEnvironment).Add(SymbolPath.MicrosoftSymbolServerPath);
_symbolReader = new SymbolReader(_symbolLookupMessages, symbolPath.ToString());

// By default the symbol reader will NOT read PDBs from 'unsafe' locations (like next to the EXE)  
// because hackers might make malicious PDBs. If you wish ignore this threat, you can override this
// check to always return 'true' for checking that a PDB is 'safe'.  
_symbolReader.SecurityCheck = (path => true);

Then, displaying a TraceCallStack from a received CLR event in a human-readable format is simple:

Get one frame after the other from the linked list,
If the CodeAddress field is not cached yet, load the symbols for its module,
Display the FullMethodName field of the frame (or the address if not found).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


private void DumpStack(TraceCallStack callStack)
{
    while (callStack != null)
    {
        var codeAddress = callStack.CodeAddress;
        if (codeAddress.Method == null)
        {
            var moduleFile = codeAddress.ModuleFile;
            if (moduleFile == null)
            {
                Debug.WriteLine($"Could not find module for Address 0x{codeAddress.Address:x}");
            }
            else
            {
                codeAddress.CodeAddresses.LookupSymbolsForModule(_symbolReader, moduleFile);
            }
        }
        if (!string.IsNullOrEmpty(codeAddress.FullMethodName))
            Console.WriteLine($"     {codeAddress.FullMethodName}");
        else
            Console.WriteLine($"     0x{codeAddress.Address:x}");
        callStack = callStack.Caller;
    }
}

Note that the first frame in the linked list is the last on the stack (i.e. last executed method).

As I mentioned at the beginning of the post, I have been facing OutOfMemory errors due to the TraceEvent symbols management large memory usage when a few other .NET applications were running. Let’s see how to get the call stacks in a less memory consuming way.

Manually rebuilding the allocations call stack

Instead of using the call stack and symbol management provided by TraceLog in TraceEvent, I would prefer to manually get them. If you remember the last post, thanks to GCSampledObjectAllocation CLR events, it is possible to have a sampling of the allocation size and count per process and per type. What I would like to add to the type picture is the list of call stacks leading to these allocations.

How to manually get CLR events call stack

The first step is to understand how to get the CLR events call stacks. If you use the TraceLog-based code just presented, you should see the following kind of call stack:

The ETWCallout CLR helper function is in charge of sending a special event containing the call stack of other normal events from the four supported CLR providers. If you set the Stack keyword to the CLR provider, each time an event is sent by a thread, a ClrStackWalk event will be sent just after. It means after each SampleObjectAllocation event, a ClrStackWalk event containing the call stack will be immediately received. In fact, since an application will probably be using more than one thread, it is required to do the mapping between the two events on a per-thread basis.

Each allocation event received by the OnSampleObjectAllocation handler contains the ThreadID property so it is easy to keep track of the last received allocation event per thread. In my case, the ProcessAllocations class stores this information in its _perThreadLastAllocation field:

1
2
3
4
5


public class ProcessAllocations
{
    ...
    private readonly Dictionary<string, AllocationInfo> _allocations;
    private readonly Dictionary<int, AllocationInfo> _perThreadLastAllocation;

Now, each time a SampleObjectAllocation event is received, the id of the sending thread is passed to the updatedProcessAllocations.AddAllocation() method:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


public AllocationInfo AddAllocation(int pid, ulong size, ulong count, string typeName)
{
    if (!_allocations.TryGetValue(typeName, out var info))
    {
        info = new AllocationInfo(typeName);
        _allocations[typeName] = info;
    }

    info.AddAllocation(size, count);

    // the last allocation is still here without the corresponding stack
    if (_perThreadLastAllocation.TryGetValue(pid, out var lastAlloc))
    {
        Console.WriteLine("no stack for the last allocation");
    }

    // keep track of the allocation for the given thread
    // --> will be used when the corresponding call stack event will be received
    _perThreadLastAllocation[pid] = info;

    return info;
}

The _perThreadLastAllocation dictionary stores the AllocationInfo per thread. If an allocation happens, it is added into the dictionary. When a ClrStackWalk event is received for a given thread, the stack will be associated with the last AllocationInfo and removed from the dictionary. If some events are missed (it never happens during my tests but who knows), error message could be logged.

The ClrStackWalkTraceData argument received by the ClrStackWalk listener has a FrameCount property that returns the number of frames in the call stack. In addition, its InstructionPointer() method takes a frame position in the stack (starting at 0) and returns the address (in assembly code) at this position on the call stack.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


private void OnClrStackWalk(ClrStackWalkTraceData data)
{
    if (FilterOutEvent(data)) return;

    var callstack = BuildCallStack(data);
    GetProcessAllocations(data.ProcessID).AddStack(data.ThreadID, callstack);
}
private AddressStack BuildCallStack(ClrStackWalkTraceData data)
{
    var length = data.FrameCount;
    AddressStack stack = new AddressStack(length);

    // frame 0 is the last frame of the stack (i.e. last called method)
    for (int i = 0; i < length; i++)
    {
        stack.AddFrame(data.InstructionPointer(i));
    }

    return stack;
}

The AddressStack class returned by BuildCallStack stores the frames as a list of addresses so it can be stored in AllocationInfo.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36


public class AddressStack
{
    // the first frame is the address of the last called method 
    private readonly List<ulong> _stack;

    public AddressStack(int capacity)
    {
        _stack = new List<ulong>(capacity);
    }

    // No need to override GetHashCode because we don't want to use it as a key in a dictionary
    public override bool Equals(object obj)
    {
        if (obj == null) return false;

        var stack = obj as AddressStack;
        if (stack == null) return false;

        var frameCount = _stack.Count;
        if (frameCount != stack._stack.Count) return false;

        for (int i = 0; i < frameCount; i++)
        {
            if (_stack[i] != stack._stack[i]) return false;
        }

        return true;
    }

    public IReadOnlyList<ulong> Stack => _stack; 

    public void AddFrame(ulong address)
    {
        _stack.Add(address);
    }
}

This class overrides the Equals method for a single reason: I want to be able to detect when the “same” stack (i.e. with the exact same frame addresses) is received for a given type allocation. That way, I just need to keep a counter for each different AddressStack and not all call stacks in AllocationInfo. Remember that AllocationInfo is used to keep track of allocations per type:

1
2
3
4
5
6


public class AllocationInfo
{
    private readonly string _typeName;
    private ulong _size;
    private ulong _count;
    private List<StackInfo> _stacks;

The StackInfo class contains an AddressStack and how many times it led to this type of allocation.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


public class StackInfo
{
    private readonly AddressStack _stack;
    public ulong Count;

    internal StackInfo(AddressStack stack)
    {
        Count = 0;
        _stack = stack;
    }

    public AddressStack Stack => _stack;
}

So, when a stack event is received, AddStack is called on the last AllocationInfo for the same thread:

1
2
3
4
5
6
7
8
9


public void AddStack(int tid, AddressStack stack)
{
    if (_perThreadLastAllocation.TryGetValue(tid, out var lastAlloc))
    {
        lastAlloc.AddStack(stack);
        _perThreadLastAllocation.Remove(tid);
        return;
    }
}

The job of AllocationInfo.AddStack() the method is to check if a previous allocation was made with the same call stack (hence the Equals override). If this is the case, just increment the corresponding StackInfo count. Otherwise, create a new StackInfo for this call stack with a count set to 1.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


internal void AddStack(AddressStack stack)
{
    var info = GetInfo(stack);
    if (info == null)
    {
        info = new StackInfo(stack);
        _stacks.Add(info);
    }

    info.Count++;
}

private StackInfo GetInfo(AddressStack stack)
{
    for (int i = 0; i < _stacks.Count; i++)
    {
        var info = _stacks[i];
        if (stack.Equals(info.Stack)) return info;
    }

    return null;
}

Knowing the address in code of each frame for all events call stack is nice but it would be much more useful to translate them into method names… You have to deal with two different cases: managed and native methods. I will cover these topics in the next episode.

Resources

Source code available on Github.
TraceEvent sample 41 source code.

Missed the first part of this story? Check this out:

Build your own .NET memory profiler in C# *This post explains how to collect allocation details by writing your own memory profiler in C#.*medium.com

Interested in joining our journey? Check this out:

Product, Research & Development | Criteo Careers careers.criteo.com