.NET Core Diagnostics in Docker

In .NET Core 3.0, Microsoft introduced a suite of tools that use new features in the .NET runtime that make it easier to diagnose and solve performance problems.

These runtime features help you answer some common diagnostic questions you may have:

1. Is my application healthy?
2. Why does my application have anomalous behavior?
3. Why did my application crash?

The Tools

dotnet-counters

The dotnet-counters tool is a new command-line tool for observing metrics emitted by .NET Core Applications in real time. This metrics allow to identify and analyse situations where an application can slowly start leaking memory and eventually result in an out of memory exception, or spikes in CPU utilization.

dotnet-trace

The dotnet-trace tool is a new cross-platform tool that enables the collection of .NET Core traces of a running process without any native profiler involved.

While metrics help identify the occurence of anomalous behavior, they offer little visibility into what went wrong. To answer the question why your application has anomalous behavior you need to collect additional information via traces. As an example, CPU profiles collected via tracing can help you identify the hot path in your code.

dotnet-dump

The dotnet-dump tool is a way to collect and analyze the managed data structures in Windows and Linux dumps all without any native debugger involved. There are cases where it’s not possible to identify what caused an anomalous behavior by just tracing the process. In the event that the process crashed or situations where we may need more information like access to entire process heap, a process dump may be more suitable for analysis.

Collect Diagnostics in Docker Containers

There is already a lot of documentation on how to install and use these tools, so in this blog post I will focus on how to install and use one of the tools in a Docker container.

To install the tools in the container there are two possibilities:

1. Install the tools anytime via a single-file distribution mechanism that only requires a runtime (3.1+)
2. Install the tools in the initial Docker image

1. Install the tools via single-file distribution mechanism

Until recently, the .NET diagnostics suite of tools was available only as .NET SDK global tools. While this provided a convenient way to acquire and update the tools, this meant it was difficult to acquire them in environments where the full SDK was not present. Microsoft now provides a single-file distribution mechanism that only requires a runtime (3.1+) to be available on the target machine.

The latest version of the tools is always available at a link that follows the following schema:

<https://aka.ms/><tool-name>/<platform-runtime-identifier>

For example, if you’re running .NET Core on x64 Ubuntu, you can acquire the dotnet-trace from https://aka.ms/dotnet-trace/linux-x64.

2. Install the tools in the initial Docker image

It’s possible to create a Dockerfile with a multi-stage build that installs the tools in a build stage and then copies the binaries into the final image. The only downside to this approach is increased Docker image size.

# In build stage
# Install desired .NET CLI diagnostics tool
RUN dotnet tool install --tool-path /tools dotnet-dump
...
# In final stage
# Copy diagnostics tools
WORKDIR /tools
COPY --from=build /tools .

Accessing the tools at runtime

In order to access these tools at runtime, we need to be able to access the container at runtime. The diagnostic tools only work with .NET Core 3.1 or above. We can run the following command to get the running containers:

docker ps

Which results in output similar to the following:

From here, we can use the docker exec command to launch a shell in the new container, using the container ID from above:

docker exec -it -w /tools <ID> /bin/sh

For example, now you should be able to run dotnet-dump as normal. You can list the processes that can be used by the tool with:

./dotnet-dump ps

If the process is not listed, it won’t be possible to collect the dump.

To collect the dump from a process you can use:

./dotnet-dump collect -p <PID> --type Full -o /tmp/dump1.core

Once capture completed, we should be able to see dump file in the “/tmp” directory.