In this article, we will learn how to integrate BenchmarkDotNet with Unit Tests and use some advanced APIs of the BenchmarkDotNet library when doing that.
In the .NET ecosystem, BenchmarkDotNet is the go-to library for conducting performance benchmarks. Typically we encounter this library being used in a console application to run these benchmarks. However, the process is manual, requiring developers to trigger the application and examine the results on the console.
Automating these benchmark tests and validating the resulting values would be incredibly helpful by incorporating them into unit tests. By doing so, we can execute these unit tests as part of our CI/CD pipeline, enabling us to receive continuous feedback on our code performance after each build.
Let’s dive in.
Benchmark Test Cases
We will keep the benchmark tests code simple, showcasing the various string concatenation techniques in C#. This is the summary of the test results executed on our machine (keep in mind that benchmark results may differ across various machines):
| Method | Mean | Error | StdDev | Median | Rank | Gen0 | Allocated | |--------------------- |-----------:|----------:|----------:|-----------:|-----:|-------:|----------:| | StringConcat | 0.0000 ns | 0.0000 ns | 0.0000 ns | 0.0000 ns | 1 | - | - | | StringInterpolation | 0.0091 ns | 0.0133 ns | 0.0124 ns | 0.0005 ns | 1 | - | - | | StringConcatWithJoin | 28.0945 ns | 0.4573 ns | 0.4054 ns | 27.9739 ns | 2 | 0.0095 | 80 B | | StringFormat | 66.2081 ns | 0.7046 ns | 0.6591 ns | 66.1982 ns | 3 | 0.0048 | 40 B |
We will be focussing only on the StringInterpolation test case and assert its values. As the execution of benchmark test cases can take considerable time, we want to run the performance benchmark only once. To achieve this, we will utilize the xUnit TestFixture class.
Let’s look at the code:
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Order;
using BenchmarkDotNet.Reports;
using BenchmarkDotNet.Running;
using BenchmarkDotNetWithUnitTests;
namespace Tests;
public class BenchmarkFixture
{
public Summary BenchmarkSummary { get; }
public BenchmarkFixture()
{
var config = new ManualConfig
{
SummaryStyle = SummaryStyle.Default.WithMaxParameterColumnWidth(100),
Orderer = new DefaultOrderer(SummaryOrderPolicy.FastestToSlowest),
Options = ConfigOptions.Default
};
BenchmarkSummary = BenchmarkRunner.Run(config);
}
}
In the constructor, we run the string concat benchmarks with the appropriate configuration, exposing the results as the BenchmarkSummary property of type Summary. We then initialize this property in the class constructor.
So now the text fixture class is done, let’s use it in our test class:
public class StringConcatBenchmarkLiveTest : IClassFixture
{
private readonly ImmutableArray<BenchmarkReport> _benchmarkReports;
private readonly BenchmarkReport _stringInterpolationReport;
public StringConcatBenchmarkLiveTest(BenchmarkFixture benchmarkFixture)
{
_benchmarkReports = benchmarkFixture.BenchmarkSummary.Reports;
_stringInterpolationReport = _benchmarkReports.First(x =>
x.BenchmarkCase.Descriptor.DisplayInfo == "StringConcatBenchmarks.StringInterpolation");
}
}
We hold the reference to the benchmark’s calculated reports and assign it to an immutable collection field _benchmarkReports. Additionally, we fetch the report related to StringInterpolation and hold a reference on the _stringInterpolationReport field. Both these fields are used on test assertions.
Integrate BenchmarkDotNet With Unit Tests By Asserting the Count
Now that the test setup is complete, it’s time to execute the tests. Firstly, let’s go over the basics. We need to validate the number of test cases that are being executed.
Let’s understand with the code snippet:
[Fact]
public void WhenBenchmarkTestsAreRun_ThenFourCasesMustBeExecuted()
{
var benchmarkCases = _benchmarkReports.Length;
Assert.Equal(4, benchmarkCases);
}
We create a benchmarkCases variable and check if its value, which is the number of benchmark reports in the array, equals 4 using xUnit’s Assert.Equal(). If not, then the test fails.
Asserting the Speed
After verifying the count, we can proceed to test the execution speed of the StringInterpolation benchmark case:
[Fact]
public void WhenStringInterpolationCaseIsExecuted_ThenItShouldNotTakeMoreThanFifteenNanoSecs()
{
var stats = _stringInterpolationReport.ResultStatistics;
Assert.True(stats is { Mean: < 15 }, $"Mean was {stats.Mean}");
}
We need to obtain the statistical results of the benchmark case, and for that, we extract the ResultStatistics value from the _stringInterpolationReport object. We then verify the mean value of the result statistics (stats.Mean) using the Assert.True() method to ensure it is below 15 nanoseconds. If the condition is unmet, the test fails and displays a message containing the actual mean value.
Asserting the Memory Allocation
Let’s move on to some advanced stuff now by checking the memory allocation of the StringInterpolation code is within a particular threshold value:
[Fact]
public void WhenStringInterpolationCaseIsExecuted_ThenItShouldNotConsumeMemoryMoreThanMaxAllocation()
{
const int maxAllocation = 1342178216;
var memoryStats = _stringInterpolationReport.GcStats;
var stringInterpolationCase = _stringInterpolationReport.BenchmarkCase;
var allocation = memoryStats.GetBytesAllocatedPerOperation(stringInterpolationCase);
var totalAllocatedBytes = memoryStats.GetTotalAllocatedBytes(true);
Assert.True(allocation <= maxAllocation, $"Allocation was {allocation}");
Assert.True(totalAllocatedBytes <= maxAllocation,
$"TotalAllocatedBytes was {totalAllocatedBytes}");
}
We begin by defining a constant variable maxAllocation that represents the maximum allowed allocation. We then retrieve the garbage collection statistics (GcStats) of the benchmark case from the _stringInterpolationReport object. Then we also retrieve the benchmark case itself using the same object to obtain the stringInterpolationCase. After this, we calculate the allocation in bytes per operation by passing stringInterpolationCase it as a parameter to GetBytesAllocatedPerOperation() on the memoryStats object.
We validate two conditions using the Assert.True() method for validating memory allocated per operation and total memory allocated for all operations:
- We check if the allocation per operation (
allocation) is less than or equal to themaxAllocation. If the condition is false, the test fails with a failure message, including the actual allocation. - We also check if the total allocated bytes (
memoryStats.GetTotalAllocatedBytes(true)) is less than or equal to themaxAllocation. If the condition is false, the test fails with a failure message that includes the total allocated bytes.
Asserting the Gen1 and Gen2 Allocation
We’ll now look closer at memory allocations and examine the Gen0 and Gen1 allocations for the StringInterpolation benchmark case:
[Fact]
public void WhenStringInterpolationCaseIsExecuted_ThenZeroAllocationInGen1AndGen2()
{
var memoryStats = _stringInterpolationReport.GcStats;
Assert.True(memoryStats.Gen1Collections == 0, $"Gen1Collections was {memoryStats.Gen1Collections}");
Assert.True(memoryStats.Gen2Collections == 0, $"Gen2Collections was {memoryStats.Gen2Collections}");
}
We retrieve the garbage collection statistics (GcStats) of the benchmark case from the _stringInterpolationReport object. We then use the Assert.True() method is used to perform two assertions.
First, we check if the number of Gen1 garbage collections (memoryStats.Gen1Collections) is equal to zero. If the condition is false, the test fails with a failure message, including the number of Gen1 collections.
Then, we check if the number of Gen2 garbage collections (memoryStats.Gen2Collections) is equal to zero. If the condition is false, the test fails with a failure message, including the actual number of Gen2 collections.
Conclusion
It’s not enough to measure performance once and be done with it. In this article, we discussed how to integrate BenchmarkDotNet with unit tests and emphasize the significance of that integration. This approach provides ongoing feedback on our code and boosts our confidence when making significant changes.
We also explore utilizing advanced APIs to measure crucial factors of the benchmark, like speed and memory allocations. Automation is critical, and we covered how to achieve it.
