As .NET developers, we often benchmark the performance of our methods using a single .NET version. However, we can also benchmark a method’s performance across different .NET versions. Such benchmarks can come in handy when we want to upgrade our codebase to a newer version. In such scenarios, we can use the benchmarks to identify potential performance improvements or downgrades that may come with the upgrade.
In this article, let’s explore how to benchmark a method’s performance across different .NET versions using the BenchmarkDotNet library.
Alright, let’s dive in.
Prepare the Benchmark Environment
Before we define the method we want to benchmark, let’s create a project from the command-line interface:
dotnet new console -o BenchmarkAcrossDifferentDotNETVersions
Next, let’s add the BenchmarkDotNet package to this project:
dotnet add package BenchmarkDotNet
Finally, we need to download the various .NET SDKs we want to use for our benchmark from the .NET download page. For this tutorial, we will be using the .NET 6.0, .NET 7.0, and .NET 8.0 SDKs.
After downloading and installing these SDKs, let’s configure our project to work with them. We can achieve this by modifying our .csproj file:
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFrameworks>net6.0;net7.0;net8.0</TargetFrameworks>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="BenchmarkDotNet" Version="0.13.12" />
</ItemGroup>
</Project>
This configuration is quite straightforward, we simply replace the <TargetFramework>netx.0</TargetFramework> line in our .csproj file with <TargetFrameworks>net6.0;net7.0;net8.0</TargetFrameworks>.
Now, let’s define the method we wish to benchmark:
public record Book(string Title, string Author, string Publisher);
public class BookService
{
public static List<Book> GetBooks()
=> new()
{
new("Measuring Time", "Helon Habila", "W. W. Norton & Company"),
new("Americanah", "Chimamanda Adichie", "Alfred Knopf"),
new("Things Fall Apart", "Chinua Achebe", "William Heinemann Ltd.")
};
}
We start by creating a Book record with three positional parameters. Then, we define a BookService class with a single method that returns a list of books.
It’s important to note that our method has to be in a version of C# that is compatible with all the .NET versions we want to test. For example, we can’t use collection expressions in this method, as they were introduced in C# 12 with the .NET 8 SDK. This feature is not available in C# 10 and C# 11, which were shipped with the .NET 6 and .NET 7 SDKs, respectively.
With this, our setup is complete. Let’s now proceed to creating and executing our benchmark class.
Create and Run Our Benchmark Across Different .NET Versions
First, let’s create a GetBooksBenchmark class:
[SimpleJob(RuntimeMoniker.Net60, baseline: true)]
[SimpleJob(RuntimeMoniker.Net70)]
[SimpleJob(RuntimeMoniker.Net80)]
[Config(typeof(StyleConfig))]
public class GetBooksBenchmark
{
private class StyleConfig : ManualConfig
{
public StyleConfig()
=> SummaryStyle = SummaryStyle.Default.WithRatioStyle(RatioStyle.Trend);
}
[Benchmark]
public List<Book> GetBooks() => BookService.GetBooks();
}
Here, we use the [SimpleJob(RuntimeMoniker.Netxx)] attribute to specify that our benchmark should be executed with the .NET 6.0, .NET 7.0, and .NET 8.0 runtimes. Also, we designate the .NET 6.0 runtime as the baseline for our benchmark. This means that BenchmarkDotnet will compare the results we get from running our method with the other runtimes to those obtained from running our method with .NET 6.0.
For more informative benchmark results, we apply a custom configuration to the class using the [Config(typeof(StyleConfig))] attribute. This attribute directs our benchmark class to the style configuration we define in the StyleConfig class.
In the StyleConfig class, we define a StyleConfig() method that sets the SummaryStyle property to SummaryStyle.Default.WithRatioStyle(RatioStyle.Trend). Essentially, we are specifying that we want to use the default summary style settings, but that the ratio column in our results should show the performance trends between the .NET 6.0 results and those of the other runtimes. Kindly note that we make the StyleConfig class a private member of our benchmark class because it’s the only class we have that accesses its configuration.
Finally, to indicate that the GetBooks() method is the method we intend to benchmark, we apply the Benchmark attribute to it.
As a final step, let’s add code that will execute our benchmark to the Program.cs class:
BenchmarkRunner.Run<GetBooksBenchmark>();
With that, let’s run our project and inspect the benchmark results:
| Method | Job | Runtime | Mean | Error | StdDev | Median | Ratio | RatioSD | |--------- |--------- |--------- |---------:|---------:|---------:|---------:|-------------:|--------:| | GetBooks | .NET 6.0 | .NET 6.0 | 55.06 ns | 1.171 ns | 1.753 ns | 54.90 ns | baseline | | | GetBooks | .NET 7.0 | .NET 7.0 | 56.78 ns | 1.596 ns | 4.476 ns | 55.11 ns | 1.06x slower | 0.09x | | GetBooks | .NET 8.0 | .NET 8.0 | 42.95 ns | 1.952 ns | 5.601 ns | 40.80 ns | 1.27x faster | 0.18x |
From these benchmark results, we can view the performance of our GetBooks() method when executed with .NET 6.0, .NET 7.0, and .NET 8.0. In the Ratio column, we see how much faster or slower calling our method in .NET 7.0 and .NET 8.0 is as compared to .NET 6.0.
So, that’s it. We’ve successfully compared our method’s performance across three .NET versions.
Conclusion
In this article, we’ve explored the steps and configurations required in benchmarking a method against different .NET versions. We first added the multi-target framework attribute to our csproj file. Then in our benchmark class, we used the [SimpleJob(RuntimeMoniker.Netxx)] attribute to specify which .NET runtimes we want to execute our benchmark.
Finally, to make the results more informative, we added a baseline and specified that the Ration column should display the performance trend across the different versions related to the baseline.
