In this article, we will learn how to set a baseline for our benchmark in BenchmarkDotNet and discuss styling the baseline results.
As software developers, benchmarking the performance of our methods is a crucial part of our day-to-day activities. With the BenchmarkDotNet library, we can accurately perform this operation. When we use this library for our benchmarking, there are numerous functionalities that we can utilize to make our benchmark results more insightful. One such functionality is defining a baseline for our benchmark.
Without further ado, let’s dive in.
Baselines in BenchmarkDotNet
Let’s start with a quick definition of what a baseline is. In BenchmarkDotnet, a baseline is any method in our benchmark class that we want the other methods to compare to. When we define this baseline, it becomes the standard for assessing the performance of the other methods in our benchmark class.
To illustrate this, let’s create a benchmark class:
public class BaselineStylingBenchmark(int finalNumber = 2000)
{
[Benchmark(Baseline = true)]
public int UseForLoop()
{
var sum = 0;
for (int i = 1; i <= finalNumber; i++)
{
sum += i;
}
return sum;
}
[Benchmark]
public int UseWhileLoop()
{
var sum = 0;
int i = 1;
while (i <= finalNumber)
{
sum += i;
i++;
}
return sum;
}
[Benchmark]
public int UseEnumerableSum()
=> Enumerable.Range(1, finalNumber).Sum();
}
In this class, we have three methods that sum all the numbers ranging from 1 to a specified final number. Here, we set a default value of 2000 for the final number.
Most importantly, we set the UseForLoop() method as the baseline for this benchmark by simply placing the [Benchmark(Baseline = true)] attribute on it. It should be noted that the default value for Baseline is false and so explicitly setting it to false is the same as not including it.
Now, let’s execute this benchmark and check the results:
| Method | Mean | Error | StdDev | Ratio | RatioSD | |----------------- |-----------:|---------:|---------:|------:|--------:| | UseForLoop | 566.8 ns | 0.89 ns | 0.79 ns | 1.00 | 0.00 | | UseWhileLoop | 573.4 ns | 6.81 ns | 6.04 ns | 1.01 | 0.01 | | UseEnumerableSum | 3,455.0 ns | 62.10 ns | 58.09 ns | 6.10 | 0.10 |
As we can see from these results, there are two additional columns, Ratio and RatioSD. These columns show the relationship between our baseline and the other methods.
At first glance, we are okay with the results as they accurately show the performance relationships. However, these results are not intuitive. These columns display the comparison results as decimals and do not offer any additional context. To understand these results, we would first need to perform some calculations.
However, with some styling, we can make these results better.
Styling the Baseline Ratio Column in BenchmarkDotNet
To style the ratio columns, first, we need to create a configuration class that inherits from ManualConfig:
public class StyleConfig : ManualConfig
{
public StyleConfig()
{
SummaryStyle = SummaryStyle.Default.WithRatioStyle(RatioStyle.Trend);
}
}
In the constructor of this class, we change the style of the ratio columns by adding a SummaryStyle value. The SummaryStyle property allows us to set the style of our benchmark summary.
Here, with the SummaryStyle.Default field, we tell BenchmarkDotNet that we want the default summary style. Then, we pass the RatioStyle.Trend enum to the WithRatioStyle() method to specify that the results in our ratio columns should be performance trends and not just decimals.
Note that we can also use the RatioStyle.Percentage enum to display our results as percentages.
With that, let’s add the [Config(typeof(StyleConfig))] attribute to our benchmark class:
[Config(typeof(StyleConfig))]
public class BaselineStylingBenchmark(int finalNumber = 2000)
{
// Our benchmark code
}
Finally, let’s execute the benchmark class again and see the new results:
| Method | Mean | Error | StdDev | Ratio | RatioSD | |----------------- |-----------:|--------:|--------:|-------------:|--------:| | UseForLoop | 567.8 ns | 2.22 ns | 1.96 ns | baseline | | | UseWhileLoop | 565.8 ns | 0.38 ns | 0.36 ns | 1.00x faster | 0.00x | | UseEnumerableSum | 3,379.6 ns | 3.37 ns | 3.16 ns | 5.95x slower | 0.02x |
Here, we can see that the results in the ratio columns provide the same information as before, but this time, in a more insightful format.
With this, we quickly see if a method is faster or slower than the baseline without performing any calculations.
Conclusion
In this blog post, we discussed the Baseline construct in BenchmarkDotNet and how to style it. We also saw how this styling makes our benchmark results easier to read and analyze, helping to highlight the performance difference between methods.
