Introduction to Benchmarking in C# and ASP.NET Core Projects

In this article, we are going to cover how to benchmark C# code and ASP.NET Core projects. To accomplish this, we are going to set up a benchmarking test on a simple project and then apply the same concept to a real-world kind of application (comparing gRPC and REST).

Let’s dive in.

Performance and Benchmarking

To get us started, first, we need to talk a little bit about performance and benchmarking. When we are evaluating software, we are looking for code quality, use of design patterns, and performance. Evaluating and improving performance is probably one of the hardest parts of developing software, but it’s what makes our product or service valuable. Performance is how well an application or a unit within the application performs (usually in terms of responsiveness) under a particular workload.

Good software performs well. In other words, good software is “fast” under heavy workloads.

So how do we know how “fast” our code is?

Benchmarks.

We can think of a benchmark like a timer. When we are attempting to optimize our code, we can use benchmarks to verify whether our code has truly been optimized. Suppose we have an ASP.NET Core REST API and we wish to optimize our project by migrating our REST API to gRPC. Is this optimization worth it? There are many other uses we can consider. Should we use Razor pages or MVC patterns? Should we use a custom ORM or EF Core? These are questions that can be difficult to answer without running proper benchmarks. In our example for this article, we are going to look at how the REST protocol compares to the gRPC protocol.

We can use benchmarks to compare the performance of our original code versus the performance of our optimized code. Based on those benchmarks, we can make the appropriate decision.

That’s enough theory for now. Let’s take a look at what benchmarking really looks like.

Simple Benchmarking Project

To get us familiar with BenchmarkDotNet, we are going to start with a simple C# project. A simple benchmark project has 3 components: the client class, the benchmark class, and the program class. The client class is the class that is under question. The benchmark class executes the client class and registers the execution as a benchmark into BenchmarkRunner. And lastly, the Program class runs the BenchmarkRunner.

First, let’s define our SimpleProject.csproj:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net5.0</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="BenchmarkDotNet" Version="0.12.1" />
  </ItemGroup>
</Project>

In the project file, we are specifying a package reference to BenchmarkDotNet. This allows us to use BenchmarkDotNet as our benchmarking tool. When we run or build the project, dotnet will install the NuGet package. We can also install the package manually:

dotnet add package BenchmarkDotnet

Secondly, let’s define BottleneckProcess.cs:

public class BottleneckProcess
{
    public string GetLastItem(string csvString)
    {
        var items = csvString.Split(",");
        var lastItem = items.LastOrDefault();
        return lastItem ?? string.Empty;
    }
}

For the sake of this example, the BottleneckProcess class is the class we are interested in. The class only has one method, GetLastItem, which is supposed to get the last item in a comma-separated string. This is the method we want to benchmark.

Now, we can define BottleneckProcessBenchmark.cs:

public class BottleneckProcessBenchmark
{
    private const string csvString = "Code,Maze";
    private static readonly BottleneckProcess process = new BottleneckProcess();

    [Benchmark]
    public void GetLastItem()
    {
        process.GetLastItem(csvString);
    }
}

In this class, we are initializing a string object and a BottleneckProcess object. We are also defining a GetLastItem method that executes the method in BottleneckProcess. Lastly, we add the Benchmark attribute to register the method as a benchmark in the BenchmarkRunner. This means that the benchmark is going to measure how fast it can find and return “Maze” from “Code,Maze”.

Finally, we can create Program.cs:

public class Program
{
    public static void Main(string[] args)
    {
        var summary = BenchmarkRunner.Run<BottleneckProcessBenchmark>();
    }
}

In this class, we are executing Run from the BenchmarkRunner class. The Run method is going to execute all the methods with the Benchmark attribute of the specified class (BottleneckProcessBenchmark).

To execute our benchmarks, we are going to run the project in Release configuration:

dotnet run -c Release

When the program finishes, a BenchmarkDotNet.Artifacts folder is created with results in various file formats. Let’s take a look at the markdown file:

That’s it!

Let’s take a close look at these benchmark results.

Firstly, the method column gives a list of all the methods that have been benchmarked. This column is more critical when several methods need to be benchmarked as we are going to see in the next example.

The mean column gives us the average (or the arithmetic mean) of all measurements for the corresponding method. So in this example, the GetLastItem method takes about 125.2 nanoseconds to execute on average. We use an average because there is no way to measure the absolute execution duration of a method. BenchmarkDotNet executes this method several times and records the duration of each execution to give us an average.

The error column refers to half of the 99.9% confidence interval. In other words, we can be 99.9% sure that the actual mean is within 2.55 nanoseconds of the sampled mean.

The standard deviation (StdDev) column tells us how much the executions varied from the mean. A small standard deviation generally means that the sample mean is a good representation of the actual mean. Because we’re dealing in the world of nanoseconds, we can certainly trust these results.

Lastly, the median column gives us the midpoint of the dataset and along with the other statistical values tells us how reliable the sampled mean is. BenchmarkDotNet only generates this column if the distribution seems strange.

Now, let’s move on to a more real-world application.

Revisiting gRPC and REST

Now that we are familiar with BenchmarkDotNet, let’s go back revisit our gRPC and MongoDb article, we explained how to set up a gRPC application with a MongoDB backend. Suppose, now we want to figure out how much faster gRPC is than REST. Migrating from REST to Grpc could be a lot of work, so we want to make sure it’s worth it.

In this article, we are going to modify the gRPC project and add a REST implementation of the same application. We are not going to go into detail on how to do that since our focus is on learning how to use BenchmarkDotNet. Details on how to set up a REST application can be found in our Getting Started with ASP.NET Core and MongoDb article and details on how to set up a gRPC application can be found in our Adding gRPC to ASP.NET Core article.

To summarize, we are going to have four projects. The SharedLibray project is going to hold our data models and data access files. The GrpcBackend and RestBackend projects are going to hold our API applications. Lastly, the GrpcVsRest project is going to hold our client and benchmark code. More details are available in the source code.

Benchmarking gRPC and REST

We are going to compare gRPC and REST by benchmarking the controller classes. Let’s take a look at StudentApiController.cs within the RestBackend project:

[Route("api/[controller]")]
[ApiController]
public class StudentApiController : ControllerBase
{
    private readonly StudentDataAccess _students;
    private readonly IMapper _mapper;
    public StudentApiController(StudentDataAccess students, IMapper mapper)
    {
        _students = students;
        _mapper = mapper;
    }
    [HttpGet("{id}")]
    public async Task<IActionResult> GetById(string id)
    {
        try
        {
            var student = await _students.GetByIdWithCoursesAsync(id);
            return Ok(student);
        }
        catch (Exception ex)
        {
            return StatusCode(500, $"{ex.Message}");
        }
    }
}

We want to know how fast it takes for an HTTP client to retrieve student data from the MongoDB database using REST. Therefore, we want to benchmark the HTTP request that executes the GetById action method within the StudentApiController. For this example, the action method responds with a deserialized Student object in the response body if the student id is valid.

Now, let’s look at the StudentGrpcController within the GrpcBackend project:

public class StudentGrpcController : StudentService.StudentServiceBase
{
    private readonly StudentDataAccess _students;
    private readonly IMapper _mapper;
    public StudentGrpcController(StudentDataAccess students, IMapper mapper)
    {
        _students = students;
        _mapper = mapper;
    }

    public override async Task<GetStudentResponseDto> GetStudent(GetStudentRequestDto request, ServerCallContext context)
    {
        try
        {
            if (request.Id != null)
            {
                var student = await _students.GetByIdWithCoursesAsync(request.Id);
                return new GetStudentResponseDto
                {
                    Student = _mapper.Map<StudentDto>(student)
                };
            }
            else
            {
                return new GetStudentResponseDto
                {
                    Error = "ID is null or empty"
                };
            }
        }
        catch (Exception ex)
        {
            return new GetStudentResponseDto
            {
                Error = $"{ex.Message}"
            };
        }
    }
}

We also want to know how fast does it take for a gRPC client to retrieve student data from the MongoDB database using gRPC. Therefore, we want to benchmark the gRPC routine that executes the GetStudent method within the StudentGrpcController.

We are going to begin our GrpcVsRest project by creating a Clients folder. Our benchmark code is going to use the client code to test how fast each protocol is. Let’s begin by defining RestClient.cs:

public class RestClient
{
    private static readonly HttpClient client = new HttpClient();
    public async Task<Student> GetSmallPayloadAsync()
    {
        client.DefaultRequestHeaders.Accept.Clear();
        client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
        var id = "5eeffdd1a28671a6e62dbda2";
        return await client.GetFromJsonAsync<Student>($"http://localhost:6000/api/studentapi/{id}");
    }
}

The RestClient class has the GetSmallPayloadAsync method which creates an HTTP GET request to the URL endpoint for our API. It’s important to note that the request is made to port 6000. For benchmarking we are going to need both applications running on the same machine at the same time, therefore the ports must be different for each application. The port for the REST application is 6000 and the port for the gRPC application is 5000.

Now, let’s create the GrpcClient.cs within the Clients folder:

public class GrpcClient
{
    private readonly GrpcChannel channel;
    private readonly StudentService.StudentServiceClient client;

    public GrpcClient()
    {
        AppContext.SetSwitch(
            "System.Net.Http.SocketsHttpHandler.Http2UnencryptedSupport", true);
        channel = GrpcChannel.ForAddress("http://localhost:5000");
        client = new StudentService.StudentServiceClient(channel);
    }

    public async Task<StudentDto> GetSmallPayloadAsync()
    {
        return (await client.GetStudentAsync(
                new GetStudentRequestDto (){Id = "5eeffdd1a28671a6e62dbda2" }
            )).Student;
    }
}

Like the RestClient class, the GrpcClient also has a GetSmallPayloadAsync method. This method simply invokes the GetStudentAsync method within the StudentServiceClient class generated by the Protobuf file.

Now that our clients are ready, adding benchmarks is going to be easy.

Let’s create a BenchmarkHarness.cs class:

[HtmlExporter]
public class BenchmarkHarness
{
    [Params(100, 200)]
    public int IterationCount;

    private readonly RestClient _restClient = new RestClient();
    private readonly GrpcClient _grpcClient = new GrpcClient();

    [Benchmark]
    public async Task RestGetSmallPayloadAsync()
    {
        for(int i = 0; i < IterationCount; i++)
        {
            await _restClient.GetSmallPayloadAsync();
        }
    }
    [Benchmark]
    public async Task GrpcGetSmallPayloadAsync()
    {
        for(int i = 0; i < IterationCount; i++)
        {
            await _grpcClient.GetSmallPayloadAsync();
        }
    }
}

In this class, we are first defining a class member of type int with a Params attribute. The Params attribute lets us specify a set of values for the benchmark runner to execute with. So in our code, our benchmarks will execute when IterationCount is 100 and also when IterationCount is 200. Therefore, we are expecting a total of four results.

The RestGetSmallPayloadAsync method calls the RestClient’s GetSmallPayloadAsync method 100 times and then 200 times. The same happens with GrpcClient.

Lastly, we need to create the Program.cs class:

class Program
{
    static void Main(string[] args)
    {
        BenchmarkRunner.Run<BenchmarkHarness>();
        Console.ReadKey();
    }
}

That’s it!

We are ready to run our projects.

Let’s run the RestBackend, GrpcBackend, and GrpcVsRest projects in release configuration:

dotnet run -p RestBackend/RestBackend.csproj -c Release

dotnet run -p GrpcBackend/GrpcBackend.csproj -c Release

dotnet run -p GrpcVsRest/GrpcVsRest.csproj -c Release

After that, we can check out the markdown results:

Seems like gRPC might be the way to go!

Conclusion

We have successfully learned how to use benchmarking tools in our project. We saw how to use BenchmarkDotNet and how to generate very detailed results. In this article, we also learned how to write a benchmark project to compare the different implementations of the same application. Although we used a very specific example (comparing gRPC and REST), we can apply the same methodology to any part of our C# project that we are considering optimizing.

Until the next article.

All the best.