Let’s benchmark .NET using BenchmarkDotNet!

In this blog, we will deep dive into BenchmarkDotNet package and see how it can help us improve our dotnet application.

What is Benchmarking?

Benchmarking is the act of comparing similar products, services, or processes in the same industry. When benchmarking it is very essential to measure the quality, time, and cost. Benchmarking will let you know if you are working on the latest and best practices across other parties in the same industry and it will help you identify your strengths and weaknesses.

And what is BenchmarkDotNet

BenchmarkDotNet is a lightweight, open-source powerful .NET library that is used for benchmarking. BenchmarkDotNet will help you transform methods into benchmarks, track their performance, and examine measurement experiments. It is very similar to unit tests, by creating functions and running them to generate a user-friendly result with all the important facts about the experiment.

Stay In Touch

Subscribe to our mailing list to stay updated on topics and videos related to .NET, Azure, and DevOps!

By submitting your information, you’re giving us permission to email you. You may unsubscribe at any time.

It’s Demo Time

It is easy to start benchmarking in C#, first start by creating a new console application

dotnet new console -o BenchmarkDotNetDemo

Navigate to the newly created folder BenchmarkDotNetDemo and then add the BenchmarkDotNet library using the following command

dotnet add package BenchmarkDotNet

Let’s say we have a class that filters and gets data using different methods, we are going to benchmark multiple get methods, using LinQ and for loops. Create the following class and add the following code to it:

public class DataService
{
    List<string> _data = new()
    {
        "Carlos", "Adelaide", "Dexter", "Connie", "Annabella", "Sophia", "Alissa", "Kimberly", "Isabella", "Adam", "Valeria",
        "Tiana", "Michelle", "Justin", "Cadie", "Owen", "Mary", "Edwin", "Audrey", "Eddy", "Sarah", "Patrick", "Daniel",
        "Emily", "Sam", "Clark", "James", "Alen", "Michael", "Lenny", "Penelope", "Victor", "Ryan", "Lilianna", "Aida",
        "Lana", "Andrew", "Maximilian", "Savana", "Edward", "Sofia", "Harold", "Amelia", "Rosie", "William"
    };


    public string GetDataByFirstOrDefault(string key)
    {
        return _data.FirstOrDefault(x => key == x);
    }
    
    public string GetDataByFirst(string key)
    {
        return _data.First(x => key == x);
    }
    
    public string GetDataBySingle(string key)
    {
        return _data.Single(x => key == x);
    }

    public string GetDataByForEach(string key)
    {
        foreach (var item in _data)
        {
            if (item == key)
            {
                return item;
            }
        }
        return default;
    }

    public string GetDataByForLoop(string key)
    {
        for (int i = 0; i < _data.Count; i++)
        {
            if (_data[i] == key)
            {
                return _data[i];
            }
        }
        return default;
    }
}

As you can see, we have the DataService‘s class that contains a List of strings with random names, and we have multiple methods to retrieve the data using different functionalities. We are going to benchmark different methods to see which one is the most efficient and which one we should exclude (depending on the use case of course)

Now let’s create a new class, DataBenchmark and here we will create multiple benchmark methods, you will see here it is very similar to unit tests, keep in mind we need to add the Benchmark attribute on each of the methods.

using BenchmarkDotNet.Attributes;

[MemoryDiagnoser]
[Orderer(BenchmarkDotNet.Order.SummaryOrderPolicy.FastestToSlowest)]
[RankColumn]
public class DataBenchmark
{
    DataService _service = new DataService();

    [Benchmark(Baseline = true)]
    public void GetDataByFirstOrDefault()
    {
        _service.GetDataByFirstOrDefault("Isabella");
    }

    [Benchmark]
    public void GetDataByForEach()
    {
        _service.GetDataByForEach("Isabella");
    } 
    [Benchmark]
    public void GetDataByFirst()
    {
        _service.GetDataByFirst("Isabella");
    } 
    [Benchmark]
    public void GetDataBySingle()
    {
        _service.GetDataBySingle("Isabella");
    }

    [Benchmark]
    public void GetDataByForLoop()
    {
        _service.GetDataByForEach("Isabella");
    }
}

Add the following in Program.cs file

using BenchmarkDotNet.Running;

BenchmarkRunner.Run<DataBenchmark>();

Now, you need to run the console app in Release mode, run the app using the following command:

dotnet run -c Release

After benchmarking is completed, you will see the following result in the terminal

|                  Method |      Mean |     Error |    StdDev |    Median | Ratio | RatioSD | Rank |   Gen0 | Allocated | Alloc Ratio |
|------------------------ |----------:|----------:|----------:|----------:|------:|--------:|-----:|-------:|----------:|------------:|
|        GetDataByForLoop |  36.56 ns |  0.662 ns |  1.211 ns |  36.24 ns |  0.19 |    0.01 |    1 |      - |         - |        0.00 |
|        GetDataByForEach |  37.09 ns |  0.772 ns |  1.450 ns |  36.80 ns |  0.19 |    0.01 |    1 |      - |         - |        0.00 |
| GetDataByFirstOrDefault | 195.22 ns |  4.120 ns | 11.689 ns | 193.47 ns |  1.00 |    0.00 |    2 | 0.0305 |     128 B |        1.00 |
|          GetDataByFirst | 215.05 ns | 11.025 ns | 31.632 ns | 199.19 ns |  1.10 |    0.19 |    3 | 0.0305 |     128 B |        1.00 |
|         GetDataBySingle | 857.50 ns | 27.473 ns | 79.704 ns | 827.47 ns |  4.42 |    0.50 |    4 | 0.0305 |     128 B |        1.00 |

The same result can be found in the folder BenchmarkDotNet.Artifacts in the release bin folder, with CSV, HTML, and MD files.

Now let us explain the result we got

Results and Summary

As we can see, GetDataBySingle ranked the worst by 857.50 ns and GetDataByForLoop ranked first by 36.56 ns. It is very logical for Single to be the last one on the list since SignleOrDefault will need to go through all the items in the list to check if the item is found and is unique. BenchmarkDotNet helped us here to cross-check different Get methods and allowed us to optimize our code base in a way that meets the performance standard.

SingleOrDefault in EF SqlServer

In EntityFramework – SQL Server, the call of SingleOrDefault is being translated into SELECT TOP(2), so if the returned result is 0 => the default value. If the result is 1, return that value and if it is 2, throw an exception.

If you are still here (thanks), please let me know what kind of benchmarking you are going to use by leaving a comment.

Thank you for reading, till next time 👋