A testing framework provides us with an approach to automating the validation of the functionality and performance of our code. In this article, we will look at the three major testing frameworks (NUnit, xUnit, and MSTest) that are available for a .NET project.Ā
However, before we dive in, let’s understand why we need a testing framework.
Let’s begin.
The Role of a Testing Framework
As projects become larger and more complex, manually validating every code change becomes impractical and error-prone. A testing framework addresses this challenge by providing us with pre-built functionalities to create and execute tests.
They provide a standardized way to express and manage test scenarios. Thus, it becomes easier to express expected outcomes and validate results.
In essence, testing frameworks are tools that enhance the reliability of an application as well as streamline the development process. Hence, it leads to more robust and maintainable applications.
If you’d like to learn about testing exception-throwing code specifically, using these testing frameworks, check out our article covering that topic.
NUnit Testing Framework
Originally based on JUnit, NUnit is an open-source .NET unit testing framework. We can use NUnit to write tests for C#, VB.NET, and F# applications.
Let’s look at its syntax, test execution capabilities, parameterized testing, assertions, and integration with mocking frameworks.
Syntax of NUnit Testing Framework
NUnit uses attributes like [TestFixture] and [Test] to mark test classes and methods. Let’s create a sample NUnit test:
[TestFixture]
public class NUnitCalculatorTests
{
[Test]
public void GivenTwoIntegers_WhenAdded_ThenShouldReturnSum()
{
var result = Calculator.Add(3, 7);
Assert.That(result, Is.EqualTo(10));
}
[Test]
public void GivenTwoIntegers_WhenAdded_ThenShouldReturnCorrectNegativeSum()
{
var result = Calculator.Add(-3, 7);
Assert.That(result, Is.EqualTo(4));
}
}
Here, the [TestFixture] attribute marks a class that contains one or more test methods. The attribute is technically optional for simple test cases since NUnit 2.5. However, it is recommended to use it for clarity.
If a class contains methods marked with the [Test] attribute, NUnit recognizes those methods as tests.
If we have to ignore any test case from a test class in NUnit, then we can use [Ignore("reason")] to ignore that specific test method. This attribute also works for the whole test class.
NUnit Assertions
NUnit allows two models of writing test assertions: the classic model and the constraint model.
The classic model refers to the traditional way of using methods like Assert.AreEqual(), Assert.IsTrue(), etc.Ā
The constraint model allows us to use the Assert.That syntax along with constraints provided by NUnit, such as Is.EqualTo(), Is.GreaterThan(), and Is.Not.EqualTo(). It makes the assertions more readable in some cases.
The assertion in the example, Assert.That(result, Is.EqualTo(10)) could be written like Assert.AreEqual(10, result) in the classic model.
Both assertion models are valid and we can choose either based on our style preferences.
NUnit Fixture Setup and Teardown
We have the [SetUp] and [TearDown] attributes for test fixture setup and teardown in NUnit.
The method marked by the [SetUp] attribute executes before each test in a test fixture. Similarly, the method marked by the [TearDown] attribute executes after each test in the test fixture.
It’s important to note that while the convention is to name the setup and teardown methods as SetUp() and TearDown()respectively, NUnit allows us to use any method name as long as they are marked with the [SetUp] and [TearDown] attributes:
[TestFixture]
public class NUnitUserServiceTests
{
private const string USER_NAME = "Admin";
private const string PASSWORD = "pass123";
private const string OTHER_PASSWORD = "Pass124";
private UserService? userService;
[SetUp]
public void Setup()
{
userService = new UserService();
userService.AddUser(new User(USER_NAME, PASSWORD));
}
[TearDown]
public void TearDown()
{
userService?.RemoveUser(USER_NAME);
}
[Test]
public void GivenValidCredentials_WhenAuthenticate_ThenShouldReturnTrue()
{
var isAuthenticated = userService?.Authenticate(USER_NAME, PASSWORD);
Assert.That(isAuthenticated, Is.True);
}
[Test]
public void GivenInvalidCredentials_WhenAuthenticate_ThenShouldReturnFalse()
{
var isAuthenticated = userService?.Authenticate(USER_NAME, OTHER_PASSWORD);
Assert.That(isAuthenticated, Is.False);
}
}
Here, the Setup() method will initialize a UserService instance and add a test user with the username “Admin” and password “pass123” before each test.
The TearDown() method will remove the admin userĀ after each test.
If we want to execute some logic only once for the entire class, NUnit provides us with theĀ [OneTimeSetUp] attribute.
It is important to note that in NUnit, we can have multiple methods marked with theĀ [OneTimeSetUp]and [OneTimeTearDown] attributes. Each of these methods will execute before and after all the test methods respectively.
NUnit Parametrized Tests
NUnit supports parameterized tests using the [TestCase] attribute. This allows us to execute a single test method with different sets of inputs:
[TestCase(8, 9, 17)]
[TestCase(0, 0, 0)]
[TestCase(-2, 56, 54)]
public void GivenTwoIntegersInTestCase_WhenAdded_ThenShouldReturnSum(int a, int b, int expected)
{
var result = Calculator.Add(a, b);
Assert.That(result, Is.EqualTo(expected));
}
Here, each [TestCase] attribute represents a test case with specific values for the operands and the expected result. It provides multiple sets of input values, and the test method is executed once for each set of parameters.Ā
NUnit Test Filters and Traits
There may be times when we don’t want to execute the complete test suite and instead want to focus on a specific subset. Test filters and traits help us in this case by enabling us to selectively run our tests based on certain criteria.Ā
Test filters in NUnit allow us to selectively run tests based on category, or name of the test method. We use [Category] attribute to categorize our tests into specific categories.
Traits are specified using the [Property] attribute. They allow us to add descriptions to our tests by applying metadata to them.
Let’s update our test methods to see them in action:
[Test]
[Category("Sum")]
[Property("NegativeNumbers", "false")]
public void GivenTwoIntegers_WhenAdded_ThenShouldReturnSum()
{
var result = Calculator.Add(3, 7);
Assert.That(result, Is.EqualTo(10));
}
[Test]
[Category("Sum")]
[Property("NegativeNumbers", "true")]
public void GivenTwoIntegers_WhenAdded_ThenShouldReturnCorrectNegativeSum()
{
var result = Calculator.Add(-3, 7);
Assert.That(result, Is.EqualTo(4));
}
Here, we have categorized our tests using filters and traits. Now, we can use the dotnet test command on the command line to leverage these test filters and traits selectively running our tests.
Simply running the dotnet test command without any filter results in the execution of all the tests:
Starting test execution, please wait... A total of 1 test files matched the specified pattern. Passed! - Failed: 0, Passed: 7, Skipped: 0, Total: 7, Duration: 11 ms - NUnitTests.dll (net8.0)
To apply filters, we need the --filter option.Ā Using --filter, we can filter tests by category, enabling us to run tests grouped under a specific category:
dotnet test --filter Category=Sum
This will pick up only tests marked with [Category("Sum")]:
Starting test execution, please wait... A total of 1 test files matched the specified pattern. Passed! - Failed: 0, Passed: 2, Skipped: 0, Total: 2, Duration: 6 ms - NUnitTests.dll (net8.0)
Similarly, we can also filter tests based on traits:
dotnet test --filter NegativeNumbers=true
In addition, we can combine multiple filters to narrow down our selection:
dotnet test --filter "NegativeNumbers!=true&Category=Sum"
These filtering options provide granular control over test execution. They ensure that we can run the subset of tests that are most relevant at any point.
xUnit Testing Framework
xUnit is an open-source testing framework known for its clean and extensible architecture. This testing framework follows a convention-over-configuration approach and adheres to the test-driven development (TDD) guidelines.
Syntax of xUnit Testing Framework
xUnit uses the [Fact] attribute to mark a method as a test method. Let’s create a sample xUnit test:
public class XUnitCalculatorTests
{
[Fact]
public void GivenTwoIntegers_WhenSubstracted_ThenShouldReturnDifference()
{
var result = Calculator.Substract(7, 3);
Assert.Equal(4, result);
}
[Fact]
public void GivenTwoIntegers_WhenSubstracted_ThenShouldReturnCorrectNegativeDifference()
{
var result = Calculator.Substract(-7, 3);
Assert.Equal(-10, result);
}
}
In xUnit, we don’t need any specific attribute at the class level to denote it as a test class. It considers every public class containing test methods as a test class by default.
We can use [Fact(Skip = "reason")] to ignore any specific test method.
xUnit Assertions
xUnit offers a set of assertions through its Assert class, with a syntax similar to NUnit.
We have assertion methods such as Assert.Equal() ,Ā Assert.True(), and Assert.False() available in xUnit to directly check whether a result matches the expected outcome.
xUnit Test Setup and Teardown
We don’t have any specific methods for test setup and teardown activities in xUnit.Ā Instead, we create a constructor of our test class and add the setup code to it.Ā
For teardown, we implement the IDisposable interface in our test class and put our teardown code in the the Dispose() method:
public class XUnitUserServiceTest : IDisposable
{
private const string USER_NAME = "Admin";
private const string PASSWORD = "pass123";
private const string OTHER_PASSWORD = "Pass124";
private UserService? userService;
public XUnitUserServiceTest()
{
userService = new UserService();
userService.AddUser(new User(USER_NAME, PASSWORD));
}
public void Dispose()
{
userService?.RemoveUser(USER_NAME);
}
[Fact]
public void GivenValidCredentials_WhenAuthenticate_ThenShouldReturnTrue()
{
var isAuthenticated = userService?.Authenticate(USER_NAME, PASSWORD);
Assert.True(isAuthenticated);
}
[Fact]
public void GivenInvalidCredentials_WhenAuthenticate_ThenShouldReturnFalse()
{
var isAuthenticated = userService?.Authenticate(USER_NAME, OTHER_PASSWORD);
Assert.False(isAuthenticated);
}
}
Here, the XUnitUserServiceTestĀ method initializes a UserService instance and adds a test user with the username “Admin” and password “pass123” before each test.
The Dispose() method removes the admin user and sets the UserService instance to null after each test.
If we want to execute some logic only once for the entire class, XUnit provides us with the ICollectionFixture<T> interface. This interface is a way to share a single instance of a class across all tests in a test class. We can use this to implement global setup and teardown logic.
xUnitĀ Parametrized Tests
xUnit enables parametrized testing through the use of the [Theory] attribute. We can use other attributes like [InlineData], [MemberData], [ClassData] along with [Theory] to create parametrized tests.
It’s important to note that parametrized test methods need to be marked with [Theory] instead of [Fact]. Using [Fact] on parametrized test methods results in a compiler error saying “Fact methods cannot have parameters. Remove the parameters from the method or convert it into a Theory“.
Using the [InlineData] attribute allows us to execute a single test method with different sets of inputs:
[Theory]
[InlineData(3, 2, 1)]
[InlineData(0, 0, 0)]
[InlineData(-2, 3, -5)]
public void GivenTwoIntegers_WhenSubstracted_ThenReturnsDifferenceUsingInlineData(int a, int b, int expected)
{
var result = Calculator.Substract(a, b);
Assert.Equal(expected, result);
}
Here, each [InlineData] attribute represents a test case with specific values for the operands and the expected result. It provides multiple sets of input values, and the test method is executed once for each set of parameters.Ā
The [MemberData] attribute allows us to specify a class member that returns the data for the parameterized tests:
public static IEnumerable<object[]> TestData => new List<object[]> {
new object[] { 3, 2, 1 },
new object[] { 0, 0, 0 },
new object[] { -2, 3, -5 },
};
[Theory]
[MemberData(nameof(TestData))]
public void GivenTwoIntegers_WhenSubstracted_ThenReturnsDifferenceUsingMemberData(int a, int b, int expected)
{
var result = Calculator.Substract(a, b);
Assert.Equal(expected, result);
}
Here, the TestData property is an IEnumerable<object[]> with the testing data. The [MemberData(nameof(TestData))] attribute indicates that this property should be used as the source of data for the parameterized test.
The [ClassData] attribute is similar to the [MemberData] attribute but allows us to specify a separate class instead of a property or method within the same class. This test data class should implement IEnumerable<object[]>.
Let’s create a XUnitTestDataGenerator class:
public class XUnitTestDataGenerator : IEnumerable<object[]>
{
public IEnumerator<object[]> GetEnumerator()
{
yield return new object[] { 3, 2, 1 };
yield return new object[] { 0, 0, 0 };
yield return new object[] { -2, 3, -5 };
}
IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();
}
Here, the class provides the test data through the GetEnumerator() method. Then, we use the [ClassData(typeof(XUnitTestDataGenerator))] attribute to specify this class as the source of data for the parameterized test:
[Theory]
[ClassData(typeof(XUnitTestDataGenerator))]
public void GivenTwoIntegers_WhenSubstracted_ThenReturnsDifferenceUsingClassData(int a, int b, int expected)
{
var result = Calculator.Substract(a, b);
Assert.Equal(expected, result);
}
Using [ClassData] is particularly useful to encapsulate the test data generation logic in a separate class, which, for example, enables reusability.
xUnitĀ Test Filters
xUnit uses [Trait] attribute for categorization:
[Fact]
[Trait("Type", "Subtraction")]
[Trait("Category", "Positive")]
public void GivenTwoIntegers_WhenSubstracted_ThenShouldReturnDifference()
{
var result = Calculator.Substract(7, 3);
Assert.Equal(4, result);
}
[Fact]
[Trait("Type", "Subtraction")]
[Trait("Category", "Negative")]
public void GivenTwoIntegers_WhenSubstracted_ThenShouldReturnCorrectNegativeDifference()
{
var result = Calculator.Substract(-7, 3);
Assert.Equal(-10, result);
}
[Traits] are metadata attributes that allow us to categorize our tests.
Now, we can use the dotnet test command on the command line to selectively run our tests based on these traits. Simply running the dotnet test command without any filter results in the execution of all the tests:
Starting test execution, please wait... A total of 1 test files matched the specified pattern. Passed! - Failed: 0, Passed: 13, Skipped: 0, Total: 13, Duration: 24 ms - XUnitTests.dll (net8.0)
To filter the tests, we need the --filter option.Ā Using --filter, we can filter tests by traits, enabling us to run tests grouped under a specific trait:
dotnet test --filter Type=Subtraction
This will pick up only tests marked with [Trait("Type", "Subtraction")]:
Starting test execution, please wait... A total of 1 test files matched the specified pattern. Passed! - Failed: 0, Passed: 2, Skipped: 0, Total: 2, Duration: 2 ms - XUnitTests.dll (net8.0)
Additionally, we can also combine multiple traits to narrow down our selection:
dotnet test --filter "Type=Substraction&Category=Negative"
These filtering options provide granular control over test execution. They ensure that we can run the subset of tests that are most relevant at any point.
MSTest Testing Framework
MSTest, also known as the Visual Studio Unit Testing Framework, is Microsoft’s official testing framework for .NET applications.
Syntax of MSTest Testing Framework
MSTest uses the [TestClass] and [TestMethod] attributes to denote a class as a test class and, a method as a test method, respectively. Let’s create a sample MSTest test:
[TestClass]
public class MSTestCalculatorTests
{
[TestMethod]
public void GivenTwoIntegers_WhenMultiplied_ThenShouldReturnPositiveProduct()
{
var result = Calculator.Multiply(3, 7);
Assert.AreEqual(21, result);
}
[TestMethod]
public void GivenTwoIntegers_WhenMultiplied_ThenShouldReturnCorrectNegativeProduct()
{
var result = Calculator.Multiply(-3, 7);
Assert.AreEqual(-21, result);
}
}
Here, we use the [TestClass] attribute to mark a class as a test class and the [TestMethod] attribute to mark a method as a test method.
MSTest Assertions
Similar to NUnit and xUnit, we can execute assertions in MSTest through its Assert class.
Whenever the result does not match with expected outcome, the Assert.AreEqual() methodĀ throws AssertFailedException along with a message like “Assert.AreEqual failed. Expected:<-22>. Actual:<-21>.“
Similarly, we have Assert.AreNotEqual(), Assert.AreSame(), Assert.AreNotSame(), Assert.IsInstanceOfType(), and other methods for performing different assertions.
MSTest Test Setup and Teardown
MSTest uses [TestInitialize] and [TestCleanup] for test setup and teardown, respectively:
[TestClass]
public class MSTestUserServiceTests
{
private const string USER_NAME = "Admin";
private const string PASSWORD = "pass123";
private const string OTHER_PASSWORD = "Pass124";
private UserService? userService;
[TestInitialize]
public void TestInitialize()
{
userService = new UserService();
userService.AddUser(new User(USER_NAME, PASSWORD));
}
[TestCleanup]
public void TestCleanup()
{
userService?.RemoveUser(USER_NAME);
}
[TestMethod]
public void GivenValidCredentials_WhenAuthenticate_ThenShouldReturnTrue()
{
var isAuthenticated = userService?.Authenticate(USER_NAME, PASSWORD);
Assert.IsTrue(isAuthenticated);
}
[TestMethod]
public void GivenInvalidCredentials_WhenAuthenticate_ThenShouldReturnFalse()
{
var isAuthenticated = userService?.Authenticate(USER_NAME, OTHER_PASSWORD);
Assert.IsFalse(isAuthenticated);
}
}
Here, the TestInitialize() method initializes a UserService instance and adds a test user with the username “Admin” and password “pass123” before each test.
The TestCleanup() method removes the admin user and sets the UserService instance to null after each test.Ā
If we want to execute some logic only once for the entire class, MSTest provides us with the ClassInitialize() method. This method marked with the [ClassInitialize] attribute is useful for one-time setup that is common to all tests in the class. However, this method must be a static method.
It is important to note that in MSTest all the methods used to initialize and cleanup the tests follow strict naming conventions. MSTest can only recognize these methods as part of the test setup if we follow the convention. For instance, we need to mark the TestInitialize() method with a [TestInitialize] attribute. In addition, the method name must always start with the attribute name i.e. “TestInitialize” in this case. This is true for the methods TestCleanup(), ClassInitialize(), and ClassCleanup() as well.
MSTestĀ Parametrized Tests
In MSTest, we have multiple ways to perform parametrized testing.Ā
MSTest supports parameterized tests using the [DataRow] attribute. This allows us to execute a single test method with different sets of inputs:
[TestMethod]
[DataRow(0, 0, 0)]
[DataRow(5, 2, 10)]
[DataRow(-3, 7, -21)]
public void GivenTwoIntegers_WhenMultiplied_ThenReturnsProductUsingDataRow(int a, int b, int expected)
{
var result = Calculator.Multiply(a, b);
Assert.AreEqual(result, expected);
}
Here, the test method takes three parameters and the [DataRow] attribute specifies different sets of input for each test run.
We can use the [DynamicData] attribute to declare the test data as a property or a method and use it dynamically in multiple tests:
[TestMethod]
[DynamicData(nameof(GetData), DynamicDataSourceType.Method)]
public void GivenDynamicTestData_WhenMultiplied_ThenReturnsProduct(int a, int b, int expected)
{
var actual = Calculator.Multiply(a, b);
Assert.AreEqual(expected, actual);
}
public static IEnumerable<object[]> GetData()
{
yield return new object[] { 1, 2, 2 };
yield return new object[] { 12, 30, 360 };
yield return new object[] { 14, 1, 14 };
}
Here, we specify that the data for the test method needs to come from a method using DynamicDataSourceType.Method. We also provide the method’s name as a parameter to the DynamicData attribute.
In MSTest, we can specify a data source for the test method using the [DataSource] attribute:
[DataSource("Microsoft.VisualStudio.TestTools.DataSource.CSV", "TestsData.csv", "TestsData#csv", DataAccessMethod.Sequential)]
In this case, we use the Microsoft.VisualStudio.TestTools.DataSource.CSV data source provider to read data from a CSV file.Ā
MSTestĀ Test Filters
In MSTest, we use [TestCategory] attribute to categorize tests. This allows us to selectively run specific tests based on required criteria.
Let’s see them in action:
[TestMethod]
[TestCategory("Multiplication")]
[TestCategory("PositiveNumbers")]
public void GivenTwoIntegers_WhenMultiplied_ThenShouldReturnPositiveProduct()
{
var result = Calculator.Multiply(3, 7);
Assert.AreEqual(21, result);
}
[TestMethod]
[TestCategory("Multiplication")]
[TestCategory("NegativeNumbers")]
public void GivenTwoIntegers_WhenMultiplied_ThenShouldReturnCorrectNegativeProduct()
{
var result = Calculator.Multiply(-3, 7);
Assert.AreEqual(-21, result);
}
Here, we have categorized our tests using [TestCategory] attributes. Now, we can use the dotnet test command on the command line to selectively run our tests.
Simply running the dotnet test command without any filter results in the execution of all the tests:
Starting test execution, please wait... A total of 1 test files matched the specified pattern. Passed! - Failed: 0, Passed: 10, Skipped: 0, Total: 10, Duration: 16 ms - MSTests.dll (net8.0)
Similar to the other frameworks, to apply filters, we need the --filter option.Ā
Using --filter, we can filter tests by category, enabling us to run tests grouped under a specific category:
dotnet test --filter TestCategory=Multiplication
This will pick up only tests marked with [TestCategory("Multiplication")]:
Starting test execution, please wait... A total of 1 test files matched the specified pattern. Passed! - Failed: 0, Passed: 2, Skipped: 0, Total: 2, Duration: 12 ms - MSTests.dll (net8.0)
Similarly, we can also combine multiple categories to narrow down our filters further:
dotnet test --filter "TestCategory=Multiplication&TestCategory=PositiveNumbers"
When using multiple filters, we need to enclose the filter expression within ” “.Ā These filtering options ensure that we run only the relevant subset of tests.
Choosing the Correct Testing Framework
Choosing the appropriate testing framework is important as it influences the way tests are written and executed. Among the three .NET testing frameworks we discussed, each has its own set of advantages and challenges.
NUnit offers extensive documentation, a strong community, and a lot of plugins to choose from. It also provides great interoperability with non-Microsoft platforms. In the past, there were problems in NUnit with its Visual Studio integration, however, that has been solved and we can easily create an NUnit test project in Visual Studio. However, it’s not without its challenges. While NUnit is quite extensible, it doesn’t match the extensibility of other frameworks like xUnit.
Comparatively, xUnit provides a clean syntax helping us to write readable tests. It has extensibility designed to its core and thus makes it easy to customize test behavior. However, it’s a growing framework and hence doesn’t support as many plugins and extensions.
MSTest comes integrated with Visual Studio and hence is the most convenient for developers using it as their IDE. Historically, it has its limitations with cross-platform support. However, it’s catching up in that department.
To sum up, the choice between NUnit, xUnit, and MSTest depends on our team’s preferences and project requirements. Each framework has its strengths, and the best choice is the one that aligns with our team’s expertise. We must choose a framework that’s future-proof and meets the specific needs of our software development project.
Conclusion
In this article, we looked into the three most prominent testing frameworks available for .NET projects i.e. NUnit, xUnit, and MSTest. We learned how they work, their features, and what criteria to look for to choose between them.