Saga Pattern With NServiceBus in C#

Navigating the world of distributed systems and microservices in C#? Then we should appreciate the combination of the Saga Pattern and NServiceBus. This powerful pairing provides us with an efficient solution for coordinating long-running, distributed transactions in a reliable and testable manner.

Let’s picture an e-commerce platform where a customer’s order triggers a cascade of operations, from inventory checks to payment processing and shipping. If any of these operations fail, the entire process must roll back to maintain consistency. Complex? Certainly. But this is where the Saga Pattern with NServiceBus truly excels.

In this article, we’ll delve into these concepts and implementation. Let’s dive in!

The Saga Pattern

Firstly, we need to understand what precisely the Saga Pattern is and what kind of problems it intends to solve.

In a monolithic system, transactions are relatively straightforward. We start a transaction, perform several business operations, and then either commit the transaction if everything is successful or roll it back in case of failure.

This all-or-nothing approach is simple and effective because it’s all happening within a single system.

However, when we move to a distributed system, where different parts of a business transaction are handled by different microservices often with their own databases, things get much more complicated.  This is where the Saga Pattern comes into play.

When working with distributed systems and microservices, ensuring data consistency across different services can be a real challenge.

Introducing Sagas

In its simplest form, a saga is a sequence of local transactions. Each transaction updates data within a single service, and each service publishes an event to trigger the next transaction in the saga. If any transaction fails, the saga executes compensating transactions to undo the impact of the failed transaction.

The Saga Pattern is ideal for long-running, distributed transactions where each step needs to be reliable and reversible. It allows us to maintain data consistency across services without the need for distributed locks or two-phase commit protocols, which can add significant complexity and performance overhead.

Choreography-Based Saga

There are two distinct types of sagas, the first one is a choreography-based saga.

To understand these in the context of an example, let’s consider an e-commerce system that processes orders created by customers:

In this type of saga, each local transaction publishes domain events that other services listen to.

As we can see on the diagram, the Order Service publishes an Order Created event to a message bus. The Payment Service that subscribes to these events picks them up, processes the payment, and generates a Payment Completed event. Then, the Order Completion Service captures this event, performs the remaining actions necessary to complete the order, and sends an Order Completed event to the queue. Finally, the Order Service receives information about the order’s completion and notifies the customer.

It’s important to note that in general, there is no central coordination. Each service is responsible for knowing which events it needs to listen for and what actions it needs to take when it receives an event.

The benefit of this approach is that it reduces coupling between services, as they only need to communicate through events. This architectural choice enhances the scalability, resilience, and flexibility of our system.

However, it can lead to a situation where the business logic of the transaction is spread across multiple services, making the system as a whole more difficult to understand and maintain.

Orchestration-Based Saga

Another saga type we could utilize is known as an orchestration-based saga.

Let’s get back to our previous e-commerce system example and modify it by replacing the queuing system with a new Checkout Service that will orchestrate order fulfillment:

In this type, there is a central orchestrator that is responsible for managing the saga.

Just as in the previous example, all services generate and await events to carry out their actions. However, this time, there isn’t a queuing system to pull from. Instead, the new Checkout Service acting as the orchestrator for the fulfillment process. This orchestrator instructs each service involved in the transaction on what action to take, listens for their responses, and then determines the next steps based on those responses.

This approach centralizes the control and decision-making process, which makes the system easier to understand and manage.

However, it can lead to tighter coupling between services, as they need to know how to communicate with the orchestrator.

Long Story Short

The Saga Pattern is a powerful tool in our distributed systems toolbox, allowing us to manage complex business transactions in a reliable, scalable, and maintainable way.

Additionally, when we merge the Saga Pattern with the Event Sourcing Pattern, we significantly enhance traceability by constructing a comprehensive sequence of events that can be analyzed to comprehend the transaction flow in-depth.

Each of these saga types offers unique strengths, and we have the flexibility to choose the one that aligns best with our application requirements. Regardless of our decision, both options empower us to effectively manage complex business transactions in a reliable, scalable, and maintainable manner.

What Is NServiceBus?

Now, that we know what the Saga Pattern is, let’s have a look at NServiceBus – a .Net library we can use to implement it.

NServiceBus is here to help us build systems that are scalable, reliable, and flexible, with a particular focus on distributed architectures and microservices.

At its core, NServiceBus is exactly what its name implies, a “service bus”, or, a software architecture model we use to design and implement communication between mutually interacting software applications in a service-oriented architecture. It provides a set of features that make it easier to manage messages between different parts of a system, such as queuing, publish/subscribe, and advanced routing capabilities.

One of the key benefits of NServiceBus is its ability to handle the complexity of distributed systems. It provides us with built-in features for managing distributed transactions, handling failures, and maintaining message consistency, even when dealing with complex, long-running processes. This makes it an excellent tool for implementing patterns like the Saga Pattern.

NServiceBus also simplifies testing and maintenance. With this in mind, we can easily write unit tests for our message handlers.

In short, NServiceBus provides us with a powerful, feature-rich platform for building distributed systems in .NET. It simplifies many of the complexities of distributed systems development, and it works seamlessly with patterns like the Saga Pattern, allowing the creation of robust, reliable applications.

Implementing the Saga Pattern With NServiceBus in C#

With an understanding of both the Saga Pattern and the NServiceBus library in hand, let’s take a look at how we can implement them together.

Let’s build a simple e-commerce system where customers can place orders and track their order fulfillment process.

We’ll begin with installing NServiceBus in our project:

 dotnet add package NServiceBus

With that simple setup, we’re ready to get started.

Defining Saga Data and Messages

First, let’s define the saga data:

public class OrderSagaData : ContainSagaData
{
    public Guid OrderId { get; set; }
    public bool PaymentProcessed { get; set; }
    public bool ShipmentPrepared { get; set; }
}

We define the OrderSagaData class as an implementation of the ContainSagaData class. This allows us to explicitly define the data structure that will comprise the state of our saga.

In this case, our saga data will contain OrderId, PaymentProcessed and ShipmentPrepared properties.

Next, let’s create messages that will drive our saga:

public class StartOrder : ICommand
{
    public Guid OrderId { get; set; }
}

public class ProcessPayment : ICommand
{
    public Guid OrderId { get; set; }
}

public class PrepareShipment : ICommand
{
    public Guid OrderId { get; set; }
}

public class OrderCompleted : IEvent
{
    public Guid OrderId { get; set; }
}

In this case, we create StartOrder, ProcessPayment and PrepareShipment classes that implement the ICommand interface. The ICommand interface represents a request to perform a specific action or operation that should change the state of our saga or initiate some behavior in the system.

Additionally, we also define an OrderCompleted class implementing the IEvent interface. We use the IEvent interface to signal that something has happened in our system. We usually broadcast it to multiple listeners to inform them about past actions or state changes.

All of the messages we have defined are simplistic and contain only an OrderId property that will indicate which order is being referenced. However, our messages can carry any amount of additional information that we’d like to transfer between services.

Implementing Saga

Once we have our saga data and messages ready, we can create handlers that will process our messages. In order to do so let’s create a new class:

public class OrderSaga : Saga<OrderSagaData>,
    IAmStartedByMessages<StartOrder>,
    IHandleMessages<ProcessPayment>,
    IHandleMessages<PrepareShipment>
{
    protected override void ConfigureHowToFindSaga(SagaPropertyMapper<OrderSagaData> mapper)
    {
        mapper.ConfigureMapping<StartOrder>(message => message.OrderId)
            .ToSaga(sagaData => sagaData.OrderId);
        mapper.ConfigureMapping<ProcessPayment>(message => message.OrderId)
            .ToSaga(sagaData => sagaData.OrderId);
        mapper.ConfigureMapping<PrepareShipment>(message => message.OrderId)
            .ToSaga(sagaData => sagaData.OrderId);
    }
}

We start by defining a new OrderSaga class that derives from Saga<OrderSagaData> class. By inheriting from the generic Saga<T> class, we define sagas that both contain saga data of type T and can handle messages.

Next, we implement the IAmStartedByMessages<StartOrder> interface. In effect, we signal that if our service receives a message of StartOrder type and no currently running saga can be found, then NServiceBus should create a new saga.

In the same fashion, we implement the IHandleMessages<T> interface for each message type we’ll be handling.

With our class definition in place, we can proceed to add implementation details.

We begin by overriding the ConfigureHowToFindSaga method from the base Saga<T> class. Within this method, we specify how to match a message with the corresponding saga data. In our example, we achieve this by utilizing the mapper.ConfigureMapping<T> method to select a property from the message. Subsequently, we call the ToSaga method to establish the connection between the message property and the corresponding saga data property. We’ll follow this pattern for each message type we want to handle.

Implementing Message Handlers

Now, it’s time to create handler methods for all of our messages:

public async Task Handle(StartOrder message, IMessageHandlerContext context) 
{ 
    Data.OrderId = message.OrderId;

    var order = await _repository.GetOrderById(message.OrderId);
    order.Status = Models.OrderStatus.PaymentPending;

    await context.SendLocal(new ProcessPayment { OrderId = message.OrderId }); 
} 
        
public async Task Handle(ProcessPayment message, IMessageHandlerContext context) 
{ 
    Data.PaymentProcessed = true;

    var order = await _repository.GetOrderById(message.OrderId);
    order.Status = Models.OrderStatus.Processing;

    await context.SendLocal(new PrepareShipment { OrderId = message.OrderId }); 
} 
        
public async Task Handle(PrepareShipment message, IMessageHandlerContext context) 
{ 
    Data.ShipmentPrepared = true;

    var order = await _repository.GetOrderById(message.OrderId);
    order.Status = Models.OrderStatus.OrderCompleted;

    await context.Publish(new OrderCompleted { OrderId = message.OrderId }); 
    MarkAsComplete();
} 

In general, we create a Handle(T message, IMessageHandleContext context) method for each message type.

Those methods will handle incoming StartOrder, ProcessPayment and PrepareShipment messages, so we will need to put proper business logic inside of them. For now, let’s keep it simple and on each saga step retrieve the Order instance, update the status, and send the next message.

We can use a context object to either send out new messages using SendLocal or Send methods, or publish events using a Publish method.

Lastly, we must always complete the saga. We can do this by calling the MarkAsComplete() method, which will signal to NServiceBus that all required actions are complete and it can safely delete the saga data. This action is irreversible, so we must remember to call it only when we are completely sure that we will no longer need our saga data.

Configuring the NServiceBus Endpoint

At this point, all we need to do is to configure our NServiceBus endpoint in our Program.cs file:

builder.Host.UseNServiceBus(context =>
{
    var endpointConfiguration = new EndpointConfiguration("OrderEndpoint");
    var transport = endpointConfiguration.UseTransport<LearningTransport>();
    var persistence = endpointConfiguration.UsePersistence<LearningPersistence>();
    var routing = transport.Routing();
    routing.RouteToEndpoint(typeof(StartOrder), "OrderEndpoint");

    return endpointConfiguration;
});

In this example, we create an instance of an EndpointConfiguration class that will define how our endpoint will work. As the parameter, we pass the endpoint name that serves as its logical identity and forms a naming convention.

Next, we use the UseTransport<LearningTransport>() method to define the message transport method.

After that, we select a way in which we want to persist our saga data by calling the UsePersistence<LearningPersistence>() method.

It’s important to remember that we should never use LearningTransport and LearningPersistence in our production loads. These components are intended solely for educational purposes, as they simulate queuing and persistence infrastructure, storing data in the local file system.

Once we complete all those steps, we can define routing policies. To do so, we simply retrieve a RoutingSettings<T> instance using the transport.Routing() method and then define routing for our messages. Those routes can either lead to the same service or to external services. Thanks to that, NServiceBus will be able to autonomously resolve based on the message type of the exact endpoint it needs to be sent to.

At this point, our service is up and ready to process messages.

Testing Our Saga Implementation

Finally, it’s time for us to test our saga implementation.

In essence, all we have to do is drop a StartOrder message into our service message bus and let it do all the work. We can see this in practice in our repository. However, since we use the LearningTransport mode which emulates the message bus, we need a workaround to overcome this limitation.

Fortunately, we can utilize the IMessageSession interface provided by the NServiceBus NuGet package. In short, we use this interface to send messages and publish events outside of our saga handlers.

To demonstrate this, let’s imagine we have an API where customers can place their orders. To integrate with our saga handler, we inject an instance of IMessageSession interface into our Order controller and utilize it within our action:

[HttpPost(Name = "PostOrder")]
public async Task<ActionResult> Post([FromBody] OrderRequest request)
{
    var order = MapToOrder(request);
    await _repository.AddOrder(order);

    var startOrderMessage = new StartOrder
    {
        OrderId = order.OrderId,
    };

    await _messageSession.Send(startOrderMessage);

    return CreatedAtRoute("GetOrder", new { order.OrderId });
}

First, we map the OrderRequest instance to the actual Order instance and store it in the repository.

Next, we create a StartOrder message and set the OrderId property to the corresponding OrderId from the order.

Then, we utilize the _messageSession object, which is an implementation of the IMessageSession interface, to send the message to our message bus (we can find a full implementation in our source code). The saga handler picks up this message and executes the defined saga logic.

Finally, we return the OrderId of the newly created order, allowing the customer to track their order’s progress.

Common Pitfalls and Solutions

Now that we know how to implement the Saga Pattern in our project, let’s look at some common pitfalls we might encounter.

Catching Exceptions in Handlers

First, we will talk about catching exceptions in handlers.

Developers might instinctively attempt to catch exceptions as close as possible to the place where they are thrown. However, in the case of our Saga Pattern implementation using NServiceBus, this might lead to missing messages or even data inconsistency.

To further explain that, we need to understand that compensation actions in the case of NServiceBus implementation rely upon built-in error handling and retry mechanisms. Without an exception, NServiceBus will assume that message processing was successful and it will discard the message. By allowing exceptions to propagate in the handlers, NServiceBus can automatically retry the processing based on the configured retry policy. 

Exceptional Cases

However, it’s important to note that there might be exceptional cases where we may need to catch exceptions within the message handlers; these might be driven by special business requirements, for example.

For instance, such a case might be when a payment processing operation fails due to insufficient funds, intercepting the exception allows us to trigger a custom business workflow, such as user notification and alternative payment suggestions.

Nevertheless, it is crucial to carefully consider the potential ramifications of mishandling exceptions. If we catch an exception without properly logging or communicating it to the system, the system may incorrectly assume that a transaction has been successfully completed when, in reality, it has failed. This can introduce data inconsistencies, such as inaccurate account balances, and undermine the overall reliability of our system.

Hence, when handling exceptions within message handlers to address specific business needs, it is crucial to establish robust error handling and logging mechanisms. By doing so, we can uphold the reliability and consistency of our system, even in challenging circumstances.

Dealing With Message Ordering

Secondly, in distributed systems, it can be challenging to keep the messages in the right order.

Based on our design choices, occasional message discarding without any corresponding actions may occur, potentially causing the saga to become stuck. It’s crucial to recognize that this situation may indicate that we have a problem with handling out-of-order message arrivals, which are scenarios where a message to be processed by the saga arrives before the message intended to initiate it.

Fortunately, NServiceBus provides us with features to help manage message ordering. In order to prevent message discarding in these cases we can either implement the IAmStartedBy<T> interfaces for any message type that expects the saga instance to already exist. Alternatively, we can change the saga not found behavior and throw an exception using the IHandleSagaNotFound interface, leveraging NServiceBus recoverability to retry messages.

Handling Timeouts and Delays

Another key point to note is that when we are dealing with long-running transactions, timeouts, and delays can be a challenge. If a service takes too long to respond, it can cause a saga to fail.

To mitigate this, NServiceBus provides built-in support for timeouts.

We can set a timeout for each step in our saga, and if a service fails to respond in time, we can execute a compensating transaction to handle the failure.

It’s important to remember that once we request a timeout we cannot revoke the request. This means that eventually, the timeout message will arrive at our service. For that reason, our timeout handler should contain logic that will verify if any compensation actions should be performed.

Alternatively, we can use the same mechanism to schedule a delayed message. This might be useful, for example, in cases where we want to implement simple fraud delay capabilities.

Managing Saga Concurrency

If by chance multiple instances of a saga end up being processed at the same time, they might interfere with each other and corrupt the saga data.

For that reason, NServiceBus provides concurrency controls to handle this scenario. In the case of using combinations of persistence and transports that can enlist in transactions, consistency is provided automatically. In other cases, we must use the outbox mechanism, which is beyond the scope of this article.

Growing Complexity

Lastly, implementing the Saga Pattern – especially the choreography-based one – can add complexity to our system. At times we might feel the need to add more and more messages to refine our business logic. But this can lead to a point where it will become difficult to follow the business flows, especially for any new developers joining our teams.

With this in mind, we should always try to keep our sagas as simple as possible.

It’s important to remember that every system and application is unique, so the challenges we face might differ. However, being aware of these common pitfalls and their solutions can help us implement the Saga Pattern more effectively.

Real-World Use Cases for the Saga Pattern

The Saga Pattern has found its place across various industries and types of real-world applications, especially those involving complex business transactions across multiple microservices. We could, for example, implement it in:

• E-commerce
• Online banking
• Supply chain management
• Telecommunications provisioning or billing
• Patient care systems in healthcare
• Booking management for travel

As an illustration of what we have learned about the Saga Pattern let’s consider a sample use case.

E-commerce System Using the Saga Pattern

Let’s consider an e-commerce system. For our purposes, we’ll look at a platform for selling rail tickets.

In such a system, when a customer places an order, several operations need to occur such as inventory check, payment processing, and fulfillment of the tickets. Additionally, when we are dealing with a high-scale selling platform, there can be multiple instances of the services that will be participating in the process.

Now, let’s draw an architecture diagram for such a system:

In this example, all services communicate with each other using a common message bus.

When customers want to purchase rail tickets, they typically begin by searching for fares of interest. Subsequently, they create an order by interacting with the Order service, initiating a saga that encompasses payment, processing, and later, fulfillment.

Notably, the Fulfillment Worker group is an intriguing component. It comprises multiple instances of the same service, which is responsible for handling ticket fulfillment. Each instance listens for incoming Fulfill messages in the queue, and any available instance picks up the pending work. This setup allows for easy scalability by adjusting the number of instances dedicated to ticket fulfillment, ensuring system responsiveness through varying traffic levels.

In fact, any of the services involved in the saga can be easily transformed into such a service cluster to create a more flexible system.

Conclusion

In summary, the combination of the Saga Pattern and NServiceBus in C# offers a powerful solution for managing complex, distributed transactions.

By leveraging the benefits of the Saga Pattern’s transactional workflow and NServiceBus’s messaging capabilities, we can achieve reliable and scalable systems while maintaining data consistency across microservices. Embracing this approach empowers developers to build robust, fault-tolerant applications in the world of distributed systems.