Updated on

Polly is the .NET resilience library. It wraps a call we are about to make (an HTTP request, a database query) in strategies that decide what happens when it fails: retry it, stop calling for a while, fall back to something else, give up after a timeout.

Since version 8 the API is a pipeline. We build a ResiliencePipeline from the strategies we want, in the order we want them applied, and execute our call through it. The older Policy API still exists for compatibility, and everything written against it before 2023 uses that instead.

To download the source code for this article, you can visit our Resilience With Polly repository folder.

What Is Polly in .NET?

Polly is an open-source .NET library for handling failures in calls to things we do not control.

The idea is that transient failure is normal. A network blips, a service restarts, a database is briefly overloaded: code written as though calls always succeed turns each of those into an outage. Polly puts the recovery logic in one declared place instead of scattering try/catch and retry loops through the codebase.

We describe what should happen using strategies: retry this many times with this delay, stop calling after this many failures, give up after this long, fall back to this value. Then we execute our call through the pipeline those strategies form.

What makes it worth a library rather than a helper method is the parts that are hard to get right: exponential backoff with jitter, circuit state shared correctly across concurrent callers, timeouts that actually cancel the work.

Why Do We Need to Build Resilient Microservices?

Before we start adding resilient microservices, it’s worth spending a moment understanding what resiliency is.

What Is Resiliency?

“Resiliency” in the context of software can be described as the ability to maintain acceptable availability for the services it provides, dealing with any issue that may arise in doing so.

For example, if we were to build an API that our customers could consume, we might include in our agreement an SLA (service level agreement) ensuring a certain uptime (e.g 99.9%). In order to maintain this SLA, we need to ensure our service stays ‘up’, or be ‘resilient’.

To build ‘resilient microservices’, we need to ensure our software can deal with issues such as:

  • Increased load
  • Security issues
  • Network failures
  • Dependency failures

In this article, we will be mainly dealing with the last two, being dependency failures caused by the network.

How Can We Be Resilient?

In a previous article, we created an API Gateway to encapsulate a few microservices. Since the consumers only deal with the API Gateway, it means any downstream microservice failure (aka a ‘dependency failure’, in this case), would affect the API Gateway’s ability to service those requests, thus affecting uptime.

In order for our API Gateway to be resilient, we need to be proactive in dealing with these issues. That means, understanding that these issues can and will happen, and having a well-thought-out strategy to deal with them. That could be, but not limited to: returning a previously cached result, falling back to another piece of logic, or erroring “quickly” to not tie up resources that can affect other functionality.

This is where the .NET library Polly can help us.

How Polly Helps Build Resilient Microservices

Polly is a library that helps us build resilient microservices in .NET. It allows us to specify a set of strategies that dictate how our app should respond to various failures. A simple example could be: if a SQL timeout occurs, then “do something” (for example, try again N times). We could of course do this manually, but that would result in a lot of boilerplate and duplicated code, where Polly can do it for us in a much more graceful pattern.

In our case, we will leverage Polly to add resilient features to the API Gateway we built in the previous article so that any microservice failure can be handled properly. The gateway calls its downstream services with a client it gets from registering clients with HttpClientFactory, which is the piece Polly wraps.

Revisiting Our API Gateway and Microservices

In our previous article, we built the following architecture:

Architecture diagram of the API Gateway routing requests to the Authors and Books microservices

Requests would flow into the API Gateway, and be directed to either the Authors or Books microservice, depending on the URL.

We are now going to expand upon this example and add some resiliency features. If you’d like to download the starter project and follow along, head over to the StarterCode folder in our repository.

Simulating a Dependency Failure

In a real-world scenario, the Authors and Books microservice would serve data from a remote place, for example, a SQL Server database or a file on Azure / S3 storage. In our case, they are simply returning data from in-memory collections.

However, to ‘simulate’ a network & dependency failure, we can simply “stop” the authors and books microservices, which will result in the API Gateway returning errors.

To prove that, let’s go ahead and run all three services, and confirm both URLs are working by using our consumer.html page.

First, let’s hit the Get Books button:

GetBooks microservice

Next, the Get Authors button:

GetAuthors microservice

Both our microservices are working, and the API Gateway is in turn responding as expected.

Now, let’s stop both the Books and Authors microservices by stopping the Kestrel process.

If we hit the buttons now, we see a brief pause of a few seconds, then the following error:

Creating resilient microservices in .NET with Polly - Microservice errors

We have just simulated a dependency and network failure! This exact situation can and will happen when we run these apps in production, for example:

  • The network could blip, as a result of infrastructure upgrades
  • The microservice host machine could experience downtime due to a poor deployment strategy, or host upgrades

This is just one example of a failure. There are other reasons that errors could happen, but the point is we can’t control how a downstream dependency could behave, but we can control how we (in this case, the API Gateway) can deal with it.

Using Polly to Build Resilient Microservices

Let’s now introduce Polly to our API Gateway.

Creating a “Fallback” Strategy

The easiest way to do this is via the NuGet package manager console:

PM> install-package Polly.Extensions

Polly.Extensions brings in Polly.Core, which is the v8 API, and adds the extension methods that register a pipeline with the dependency injection container.

The first and most simple way to handle failures with Polly is to capture any Exception, and handle them accordingly. This is called the Fallback strategy.

Rather than build the pipeline inside the controller, let’s describe it once in its own class. Let’s add Resilience/ProxyPipeline.cs to the API Gateway:

using Microsoft.AspNetCore.Mvc;
using Polly;
using Polly.Fallback;

namespace Monolith.Resilience;

public static class ProxyPipeline
{
    public const string Name = "proxy";

    public const string FallbackMessage =
        "Sorry, we are currently experiencing issues. Please try again later";

    public static void Configure(ResiliencePipelineBuilder<IActionResult> builder) =>
        builder
            .AddFallback(new FallbackStrategyOptions<IActionResult>
            {
                ShouldHandle = new PredicateBuilder<IActionResult>().Handle<Exception>(),
                FallbackAction = static _ => Outcome.FromResultAsValueTask<IActionResult>(
                    new ContentResult { Content = FallbackMessage })
            });
}

The v8 API is still very readable. The above can be translated to: “I am building a pipeline for things that return IActionResult. When an Exception occurs, handle it and fall back to returning a friendly message”.

Is this material useful to you? Consider subscribing and get ASP.NET Core Web API Best Practices eBook for FREE!

Two details are worth calling out. AddFallback() exists only on the generic ResiliencePipelineBuilder<T>, so a pipeline that needs a fallback has to declare its result type. And ShouldHandle is where a strategy decides what counts as a failure — every strategy has one, and they do not have to agree with each other.

Next, let’s register the pipeline in Program.cs:

builder.Services.AddResiliencePipeline<string, IActionResult>(
    ProxyPipeline.Name,
    (pipeline, _) => ProxyPipeline.Configure(pipeline));

This is the part v7 made awkward and v8 makes ordinary. The pipeline is built once, held by the container, and shared by every request.

Now let’s open up ProxyController.cs in the API Gateway and ask for it:

public class ProxyController : ControllerBase
{
    private readonly HttpClient _httpClient;
    private readonly ResiliencePipeline<IActionResult> _pipeline;

    public ProxyController(IHttpClientFactory httpClientFactory,
        ResiliencePipelineProvider<string> pipelineProvider)
    {
        _httpClient = httpClientFactory.CreateClient();
        _pipeline = pipelineProvider.GetPipeline<IActionResult>(ProxyPipeline.Name);
    }

Let’s now modify our ProxyTo method, to execute through the pipeline:

private async Task<IActionResult> ProxyTo(string url)
    => await _pipeline.ExecuteAsync(
        async token => (IActionResult)Content(await _httpClient.GetStringAsync(url, token)),
        HttpContext.RequestAborted);

Very simply, we are wrapping our existing code with the call to ExecuteAsync on our pipeline. The callback now receives a CancellationToken, which we pass straight through to HttpClient so that a strategy which gives up can actually stop the work.

Let’s now see what happens when we hit either the Get Books or Get Authors button:

Creating resilient microservices in .NET with Polly - Fallback policy

Great! No matter what happens to our downstream microservices, we can now gracefully handle the error and show something more readable to our consumers.

Creating a “Retry” Strategy

Often in distributed applications, transient errors can occur (network blip, for example). These errors might only exist for a moment, then go away. Without any special code, normal requests would error as soon as the first error occurred. However, we can use the Retry strategy in Polly to proactively expect and handle this error.

It could be difficult to reproduce a network failure, so to simulate this scenario, let’s modify our Authors service to fail on the first request, then succeed on the second.

To do this, let’s open up Repository.cs and add a private member:

private bool _shouldFail = true;

Next, let’s modify the GetAuthors() method:

public IEnumerable<Author> GetAuthors()
{
    if (_shouldFail)
    {
        _shouldFail = false;

        throw new InvalidOperationException("Oops!");
    }

    return _authors;
}

What we are doing is simply failing the first time, then succeeding the times after that.

Let’s build and run the Authors microservice, and see what happens if we hit the Get Authors button:

Creating resilient microservices in .NET with Polly - Authors transient failure

Then let’s hit the button again:

Authors recovered

So the first request could have potentially succeeded if we “retried” again. That said, let’s stop all our applications, and go ahead and do that.

Let’s head back to ProxyPipeline and add a retry strategy after the fallback:

.AddRetry(new RetryStrategyOptions<IActionResult>
{
    ShouldHandle = new PredicateBuilder<IActionResult>().Handle<Exception>(),
    MaxRetryAttempts = 1,
    Delay = TimeSpan.Zero
})

Similar to our existing strategy, here we are handling any exception on things that return IActionResult, and this time retrying once. MaxRetryAttempts counts retries rather than total attempts, so a value of 1 means the call is made at most twice.

The defaults are worth knowing before we change them. Left alone, RetryStrategyOptions retries three times with an exponential backoff, and UseJitter adds randomness so that a fleet of callers does not retry in lockstep. We set Delay to zero here only so the walkthrough stays quick; a real service wants the backoff. If none of that appeals, implementing retry logic by hand is the alternative Polly exists to replace.

Now let’s see what happens if we build and run all applications, and hit the GetAuthors button:

Creating resilient microservices in .NET with Polly - Retry policy

We can see the request succeeded the first time. Unbeknownst to the user, it initially failed but thanks to Polly our application is now “resilient” to this failure, by means of a simple retry.

Creating a “Circuit Breaker” Strategy

Next, let’s look at another resiliency pattern called Circuit Breaker. As the name suggests, this pattern is all about creating a “breaker” so that we don’t keep failing unnecessarily. For example, if a downstream system was timing out, it’s potentially under load, so why keep slamming it? We can give it a break for a while, then try again later.

To simulate this behavior, let’s again go back to our Authors repository, and make some adjustments.

First, let’s add a private member:

private readonly DateTime _startTime = DateTime.UtcNow;

Then let’s modify GetAuthors(), which becomes asynchronous in the process:

public async Task<IEnumerable<Author>> GetAuthorsAsync()
{
    if (_shouldFail)
    {
        _shouldFail = false;

        throw new InvalidOperationException("Oops!");
    }

    if (_startTime.AddMinutes(1) > DateTime.UtcNow)
    {
        await Task.Delay(TimeSpan.FromSeconds(5));

        throw new TimeoutException("Timeout!");
    }

    return _authors;
}

What we are doing now in addition to our previous behavior, is throwing a timeout exception for the first minute of the microservice lifetime.

The five-second wait is await Task.Delay(), not Thread.Sleep(). A blocking sleep on a request thread is the thread-starvation pattern that turns one slow dependency into a whole-application outage — the exact failure this article is about. A simulation of a slow dependency should not demonstrate it.

The controller follows the signature change:

[HttpGet]
public Task<IEnumerable<Author>> Get() => repository.GetAuthorsAsync();

Let’s build and run all our applications again, then hit the Get Authors button again:

Creating resilient microservices in .NET with Polly - Authors timeout

Remember we are still using the Retry strategy, and we still have the “should fail on first request” behavior.

So let’s see what’s happening behind the scenes:

  1. Polly is calling our Authors microservice
  2. The Authors service is failing, because it’s the first request
  3. Polly is calling the Authors microservice again (because of the “Retry” strategy)
  4. The Authors service is waiting for 5 seconds then failing because of a timeout

We can hit the button again and again, and the same behavior above will occur, until finally after a minute, we then see the request succeeds:

Authors eventually succeeding

This behavior causes the following problems:

  • Continuous pressure on the Authors service
  • Resources on the API Gateway are being wasted
  • Excessive network bandwidth
  • Taking too long to fail

To solve these problems, let’s introduce a circuit breaker.

Is this material useful to you? Consider subscribing and get ASP.NET Core Web API Best Practices eBook for FREE!

Adding the Circuit Breaker

First, let’s add another strategy to ProxyPipeline, after the retry:

.AddCircuitBreaker(new CircuitBreakerStrategyOptions<IActionResult>
{
    ShouldHandle = new PredicateBuilder<IActionResult>().Handle<Exception>(),
    FailureRatio = 1.0,
    MinimumThroughput = 2,
    SamplingDuration = TimeSpan.FromSeconds(30),
    BreakDuration = TimeSpan.FromMinutes(1)
});

This is the one place where v8 does not simply rename what v7 did. The v7 circuit breaker counted consecutive failures — CircuitBreakerAsync(2, TimeSpan.FromMinutes(1)) opened after two in a row. The v8 one is rate-based: it opens when the failure ratio over a sampling window reaches FailureRatio, provided at least MinimumThroughput actions ran in that window. So the settings above mean “open the circuit if every action fails, once there have been at least two of them in the last 30 seconds, and keep it open for a minute”.

It’s worth noting how the circuit breaker gets its shared state. A breaker only works if every request consults the same instance, and by default a .NET 10 controller class is instantiated on every request. In v7 this article held the policy in a static field behind a null check — a hand-rolled singleton the article itself apologised for. That is no longer necessary: the pipeline is registered with the dependency injection container, which holds one instance and hands it to every controller.

Let’s build and run all our applications again, and see what happens when we hit the Get Authors button:

Authors Errors

We immediately get a 500 error, which is the “fail first-time” behavior we implemented in our Authors repository. This is error one.

If we hit it again, we get a short delay of a few seconds, and the same error again. This is the “timeout” behavior occurring. This is error two.

If we hit it a third time, we now fail quickly, and without the fallback in front of it the gateway would surface this:

Unhandled exception. Polly.CircuitBreaker.BrokenCircuitException: The circuit is now open and is not allowing calls.

BrokenCircuitException confirms the circuit is “open”, and Polly won’t try to perform the action for a total of 1 minute, saving precious resources and “failing fast”, which as we mentioned earlier is a great principle in building resilient microservices. The type survives the v7 to v8 move unchanged, so old log searches still find it.

If we wait a minute, we’ll then see the normal successful response, signifying the circuit is “closed” and normal behavior continues.

Combining Our Strategies

So far, in each example, we’ve dealt with one particular issue and handled them in one way. However true resilience means dealing with a variety of scenarios and handling them in different ways. In v7 this meant wrapping one policy in another with PolicyWrap. In v8 there is nothing extra to do: the three calls we have already made to the same builder are the combination.

Here is the finished Configure method:

public static void Configure(ResiliencePipelineBuilder<IActionResult> builder) =>
    builder
        .AddFallback(new FallbackStrategyOptions<IActionResult>
        {
            ShouldHandle = new PredicateBuilder<IActionResult>().Handle<Exception>(),
            FallbackAction = static _ => Outcome.FromResultAsValueTask<IActionResult>(
                new ContentResult { Content = FallbackMessage })
        })
        .AddRetry(new RetryStrategyOptions<IActionResult>
        {
            ShouldHandle = new PredicateBuilder<IActionResult>().Handle<Exception>(),
            MaxRetryAttempts = 1,
            Delay = TimeSpan.Zero
        })
        .AddCircuitBreaker(new CircuitBreakerStrategyOptions<IActionResult>
        {
            ShouldHandle = new PredicateBuilder<IActionResult>().Handle<Exception>(),
            FailureRatio = 1.0,
            MinimumThroughput = 2,
            SamplingDuration = TimeSpan.FromSeconds(30),
            BreakDuration = TimeSpan.FromMinutes(1)
        });

The order of the calls is the order of the layers, outermost first. The fallback is added first, so it sits outside everything and catches whatever the rest lets through. The retry is next, so it re-runs the call. The circuit breaker is innermost, so it counts real attempts against the dependency rather than counting retries as separate failures. Effectively we are saying: retry once, fall back to what we specify, and open the circuit for a minute once the attempts in the window have all failed.

Our ProxyTo method does not change at all — it already executes through _pipeline.

If we build and run all our apps again, we see the following behavior each time we press Get Authors:

  1. A brief pause, then the fallback error returned (behind the scenes, we actually retried once)
  2. Immediately getting the same fallback error (the circuit is now “open”)

Subsequent clicks will get the same behavior as 2 until 1-minute passes and normal behavior resumes.

This is extremely powerful as we are:

  • Retrying to deal with intermittent failures (retry)
  • Not wasting resources (circuit breaker)
  • Showing something to the user (fallback)

What Changed in Polly v8?

Version 8 replaced policies with pipelines, and the old API still works.

In v7 each strategy was a policy object with its own type (AsyncRetryPolicy, AsyncCircuitBreakerPolicy), and combining them meant wrapping one in another with PolicyWrap, where the nesting order determined the behaviour and was easy to get backwards.

In v8 we add strategies to a builder in the order they should apply, and build one ResiliencePipeline. The first strategy added is the outermost, which reads the way the code is written rather than inside-out.

The rest follows from that. Strategies are configured with options objects instead of long parameter lists, telemetry is built in, and the library provides zero-allocation APIs for advanced use cases.

Polly’s own migration guide confirms it: “The v7 API is still available and fully supported even when using the v8 version by referencing the Polly package.” So migration is a project we schedule, not an emergency.

Side-by-side diagram comparing Polly v7 PolicyWrap nesting order with v8 ResiliencePipelineBuilder declaration order.

The one v7 strategy without a same-named successor is the bulkhead. Polly’s migration guide explains why: “In v8, it’s not separately exposed because it’s essentially a specialized type of rate limiter: the ConcurrencyLimiter.”

Do We Still Need Polly on .NET 10?

For HTTP calls, often not directly, but Polly is still what runs underneath.

.NET ships a resilience package for HttpClient, Microsoft.Extensions.Http.Resilience, whose AddStandardResilienceHandler() adds a standard set of strategies to a registered client in one call. That handler is five Polly strategies with sensible defaults: a rate limiter, a total request timeout, a retry, a circuit breaker, and a per-attempt timeout.

For anything that is not an HttpClient call, we still build the pipeline ourselves. Database access, message queues, third-party SDKs, background jobs: none of them are covered by the HTTP package, and that is where Polly’s own API earns its place.

The decision is therefore about the call being made, not the library itself. HTTP through a registered client gets the standard handler; everything else gets a pipeline we define by hand. Both are Polly underneath, and mixing them in one application is normal rather than inconsistent.

StrategyWhat it doesWhen it helpsv7 name
RetryRe-executes the same call after a failure, waiting between attemptsTransient failures: a dropped connection, a brief timeoutRetryPolicy
Circuit breakerShort-circuits the call once failures cross a threshold, then tries again after a breakA dependency that is down or overloadedCircuitBreakerPolicy
TimeoutCancels a call that does not finish inside the timeoutA dependency that hangs rather than failingTimeoutPolicy
FallbackSubstitutes a value or action when everything else has failedDegrading gracefully instead of erroringFallbackPolicy
HedgingRe-executes the call in parallel when the first attempt is slowTail latency across redundant endpointsnone (new in v8)
Rate limiterLimits how many executions are permitted to pass throughProtecting a dependency from being called too oftennone (no v7 strategy)
Concurrency limiterLimits how many executions run at the same timeStopping one slow dependency from consuming every callerBulkhead

The two limiter rows need a package the others do not. AddRateLimiter() and AddConcurrencyLimiter() live in Polly.RateLimiting; with only Polly or Polly.Extensions referenced they will not compile. AddFallback() and AddHedging() have a different restriction — they exist only on the generic ResiliencePipelineBuilder<T>, which is why the pipeline we built above declares IActionResult as its result type.

If the call being wrapped is an HttpClient one, there is more to read: retrying failed HttpClient requests with Polly covers that specific recipe in detail, extending HttpClient with delegating handlers explains the mechanism the standard handler plugs into, and setting request timeouts on HttpClient covers the timeouts the standard handler configures for us.

Conclusion

There are a number of other features that exist in Polly to help build resilient microservices, and we have only touched on a few common ones.

We all know that when we move from a single, in-process application to a set of microservices, a number of challenges come with it. Even though our application isn’t doing anything meaningful, with a bit of imagination it’s easy to see how real problems can occur and how Polly can help us deal with these problems.

In this article, we have shown how Polly can help prepare us for these inevitable problems, and therefore increase our resiliency, which in turn increases uptime, saves resources, and improves customer satisfaction.

Happy coding!

Tested with .NET 10.0.10 and Polly 8.7.0.