Creating Fault-Tolerant Applications with Polly

In this post, we will cover Polly, a very common library in the world of C#, not only for companies but also internally used by Microsoft. Here, we'll go through an introduction.

As always, you know that the code is available on GitHub.

1 - What is Polly?

Polly is an open source library that lets us configure HTTP calls to make them more resilient.

Honestly, the word resilient is a bit strange; if you are used to speaking in English, you've heard resilient, but what does it mean? It just means fault-tolerant.

So basically, we can use Polly to make our apps fault-tolerant or able to recover from those.

But what kind of faults?

Polly is used to control HTTP calls we make from our applications to others.

Before moving on, we can also configure this tolerance with databases or other systems, but 99% of use cases are with HTTP calls, so that's what we’ll focus on in this blog.

The main idea of Polly is to ensure everything keeps working and that a minor network error does not crash the entire app.

Within Polly, there are two ways to execute these configurations

1.1 - Individual application of fault tolerance

Basically, it's a function in our code that lets us set a special policy for a specific call. For example, we might want the call to try 3 times, or we want the call to cancel after 3 seconds, things like that.

1.2 - Resilient pipeline in Polly

With this option, we create several policies and can combine them very easily.

We’ll see an example later.

2 - Using Polly inside our C# application

As always in these situations, the first thing to do is import the package from NuGet Polly, and now we can use it.

For this example, I’m using the default Web.API template, I’ve just added a call counter and if the counter has not reached a certain number, it throws an exception.

Obviously this is just for testing, don't do this in production.

//This times is here just to fail the first call and be able to test polly
int times = 0;
app.MapGet("/best-endpoint", () =>
    {

        if (times == 0)
        {
            times++;
            throw new Exception("just an example");
        }
      
        return "Ok";
    })
    .WithName("BestEndpoint")
    .WithOpenApi();

We also have an endpoint that makes an HTTP call to the endpoint we just created:

app.MapGet("/polly-execution1", (IHttpClientFactory httpClientFactory) =>
    {
        HttpClient client = httpClientFactory.CreateClient("PollyExample");

        return client.GetStringAsync("best-endpoint");
    })
    .WithName("PollyExample1")
    .WithOpenApi();

With this, we can now test Polly

2.1 - Retrying calls with Polly

Polly’s most popular function is definitely retrying calls. There are several ways to do it; for example, you can retry based on the response status code, or if an exception is thrown, the type of exception.

Something very important is that we can add wait times between calls, and they can be incremental, for example, between the first and second, wait 500ms, between the second and third wait 1 second, and so on. This is important because if we don’t wait and let’s say we have 5 calls, they could all execute in less than a second, and if the error is a network issue or the destination app is overwhelmed, that retry won’t help much. But if we wait, maybe it will.

app.MapGet("/polly-execution-retry", (IHttpClientFactory httpClientFactory) =>
    {
        HttpClient client = httpClientFactory.CreateClient("PollyExample");

        AsyncRetryPolicy policy = Policy
            .Handle<Exception>()
            .WaitAndRetryAsync(3, retryTime =>
                TimeSpan.FromSeconds(0.5 * retryTime));

        return policy.ExecuteAsync(async () => await client.GetStringAsync("best-endpoint"));
    })
    .WithName("PollyExampleRetry")
    .WithOpenApi();

Here, we’re setting the policy to try 3 times if there is an error.

2.2 - Circuit breaker with Polly

Circuit breaker is also quite popular, though mainly used when we have microservices and create calls between them. Because really, hardly anyone uses circuit breaker when calling a third-party service , that’s what you pay them for.

A circuit breaker means that when the service we’re calling times out, subsequent executions to that service are canceled.

In the previous example, if the first call takes 10 seconds and fails, the second takes 10 seconds and fails, why make a third, fourth, or fifth call?

To simulate this, let’s go to the endpoint we're using and simply add a thread sleep of 10 seconds:

app.MapGet("/best-endpoint", () =>
    {
        switch (times)
        {
            case 0:
                times++;
                throw new Exception("just an example");
            case 1:
                times++;
                Thread.Sleep(10000);
                throw new Exception("just an example");
            default:
                return "Ok";
        }
    })
    .WithName("BestEndpoint")
    .WithOpenApi();

Now let’s create a Polly policy with a circuit breaker:

AsyncCircuitBreakerPolicy policyCircuitBreaker = Policy
    .Handle<Exception>()
    .CircuitBreakerAsync(2, TimeSpan.FromMinutes(10));

app.MapGet("/polly-execution-circuitbreaker", (IHttpClientFactory httpClientFactory) =>
    {
        HttpClient client = httpClientFactory.CreateClient("PollyExample");
        return policyCircuitBreaker.ExecuteAsync(async () => await client.GetStringAsync("best-endpoint"));
    })
    .WithName("PollyExampleCircuitBreaker")
    .WithOpenApi();

Note: The policy initialization is outside the endpoint; if you place it inside, it gets created every time.

In this case, we check if the call fails 2 times in 1 minute, but if you are adventurous, you can create advanced thresholds so that not having 3 consecutives in 60 seconds is okay, but if one in the middle works, everything keeps running, etc.

And if we execute the endpoint, it will fail instantly. This is because the policy has no retry, so it will fail two times, the second taking longer, and when we try a third time, it will return a circuit breaker error, which will then wait the set time before working again , 10 minutes in my case.

Both retry and circuit breaker are the ones I’ve personally seen the most in production code.

2.3 - Fallback policy in Polly

Another policy you might encounter is fallback, which basically means, if the call fails, do something else. This could be hardcoding the result or, in production, more likely calling another service. Typically, I’ve seen fallback used during migrations.

app.MapGet("/polly-execution-fallback", (IHttpClientFactory httpClientFactory) =>
    {
        HttpClient client = httpClientFactory.CreateClient("PollyExample");
        AsyncFallbackPolicy<string> fallback = Policy<string>
            .Handle<Exception>()
            .FallbackAsync("All good here, nothing to see");

        return fallback.ExecuteAsync(async () => await client.GetStringAsync("best-endpoint"));
    })
    .WithName("PollyExampleFallback")
    .WithOpenApi();

Basically, if it fails it returns what we indicated in fallback, in this case, a string, but it could be the call to another method. This is the result:

2.4 - Combining policies with Polly

Finally, let’s talk about how to combine multiple policies. If you paid attention to the types (which is why it’s good to use types and not var…), the types were FallbackPolicy, CircuitBreakerPolicy, etc. But what if, for example, we want to retry 3 times, but with a maximum execution time of 3 seconds , so if it takes more than 3 seconds, we retry?

This is where configuring multiple policies comes in. We create them using a resilient pipeline.

Note: In earlier versions, we did Policy.Wrap(fallback, retrypolicy, etc); But I find this new way cleaner and clearer.

To do this, we create the pipeline with the policies we need:

app.MapGet("/polly-execution-resilience-pipeline", async (IHttpClientFactory httpClientFactory) =>
    {
        HttpClient client = httpClientFactory.CreateClient("PollyExample");

        ResiliencePipeline<HttpResponseMessage> pipeline = new ResiliencePipelineBuilder<HttpResponseMessage>()
            .AddRetry(new RetryStrategyOptions<HttpResponseMessage>()
            {
                ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
                    .Handle<Exception>()
                    .HandleResult(response => response.StatusCode >= HttpStatusCode.InternalServerError),
                MaxRetryAttempts = 3,
                BackoffType = DelayBackoffType.Exponential
            })
            .AddTimeout(TimeSpan.FromSeconds(3))
            .Build();

       await pipeline.ExecuteAsync(async token => await client.GetAsync("best-endpoint"));
    })
    .WithName("PollyExampleResilience")
    .WithOpenApi();

3 - Implementing a policy directly on HttpClient with Polly

Microsoft, in collaboration with the Polly team, created a package called Microsoft.Extensions.Http.Resilience that allows us to include these kinds of mechanisms directly on HttpClient. And using the same logic as the previous example, we can now create the policy on the httpclient, which means it will affect all calls and always be used with that httpclient.

To do this, we use AddReslicienceHandler, which takes a string parameter as an identifier. This identifier is used in telemetry information if enabled. Then an Action delegate for configuration, which is the same as the previous point.

builder.Services
    .AddHttpClient("PollyExample", client =>
    {
        client.BaseAddress = new Uri("http://localhost:5005");
    }).AddResilienceHandler("api-1", config =>
    {
        config.AddRetry(new HttpRetryStrategyOptions()
        {
            MaxRetryAttempts = 2,
            BackoffType = DelayBackoffType.Exponential,
            Delay = TimeSpan.FromMilliseconds(500)
        });

        config.AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions()
        {
            SamplingDuration = TimeSpan.FromSeconds(5),
            FailureRatio = 0.9,
            MinimumThroughput = 5,
            BreakDuration = TimeSpan.FromSeconds(10)
        });
    });

With this, no matter where you use the client, all will have those policies, which is very, very good.

By the way, if you find this too verbose, you can use addStandardResilienceHandler, which removes a lot of code, but it’s more limited , you don’t have an infinite number of policies you can apply, and it only allows one of each type.

builder.Services
    .AddHttpClient("PollyExample", client =>
    {
        client.BaseAddress = new Uri("http://localhost:5005");
    }).AddStandardResilienceHandler()
    .Configure(options =>
    {
        options.Retry.MaxRetryAttempts = 2;
        options.Retry.BackoffType = DelayBackoffType.Exponential;
        options.Retry.Delay = TimeSpan.FromSeconds(500);

        options.CircuitBreaker.SamplingDuration = TimeSpan.FromSeconds(5);
        options.CircuitBreaker.FailureRatio = 0.9;
        options.CircuitBreaker.MinimumThroughput = 5;
        options.CircuitBreaker.BreakDuration = TimeSpan.FromSeconds(10);
    });

4 - Conclusion

To wrap up, a brief conclusion. Using libraries like this, or even creating your own, is very important in the business environment, as it's more common than you'd think for a service to be down for a second, so we need to keep this functionality in mind.

That said, Polly is a much more complete library than what I’ve covered today. Here, I focused on its main features and a first contact, which, to be honest, serve 90% of use cases. But if you want to go deeper, I recommend checking its documentation.