Health checks in ASP.NET Core

If we are building microservices, we will eventually reach a point where we need to constantly know if our application is running and not suffering from any issues that might take it down. 

We could perform this check manually, but we would spend the entire day refreshing the page. 

What we're going to do is use health checks to automate this process. 

1 - What is a health check?

A health check is what we use to see if an application or part of our infrastructure is running as it should.

This check goes beyond its basic form of "is it working or not?", we can also include checks like response time, memory usage, or even dependency verification. 

1.1 - Types of health checks

You can include as many as you want, adapt them to your needs, the most common are:

• Basic check, known as "basic probe", which is an endpoint that tells you whether the application is responding or not.
• Basic check with dependencies: Same as above but checks that each dependency is running correctly. (This has to be built manually); Dependencies include other microservices as well as infrastructure elements like the database, service bus, etc. 
• System checks: From my personal experience, these are usually done when we deploy applications to the cloud, whether serverless or in containers, to make sure we aren’t allocating unnecessary resources. We usually check elements like CPU usage, memory, disk use, etc.

1.2 - Health check response levels

Once we have the types, let’s talk about the response.

As a general rule, we’ll use 3 main response levels:

• Healthy: everything is working as expected
• Degraded: The application is working but very slowly. For example, an API call that usually takes 120ms now takes 5000ms.
• Unhealthy: the application is down or not functioning as intended.

2 - How and when to use health checks

In my opinion, we should have health checks for all elements of our system, both applications and infrastructure.

This means every application we create from now on should have an endpoint that exclusively checks that everything is working.

The same applies for infrastructure elements. Every service you use should be able to be checked. Keep in mind different services use different methods, but most will be commands or HTTP requests.

And we should always include them, a system's failure should never go unnoticed since it can have very bad consequences from the client side. 

3 - Implementing Health checks in ASP.NET Core

To implement the code for this application, we’re going to continue with the code from the distributed systems course Distribt, but the code is the same in any other system. 

First, we’ll add a basic health check to ALL the services we have.

So, we navigate to where we have our setup abstraction and include in services .AddhealthChecks().

WebApplicationBuilder builder = WebApplication.CreateBuilder(args);

builder.Services.AddHealthChecks();

Afterwards, when we’ve built our WebApplication we need to call MapHealthChecks passing the path where the health check will be located.

WebApplication app = builder.Build();

app.MapHealthChecks("/health");
...

Once configured, if we run the application and call the /health endpoint we get the following result: 

Its status code will be 200, indicating it worked properly. 

• NOTE: If you are using the Distribt library, this information is in Distribt.Shared.Setup.API class DefaultDistribtWebApplication, meaning all applications will have this health check by default.  

3.1 - Checking dependencies in a health check

But what if we want to check more than just if the app is running, what if we want to check dependencies too? 

Most elements we want to check can be done via their own NuGet library. If you search NuGet for AspNetCore.HealthChecks. you will see about 70 results, these are libraries already created for these health checks. 

In our case, we use two kinds of databases as seen in the post on CQRS, so we’re going to import AspNetCore.HealthChecks.MongoDb and AspNetCore.HealthChecks.MySql.

You can do this in the application affected, or as in my particular case, in the abstraction.

If you’re not using Distribt, just after .AddHealthChecks(); include the extension method for your system:

builder.Services.AddHealthChecks.AddMongoDb(configuration.GetSection("Database:MongoDb"));

And you’ll need to repeat this for each dependency.

If you are using the Distribt library, this code will be abstracted in each service, and automatically added when you configure the service. For MongoDb, we create a method that calculates the information: 

public static IServiceCollection AddMongoHealthCheck(this IServiceCollection serviceCollection)
{
    ServiceProvider serviceProvider = serviceCollection.BuildServiceProvider();
    string mongoConnectionString = serviceProvider.GetRequiredService<IMongoDbConnectionProvider>().GetMongoConnectionString();
    serviceCollection.AddHealthChecks().AddMongoDb(mongoConnectionString);
    return serviceCollection;
}

Then, in the method called by our applications

public static IServiceCollection AddDistribtMongoDbConnectionProvider(this IServiceCollection serviceCollection,
    IConfiguration configuration)
{
    return serviceCollection
        .AddMongoDbConnectionProvider()
        .AddMongoDbDatabaseConfiguration(configuration)
        .AddMongoHealthCheck();
}

With this, our applications don’t need to configure the health check on their own, just parent functionality is enough.

You need to repeat this for every software you use.

Now if you re-check the /health endpoint it will return "healthy", but if you stop the database in docker it will return "Unhealthy"

It will also have a 503 status code, which means error.

3.2 - Creating a custom health check in .NET

Besides the health checks that come from other libraries, we can create our own.

As I mentioned before, it’s common to have health checks that monitor response time or memory usage, but those are more often created at the infrastructure level, since many services handle that for us. 

For example, if everything is in the cloud, all providers offer ways to access that information and send alerts, etc. 

What we’re going to do is create our own health check to verify another microservice we depend on is working correctly. 

In the Distribt Project, we have a direct relationship (via RabbitMq) between the Orders microservice and the Products microservice. Therefore, to always have the correct info in an order (e.g., the name), the products microservice must be running as expected. 

If not, the orders microservice will still work, since we have eventual consistency and we duplicate the critical information we need. 

To create a custom health check, create a class that inherits IHealthCheck and implement this method:

public class ProductsHealthCheck : IHealthCheck
{
    public Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context,
        CancellationToken cancellationToken = new CancellationToken())
    {
        throw new NotImplementedException();
    }
}

In our case, we’re going to make an HTTP call to the products microservice, so the code would look like this:

public class ProductsHealthCheck : IHealthCheck
{
    private readonly IHttpClientFactory _httpClientFactory;
    private readonly IServiceDiscovery _discovery;

    public ProductsHealthCheck(IHttpClientFactory httpClientFactory, IServiceDiscovery discovery)
    {
        _httpClientFactory = httpClientFactory;
        _discovery = discovery;
    }

    public async Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context,
        CancellationToken cancellationToken = new CancellationToken())
    {
        //TODO: abstract out all the HTTP calls to other distribt microservices #26
        HttpClient client = _httpClientFactory.CreateClient();
        string productsReadApi =
            await _discovery.GetFullAddress(DiscoveryServices.Microservices.ProductsApi.ApiRead, cancellationToken);
        client.BaseAddress = new Uri(productsReadApi);
        HttpResponseMessage responseMessage = await client.GetAsync($"health", cancellationToken);
        if (responseMessage.IsSuccessStatusCode)
        {
            return HealthCheckResult.Healthy("Product service is healthy");
        }
        
        return HealthCheckResult.Degraded("Product service is down");
    }
}

We mark it as Degraded because we can still create orders, but product names would not be updated if the service is down.

Note: if an exception is thrown during the health check, it will be transformed to Unhealthy.

Now we just need to add the healthcheck to our running healthchecks with .AddCheck<T>

webappBuilder.Services.AddHealthChecks().AddCheck<ProductsHealthCheck>(nameof(ProductsHealthCheck));

Now the result of our application’s health check is the combination of all the health checks we have. 

But this aggregates everything into a single result, which is not ideal. What we need to do is import the NuGet package AspNetCore.HealthChecks.UI.Client and set the following configuration:

webApp.UseHealthChecks("/health", new HealthCheckOptions()
{
    Predicate = _ => true,
    ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse
});

This way, we can see the results individually per check to know what's down and what's not.

as we can see we already have a better explanation of what is failing. 

3.3 - Including an interface for health checks 

So far all we have is an endpoint that gives us a response.

What if we want a visual interface?

For this, there is another NuGet package called AspNetCore.HealthChecks.UI which will provide us with a small interface.

It’s a simple change; just indicate in the services:

services.AddHealthChecksUI();

And since we need to store this information to show it, we’ll use AspNetCore.HealthChecks.UI.InMemory.Storage which allows us to store info in memory:

builder.Services.AddHealthChecksUI().AddInMemoryStorage();

and in the app section where we configure the URL for the interface and endpoint to check for results.

webApp.UseHealthChecksUI(config =>
{
    config.UIPath = "/health-ui";            
});

Now in each microservice, you must indicate which endpoint to call for this configuration in the appsettings.json file.

{
  ...
  "HealthChecksUI": {
    "HealthChecks": [
      {
        "Name": "Orders health check",
        "Uri": "/health"
      }
    ]
  }
  ...
}

And if we run the application, we can see the result.

• Note: In the Distribt project, this info is preconfigured to work with the Distribt.Shared.Setup project without needing to modify appsettings.

This is just for the configured microservice, but we could set up one microservice to add observability for all microservices, since the configuration file contains an array. But we'll see this global observability in another post. 

Conclusion

In this post we have seen what a health check is

What the states of a health check are

When to configure health checks

How to build and configure health checks with .net