Log Service in .NET with Graylog

When we build applications or systems, we need to know what is happening inside them.

To understand what's going on in the system, the architecture, or the flow connecting applications, we use observability, which we will cover in another video. 

And to know what happens within an individual application, we use logs. 

1 - What are logs?

Logs are the part of our system (usually libraries) that keep a record of, generally, errors or actions that occur in our applications. 

For example, if we try to create a purchase order in our system for a product that doesn't exist, the system should notify the user that there is an error, but at the same time, it should let us know about the error with more details so that we can identify the problem. 

This means that logs are not visible to the end user, only to administrators. 

1.1 - What information should we store in logs? 

We should store errors in the logs, preferably with details, always complying with GDPR, so no personal names, phone numbers, emails, passwords, etc. 

Besides the location where the error happened, in .NET we have access to the stacktrace, which tells us the exact location and the process the code followed to get there. 

1.2 - How many logs should we store? 

One of the big questions when developing is whether you should log what you are doing or not. For example, in the producer-consumer pattern, should we log when the consumer runs but there are no messages, or when there are messages, or when publishing? These day-to-day questions are very common.

The answer is: at the start of our application's life, yes, but later we can remove some logs. This means that in the initial versions of our software, we'll log all important steps, but after it has been running for a while and we know everything works as expected, we'll only log errors. 

2 - Where are logs stored

By default, when we use logs, they're saved in the file system, but obviously, we can change this and, in fact, we should.

Although when developing locally storing logs isn't so important, it is when applications are deployed, whether in dev, uat, or production. In fact, dev is very important, because if we see an alert in dev that hasn't happened in production, we can fix the bug before it affects a client.

Of course, when we have applications on servers, logs should be outside the server, preferably in an external service. 

In our case, we'll store them in Graylog (through docker), but there are other options like datadog, newrelic, or the native logging service if we're using a provider like AWS or Azure. 

3 - Creating logs in .NET

Within .NET, we have several options for creating logs. The two most common are Microsoft's own logging library, and the SeriLog library, which is, in my opinion, the most complete. 

When Microsoft developed its version, they built it with external libraries in mind and allow configuring their library to use serilog under the hood. 

This is very powerful, because serilog provides what are called "sinks" that let us configure multiple destinations for our logs. 

This means that if you're migrating from one logging service to another, you can log to both for some time without affecting your code; the same goes for console or file logging. 

Then we can use dependency injection of Microsoft's log into our services, so we don’t have to propagate the serilog dependency everywhere. 

In my opinion, it was a brilliant move by Microsoft to build the logger this way. 

4 - Implementing a logger in .NET

The first step is to add Graylog to our docker compose. Due to its operation, we must also include mongo and elasticsearch, but we won't work with them directly, Graylog will handle that:

#########################
  # Graylog configuration #
  #########################
  # mongo should be called mongo
  mongo:
    container_name: mongo_graylog
    image: mongo:4.2
  elasticsearch:
    container_name: elasticserach_graylog
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2
    environment:
      - http.host=0.0.0.0
      - transport.host=localhost
      - network.host=0.0.0.0
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    deploy:
      resources:
        limits:
          memory: 1g
  graylog:
    image: graylog/graylog:4.0
    environment:
      # at least 16 characters
      - GRAYLOG_PASSWORD_SECRET=thispassshouldbeatleast16characters
      # Password: admin
      - GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
      - GRAYLOG_HTTP_EXTERNAL_URI=http://localhost:9000/
    entrypoint: /usr/bin/tini -- wait-for-it elasticsearch:9200 --  /docker-entrypoint.sh
    restart: always
    depends_on:
      - mongo
      - elasticsearch
    ports:
      - 9000:9000 # Graylog web interface and REST API
      - 1514:1514
      - 1514:1514/udp
      - 12201:12201
      - 12201:12201/udp
  #############################
  # Graylog configuration end #
  #############################

Once this is done, with docker-compose up -d, we can go to graylog in the browser (http:\localhost:9000).

Note: username and password are admin - admin.

Before moving to the code, keep in mind: we must manually update Graylog inputs to the system. To do this, we need to create one (or more if we have multiple servers sending information) to receive glef information, which is the message format created by Graylog (sent by default).

All the information is default except for the name, and you'll see the following in the UI:

Keep in mind that in production environments, the logging system is created once and never shut down, so this wouldn't be a problem. 

But if someone knows how to create this configuration in docker, please contact me privately on Twitter, and I'll add it to the post. 

4.1 - Logger abstraction with C# and serilog

Now we need to configure our application. Like in all the videos of the series, we are going to use code from the Distribt github repo. You can create a new project or use the setup project. In my case, I’ll create a project called Distribt.Shared.Logging, where we’ll install the serilog package as well as two sinks, for console and Graylog, and finally the configuration with aspnetcore.

With the Serilog.Sinks.Graylog package, we can communicate with .Graylog; with the .Console package, we can communicate with the console running our app to see the messages. 

• Note: I recommend not enabling the console sink in dev/uat/production unless you are debugging in such environment. 

And we're going to create the classes that will contain the configurations. To do this, we create two classes:

public record ConsoleLoggerConfiguration
{
    
}

public record GraylogLoggerConfiguration
{
    
}

Which will store the information.

4.1.1- Configure a console sink with Serilog

The console sink case is very simple, we just need to indicate whether it’s enabled and what is the minimum level we want to log. 

public class ConsoleLoggerConfiguration
{
    public bool Enabled { get; set; } = false;
    public LogEventLevel MinimumLevel { get; set; }
}

When configuring the logger, we’ll do so via the LoggerConfiguration type, so we need to send this info to that type. 

We have the option of creating a method in the config class itself or creating an extension method that adds serilog through its WriteTo option:

public static class LoggerConfigurationExtensions
{
    public static LoggerConfiguration AddConsoleLogger(this LoggerConfiguration loggerConfiguration,
        ConsoleLoggerConfiguration consoleLoggerConfiguration)
    {
        return consoleLoggerConfiguration.Enabled
            ? loggerConfiguration.WriteTo.Console(consoleLoggerConfiguration.MinimumLevel)
            : loggerConfiguration;
    }  
}

This config will come from the config files and will look like this:

"Logging": {
  ...
    "Console": {
      "Enabled": true,
      "MinimumLevel": "Error"
    }
    ...
}

• Note: we can specify default values if this section is missing.

4.1.2 - Configure a Graylog sink with Serilog

Now we have to do the same to communicate with serilog, but in this case, the config needs more info (host, port).

public class GraylogLoggerConfiguration
{
    public bool Enabled { get; set; } = false;
    public string Host { get; set; } = "";
    public int Port { get; set; } 
    public LogEventLevel MinimumLevel { get; set; }
}

And as before, we need to be able to configure LoggerConfiguration:

public static LoggerConfiguration AddGraylogLogger(this LoggerConfiguration loggerConfiguration,
        GraylogLoggerConfiguration graylogLoggerConfiguration)
    {
        return graylogLoggerConfiguration.Enabled
            ? loggerConfiguration.WriteTo.Graylog(graylogLoggerConfiguration.Host, graylogLoggerConfiguration.Port, 
                TransportType.Udp, graylogLoggerConfiguration.MinimumLevel)
            : loggerConfiguration;
    }

Don’t forget how the config would look:

"Logging": {
.....
    "Graylog": {
      "Enabled": true,
      "Host": "localhost",
      "Port": "12201",
      "MinimumLevel": "Error"
    },
.....
}

4.2 - Configure serilog in .NET 

Now that we have the configuration specified, we have to connect all the points so we can log information correctly. 

In the case of serilog, we will add it to the host running the application rather than the dependency container. 

In versions before .NET6, we did it from IWebHostBuilder. Since .NET 6 we must do it from IHostBuilder, so we create an extension method to configure this info:

public static class ConfigureLogger
{

    public static IHostBuilder ConfigureSerilog(this IHostBuilder builder)
        => builder.UseSerilog((context, loggerConfiguration) 
            => ConfigureSerilogLogger(loggerConfiguration, context.Configuration));

    private static LoggerConfiguration ConfigureSerilogLogger(LoggerConfiguration loggerConfiguration,
        IConfiguration configuration)
    {
        
        GraylogLoggerConfiguration graylogLogger = new GraylogLoggerConfiguration();
        configuration.GetSection("Logging:Graylog").Bind(graylogLogger);
        ConsoleLoggerConfiguration consoleLogger = new ConsoleLoggerConfiguration();
        configuration.GetSection("Logging:Console").Bind(consoleLogger);
        
        return loggerConfiguration.AddConsoleLogger(consoleLogger).AddGraylogLogger(graylogLogger);
    }

}

As you can see, we've used config binding to read info from the appsettings.

Now we need to invoke it from our WebApplicationBuilder:

public static WebApplication Create(Action<WebApplicationBuilder>? webappBuilder = null)
{
  WebApplicationBuilder builder = WebApplication.CreateBuilder();
  builder.Host.ConfigureSerilog();
  ...
  return builder.Build();
}

And finally, to connect SeriLog with Microsoft logging, we just need to specify it when adding the logger to the dependency container:

builder.Services.AddLogging(logger => logger.AddSerilog());

Now we just have to try it out.

To inject the logger, we need to call Ilogger<T> where T represents the current service. Technically, we can use anything in the generic T, but for convention and clarity, we should use the current service. 

public class OrderController
{
    private readonly IDomainMessagePublisher _domainMessagePublisher;
    private readonly ILogger<OrderController>  _logger;
    
    public OrderController(ILogger<OrderController> logger,  IDomainMessagePublisher domainMessagePublisher)
    {
        _domainMessagePublisher = domainMessagePublisher;
        _logger = logger;
    }

    [HttpGet("{orderId}")]
    public Task<OrderDto> GetOrder(Guid orderId)
    {
        _logger.LogError($"esto es un mensaje de ejemplo con el order {orderId}");

        //Todo, change for a real one as this one is only to test the logger.
        return Task.FromResult(new OrderDto(orderId, new OrderAddress("stree1", "postalCode"), 
            new PersonalDetails("name", "surname"), new List<ProductDto>()));
    }
}

As we can see, we can view the message both in the console and in Graylog:

In the future, we will look at both serilog and Graylog in more detail.

Conclusion

• In this post we saw what logging is in programming.
• How the serilog library works
• What Graylog is and how to configure it with serilog.