Logging with ElasticSearch, Kibana, ASP.NET Core and Docker

A Step by Step Guide to Logging with ElasticSearch, Kibana, ASP.NET Core 2.1 and Docker

Published on May 31, 2018
ElasticSearch, Kibana, Docker and .NET Core - Logging with ElasticSearch, Kibana, ASP.NET Core and Docker
Logging with ElasticSearch, Kibana, Docker and .NET Core

In this tutorial, I’ll show you how you can get up and running with ElasticSearch, Kibana and ASP.NET Core 2.1

Before we get started, let’s look at what ElasticSearch, Kibana and Serilog are.

What is ElasticSearch?

In simple terms, ElasticSearch is an open source database that is well suited to indexing logs and analytical data.

What is Kibana?

Kibana is an open source data visualization user interface for ElasticSearch. Think of ElasticSearch as the database and Kibana as the web user interface which you can use to build graphs and query data in ElasticSearch.

What is Serilog?

Serilog is a plugin for ASP.NET Core that makes logging easy. There are various sinks available for Serilog - for instance you get plain text, SQL and ElasticSearch sinks to name a few.

Why is ElasticSearch so popular?

Apart from the fact that logging is a requirement for just about every application, ElasticSearch solves a number of problems and does it really well:

  • It’s free and open source
    It’s free. Well almost. The basic features are free, well mostly. If you need security and alerting features in Kibana, you can buy the commercial X-pack subscription for Kibana, or you can install some open source alternatives.
  • RESTful API
    ElasticSearch has a RESTful API. Query results are returned in JSON which means results are easy to work with. Querying and inserting data via the RESTful API means it’s easy to use any programming language to work with ElasticSearch.
  • Easy to Query
    ElasticSearch has a built-in full-text search engine that is based on Apache Lucene. Compared to other databases, Lucene is easy to query. Even non-technical people would be able to write common queries.
  • It’s fast - very fast
    Querying a large SQL database can easily take 10 or 20 seconds. It’s quite common for similar queries on a large ElasticSearch database to return results in under 10 milliseconds.
  • It’s scalable
    It’s easy to scale. Combined with the fact that it's open source means it's easy on the wallet too.
  • Easy to Set Up
    Just fire up a docker compose file containing ElasticSearch and Kibana containers and you’re ready to start logging and searching.

Why do I need ElasticSearch and Kibana?

If you’ve ever built an application, you need logging. We all log errors, but how often are those error logs stored in a text file that is inaccessible somewhere on a server? ElasticSearch makes any kind of logging easy, accessible and searchable.

ElasticSearch’s incredible speed and simple query language coupled with Kibana’s interface and graphs make for a powerful 2 punch combo. If you’re not using ElasticSearch for logging yet, I highly suggest you start using it.

Enough with all the information. Let’s start coding.

I’ll use Visual Studio Code, an open source, cross-platform code editor. For the purposes of this tutorial, I’ll use Mac OSX, but you can use Ubuntu or Windows 10.

Prerequisites

To follow along, make sure you have the following installed:

Creating the directory structure for the project

mkdir elastic-kibana
cd elastic-kibana

Create a new MVC project with the .NET Core CLI

dotnet new mvc -n elastic-kibana -o src

Open the project in Visual Studio Code

cd elastic-kibana
code .

Creating a docker compose file

Next, create a docker compose file. This file will launch the ElasticSearch and Kibana containers and eliminates the need to run separate docker run commands for each container.

mkdir docker
cd docker

Create a new file named docker-compose.yml:

version: '3.1'

services:

  elasticsearch:
   image: docker.elastic.co/elasticsearch/elasticsearch:6.2.4
   container_name: elasticsearch
   ports:
    - "9200:9200"
   volumes:
    - elasticsearch-data:/usr/share/elasticsearch/data
   networks:
    - docker-network

  kibana:
   image: docker.elastic.co/kibana/kibana:6.2.4
   container_name: kibana
   ports:
    - "5601:5601"
   depends_on:
    - elasticsearch
   networks:
    - docker-network
  
networks:
  docker-network:
    driver: bridge

volumes:
  elasticsearch-data:

Then, run the docker compose command in the docker folder to spin up the containers.

docker-compose up -d

The first time you run the docker-compose command, it will download the images for ElasticSearch and Kibana from the docker registry, so it will take a few minutes depending on your connection speed.

Once you've run the docker-compose up command, check that ElasticSearch and Kibana are up and running.

ElasticSearch

Navigate to http://localhost:9200 to ensure ElasticSearch is up and running

Ensuring ElasticSearch is up and running - Logging with ElasticSearch, Kibana, ASP.NET Core and Docker
Ensuring ElasticSearch is up and running

Kibana

 Navigate to http://localhost:5601 to ensure Kibana is up and running

Ensuring Kibana is up and running - Logging with ElasticSearch, Kibana, ASP.NET Core and Docker
Ensuring Kibana is up and running

Adding Nuget Packages to the Project

We'll add the following Serilog packages to the project:

  • Serilog
  • Serilog.Sinks.ElasticSearch
  • Serilog.Extensions.Logging
cd ..
cd elastic-kibana
dotnet add package Serilog
dotnet add package Serilog.Sinks.ElasticSearch
dotnet add package Serilog.Extensions.Logging
dotnet restore

Add some configuration settings to appsettings.json

Add the default log settings and the ElasticSearch url to the appsettings.json file

{
  "Logging": {
    "LogLevel": {
        "Default": "Information",
        "System": "Information",
        "Microsoft": "Information"
    }
  },
  "ElasticConfiguration": {
    "Uri": "http://localhost:9200/"
  }
}

Configuring Logging in Startup.cs

Next, configure logging in Startup.cs.

Add the following using statements:

using Microsoft.Extensions.Logging;
using Serilog;
using Serilog.Sinks.Elasticsearch;

Next, configure the Startup constructor to load the elastic url from appsettings.json, and to configure the ElasticSearch sink.

public Startup(IConfiguration configuration, IHostingEnvironment hostingEnvironment)
{
    var builder = new ConfigurationBuilder()
        .SetBasePath(hostingEnvironment.ContentRootPath)
        .AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
        .AddJsonFile($"appsettings.{hostingEnvironment.EnvironmentName}.json", reloadOnChange: true, optional: true)
        .AddEnvironmentVariables();

    Configuration = builder.Build();

    var elasticUri = Configuration["ElasticConfiguration:Uri"];

    Log.Logger = new LoggerConfiguration()
        .Enrich.FromLogContext()
        .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri(elasticUri))
        {
            AutoRegisterTemplate = true,
        })
    .CreateLogger();
}

Finally, add Serilog to the logger factory in the Configure method.

public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
{
    /// ...
    
    loggerFactory.AddSerilog();
    
    /// ...
}

Start Logging Events to ElasticSearch

Now, run the MVC application by hitting f5 in Visual Studio code, or by typing dotnet run.

Run the MVC Application - Logging with ElasticSearch, Kibana, ASP.NET Core and Docker
Run the MVC Application

Launching Kibana

Since we configured logging in the startup class and set the minimum log level to Information, running the application would have logged a few events to ElasticSearch.

Let's open up Kibana at http://localhost:5601 so that we can view the logs.

Once Kibana loads, you'll be presented with the default page.

Launching Kibana - Logging with ElasticSearch, Kibana, ASP.NET Core and Docker
Launching Kibana

Create an Index Pattern in Kibana to Show Data

Kibana won't show any logs just yet. You have to specify an index before you can view the logged data. To do this, click the Discover link in the navigation, and then copy and paste the logstash index name which should be listed toward the bottom of the page into the index textbox as shown below, and click the next step button. Alternatively, you can use a wilcard such as logstash-*

Creating an Index Pattern in Kibana - Logging with ElasticSearch, Kibana, ASP.NET Core and Docker
Creating an Index Pattern in Kibana

Then, specify the time filter field name by selecting the @timestamp field and click the Create index pattern button.

Configuring a Time Filter Field in Elastic Search - Logging with ElasticSearch, Kibana, ASP.NET Core and Docker
Configuring a Time Filter Field in Elastic Search

You can now view the logs by clicking the Discover link in the navigation pane.

Viewing ElasticSearch Logs with Kibana - Logging with ElasticSearch, Kibana, ASP.NET Core and Docker
Viewing ElasticSearch Logs with Kibana

Logging Custom Messages from an MVC Controller

Since we specified that we want to log messages with a log level of information or higher, a number of information messages were logged by default. But what if we want to log our own messages? Thankfully, this is pretty easy to do. I'll log a message in the HomeController.

Add a using statement to HomeController.cs

using Microsoft.Extensions.Logging;

Then, inject an instance of ILogger with constructor injection.

ILogger<HomeController> _logger;
public HomeController(ILogger<HomeController> logger)
{
    _logger = logger;
}

And finally, log a message in the Index Action of the HomeController.

public IActionResult Index()
{
    _logger.LogInformation($"oh hai there! : {DateTime.UtcNow}");

    return View();
} 

Searching in Kibana

Now that we've logged a message, simply open up Kibana and search for the log message text.

Searching in Kibana - Logging with ElasticSearch, Kibana, ASP.NET Core and Docker
Searching in Kibana

You can also view the log as a single document in order to see which information was logged against various fields.

Kibana : View Single Document - Logging with ElasticSearch, Kibana, ASP.NET Core and Docker
Kibana : View Single Document

I'll show a few basic search examples you could use to demonstrate how easy it is to search in Kibana and how powerful ElasticSearch is:

message:"oh hai there"
level:"Information"
fields.ActionName:"elastic_kibana.Controllers.HomeController.Index"
(message:"oh hai there" AND fields.ActionName:"elastic_kibana.Controllers.HomeController.Index")

Logging Errors to ElasticSearch

A typical requirement is to log error messages. It doesn't get any simpler than logging with Serilog as shown below.

try
{
    throw new Exception("oops. i haz cause error in UR codez.");
}
catch (Exception ex)
{
    _logger.LogError(ex, "ur code iz buggy.");
}

Searching for Errors in Kibana

And it's dead simple to find errors in Kibana. I'll simply narrow it down to all logs with a level of error.

level: "Error"
Searching for Errors in Kibana - Logging with ElasticSearch, Kibana, ASP.NET Core and Docker
Searching for Errors in Kibana

Let's take a look at the default level of detail that is logged with Serilog and ElasticSearch.

Default Level of Detail in Error Messages with Serilog and ElasticSearch Sink - Logging with ElasticSearch, Kibana, ASP.NET Core and Docker
Default Level of Detail in Error Messages with Serilog and ElasticSearch Sink

It's pretty decent, but you'll notice that the exception detail is logged as one big string. Searching for information in this string would still return results, but if the information was logged against specific fields, we could perform more powerful and specific searches. Thankfully, there's a Serilog plugin called Serilog.Exceptions that can help us with that.

Installing the Serilog.Exceptions Nuget Package

Install the Serilog.Exceptions Nuget Package:

dotnet add package Serilog.Exceptions
dotnet restore

Then, add the Serilog.Exceptions using statement to the Startup.cs file.

using Serilog.Exceptions;

And enrich the Logger with the Serilog.Exceptions plugin

public Startup(IConfiguration configuration, IHostingEnvironment hostingEnvironment)
{
    /// ...

    Log.Logger = new LoggerConfiguration()
        .Enrich.FromLogContext()
        .Enrich.WithExceptionDetails()
        .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri(elasticUri))
        {
            AutoRegisterTemplate = true,
        })
    .CreateLogger();
}

Then, log a new error and search for the newly logged error in Kibana to see the more structured approach to logging with Serilog.

Error Logging with Serilog.Exceptions : A More Structured Approach - Logging with ElasticSearch, Kibana, ASP.NET Core and Docker
Error Logging with Serilog.Exceptions : A More Structured Approach

Turning Down Log Level Verbosity

You'll probably find the information level logs for a little bit too verbose for your taste. By default, ASP.NET Core will log hosting related events for Microsoft Kestrel server. This can get quite noisy. A simple way to exclude hosting related event is to adjust the appsettings file by setting the Microsoft log level to Warning. Optionally you could restrict logging even further by setting the minimum log level to Error for Default and System as shown below.

  "Logging": {
    "LogLevel": {
        "Default": "Error",
        "System": "Error",
        "Microsoft": "Warning"
    }
  }

Conclusion

Traditionally, logging required a lot of upfront work to get up and running. As such, logging is usually left out entirely or it's written to some obscure text file on a server that is difficult to access.

ElasticSearch and Kibana changed all of that. And Docker has made it effortless to get both ElasticSearch and Kibana up and running with no effort required. When you think about the powerful functionality that ElasticSearch and Kibana offers, and how performant it is, it's really quite impressive especially considering that it's open source.

Even without plugins like Serilog, logging has become a lot easier in ASP.NET Core compared to ASP.NET, so kudos to the .NET Core team for making an extensible logging framework. Serilog simply built on top of this and extended this functionality by making logging for .NET Core developers an even simpler process.

When you think about all the convenience and functionality you get by combining Docker, ElasticSearch, Kibana, ASP.NET Core and Serilog, there's really no excuse for developers to not incorporate logging into applications anymore.

Download the Source Code on GitHub

Download the source code at : https://github.com/thecarlo/elastic-kibana-netcore-serilog

Resources