Monitor .NET App Logs Using Elastic Stack

In this article we will learn How to Monitor .NET App Logs Using Elastic Stack (ELK) | Send .NET App logs to Elastic Stack. Monitoring is an important part of building and running reliable .NET applications. With the Elastic Stack (Elasticsearch, Logstash, and Kibana), developers can collect, store, and analyze logs in real time. This helps identify issues quickly and understand how the application is performing. In this guide, we’ll learn how to set up the Elastic Stack on an Ubuntu server and connect it to a .NET application using Serilog. We’ll also create simple API endpoints that generate logs for testing the setup.

Prerequisites

• AWS Account with Ubuntu 24.04 LTS EC2 Instance.
• At least 2 CPU cores and 4 GB of RAM for smooth performance.
• Java and .Net SDK installed.

Step #1:Setting Up Ubuntu EC2 Instance

Update the Package List to ensure you have the latest versions.

sudo apt update

Elasticsearch requires Java, so we need to install OpenJDK 17.

sudo apt install -y openjdk-17-jdk

Install the .NET 8.0 SDK, necessary for developing and running .NET applications.

sudo apt install -y dotnet-sdk-8.0

Step #2:Install and Configure Elasticsearch

Import the Elasticsearch GPG key.

curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg

Add the Elasticsearch repository.

echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list

Now lets update the package list again. The repository is added to the system’s package sources.

sudo apt update

Install Elasticsearch.

sudo apt install -y elasticsearch

Modify Elasticsearch configuration for remote access.

sudo nano /etc/elasticsearch/elasticsearch.yml

Find the network.host setting, uncomment it, and set it to 0.0.0.0 to bind to all available IP addresses and uncomment the discovery section to specify the initial nodes for cluster formation discovery.seed_hosts: []

For a basic setup (not recommended for production), disable security features.

xpack.security.enabled: false

Save and exit the editor.

Enable and start Elasticsearch.

sudo systemctl enable elasticsearch
sudo systemctl start elasticsearch

Check the status of the elasticsearch to ensure it is running.

sudo systemctl status elasticsearch

Send a GET request to check if Elasticsearch is running and responding. If successful, you should see a JSON response with cluster information.

curl -X GET "localhost:9200"

You can access it using browser with your Public IP address:9200 port which is a default port for Elasticsearch.

Step #3:Install and Configure Logstash

Logstash processes logs before sending them to Elasticsearch. Install it using following command.

sudo apt install -y logstash

Now create a Logstash pipeline config that listens for incoming HTTP log events.

sudo nano /etc/logstash/conf.d/dotnet.conf

Add the following configuration to collect logs from .Net Application, parse them, and send them to Elasticsearch.

input {
  http {
    port => 5000
    codec => json
  }
}

filter {
  json {
    source => "message"
  }

  date {
    match => ["@t", "ISO8601"]
    target => "@timestamp"
  }

  mutate {
    rename => {
      "@l" => "level"
      "@m" => "message"
      "@t" => "log_timestamp"
      "SourceContext" => "logger_name"
    }
    add_field => {
      "application" => "KibanaLoggingDemo"
    }
    remove_field => ["headers", "@version", "host"]
    uppercase => ["level"]
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "logstash-%{+YYYY.MM.dd}"
  }
}

Explanation of the code:

1. input Block
   • Purpose: Sets up an HTTP endpoint for receiving logs.
   • port => 5000: Logstash listens for incoming logs on port 5000.
   • codec => json: Tells Logstash to expect and decode incoming data as JSON.
2. filter Block
   • json filter
     • Parses the JSON string inside the message field (if it’s a nested JSON structure).
   • date filter
     • Parses the timestamp from the @t field (commonly used by Serilog) in ISO8601 format.
     • Sets @timestamp (used by Elasticsearch/Kibana for time-based indexing and dashboards).
   • mutate filter
     • rename: Renames fields to more readable or meaningful ones:
       • @l → level
       • @m → message
       • @t → log_timestamp
       • SourceContext → logger_name
     • add_field: Adds a custom field (application) for easier filtering in Kibana.
     • remove_field: Deletes metadata fields not needed in Elasticsearch.
     • uppercase: Converts log levels (like info, warn) to uppercase (INFO, WARN).
3. output Block
   • Sends processed logs to Elasticsearch.
   • hosts: Points to a local Elasticsearch instance.
   • index: Uses a daily index pattern like logstash-2025.04.10, useful for time-based log management in Kibana.

Enable and start Logstash

sudo systemctl enable logstash
sudo systemctl start logstash

Checks the status of Logstash.

sudo systemctl status logstash

Step #4:Install and Configure Kibana

Kibana provides visualization for Elasticsearch data. Install Kibana on the system.

sudo apt install -y kibana

Open the Kibana configuration file for editing.

sudo nano /etc/kibana/kibana.yml

Uncomment and adjust the following lines to bind Kibana to all IP addresses and connect it to Elasticsearch.

server.port: 5601
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://localhost:9200"]

Enable and start Kibana.

sudo systemctl enable kibana
sudo systemctl start kibana

Checks the status of Kibana.

sudo systemctl status kibana

Access the Kibana interface by navigating to http://<your-server-ip>:5601 in your web browser. Click on Explore on my own.

This will open the Kibana dashboard where you can start exploring your data.

Step #5:Configure .NET Application Logging

We’ll create a simple ASP.NET Core Web API project to demonstrate logging.

dotnet new webapi -n ElkLoggingDemo

Navigate to the project directory.

cd ElkLoggingDemo

Add Serilog and Logging Dependencies.

dotnet add package Serilog
dotnet add package Serilog.AspNetCore
dotnet add package Serilog.Sinks.Console
dotnet add package Serilog.Sinks.Elasticsearch
dotnet add package Serilog.Sinks.Http
dotnet add package Serilog.Formatting.Compact

These packages enable the Serilog to send the logs to the console, Elasticsearch, and over the HTTP (to Logstash).

Open Program.cs file.

sudo nano Program.cs

Replace it content with the following.

using System.Globalization;
using Microsoft.AspNetCore.Mvc;
using Serilog;
using Serilog.Sinks.Http;
using Serilog.Sinks.Elasticsearch;
using Microsoft.Extensions.Hosting;
var builder = WebApplication.CreateBuilder(args);

// Configure Serilog Early
Log.Logger = new LoggerConfiguration()
    .MinimumLevel.Information()
    .Enrich.FromLogContext()
    .WriteTo.Console()
    .WriteTo.Http(
        requestUri: "http://localhost:5000",
        queueLimitBytes: null
    )
    .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("http://localhost:9200"))
    {
        AutoRegisterTemplate = true,
        IndexFormat = "logstash-{0:yyyy.MM.dd}"
    })
    .CreateLogger();

builder.Host.UseSerilog(); // Use Serilog as the logging provider

// Add services to the container.
builder.Services.AddControllers();
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();

var app = builder.Build();

if (app.Environment.IsDevelopment())
{
    app.UseSwagger();
    app.UseSwaggerUI();
}

app.UseSerilogRequestLogging();
app.UseHttpsRedirection();
app.UseAuthorization();

app.MapControllers();

string HandleRollDice([FromServices] ILogger<Program> logger, string? player)
{
    var result = RollDice();

    if (string.IsNullOrEmpty(player))
    {
        logger.LogInformation("Anonymous player is rolling the dice: {result}", result);
    }
    else
    {
        logger.LogInformation("{player} is rolling the dice: {result}", player, result);
    }

    return result.ToString(CultureInfo.InvariantCulture);
}

int RollDice()
{
    return Random.Shared.Next(1, 7);
}

app.MapGet("/rolldice/{player?}", HandleRollDice);

Log.Information("Application starting up with Roll Dice Endpoint");
app.Run();

Explanation of the code:

1. Namespaces
   • Brings in necessary namespaces for:
     • Globalization (for formatting)
     • ASP.NET Core MVC attributes
     • Serilog (logging)
     • HTTP and Elasticsearch sinks for Serilog
     • Hosting support
2. Serilog Configuration
   • MinimumLevel.Information(): Captures logs from Information and above.
   • Enrich.FromLogContext(): Adds contextual data like request ID, etc.
   • WriteTo.Console(): Outputs logs to the terminal.
   • WriteTo.Http(…):
     • Sends logs to a Logstash instance listening on port 5000 via HTTP.
   • WriteTo.Elasticsearch(…):
     • Sends logs directly to Elasticsearch.
     • IndexFormat: Creates daily indexes (e.g., logstash-2025.04.10)
     • AutoRegisterTemplate: Automatically registers index mapping in Elasticsearch.
3. ASP.NET Core App Configuration
   • Integrates Serilog into the ASP.NET Core logging pipeline.
   • Sets up MVC controllers and Swagger for API documentation.
4. Middleware and App Start
   • Swagger UI is enabled only in development mode.
5. Custom Endpoint: /rolldice/{player?}
   • This defines a lightweight GET endpoint at /rolldice/{player?}.
   • Accepts an optional player parameter.
   • Logs who rolled the dice and the result.
   • Returns the dice result as a string.
6. Application Startup
   • Logs a startup message.
   • Starts the application.

This is optional since we configured Serilog in code, but you can also define it in appsettings.json for flexibility. So open the appsettings.json file.

sudo nano appsettings.json

Replace its content with the following.

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "AllowedHosts": "*",
  "Serilog": {
    "Using": [ "Serilog.Sinks.Console", "Serilog.Sinks.Elasticsearch" ],
    "MinimumLevel": {
      "Default": "Information",
      "Override": {
        "Microsoft": "Warning",
        "System": "Warning"
      }
    },
    "WriteTo": [
      { "Name": "Console" },
      {
        "Name": "Elasticsearch",
        "Args": {
          "nodeUris": "http://localhost:9200",
          "indexFormat": "logstash-{0:yyyy.MM.dd}",
          "autoRegisterTemplate": true,
          "templateName": "dotnet-logs-template"
        }
      }
    ],
    "Enrich": [ "FromLogContext" ],
    "Properties": {
      "Application": "KibanaLoggingDemo"
    }
  }
}

Explanation of the code:

1. Logging
   • Sets log levels for built-in ASP.NET Core logging system.
   • "Default": "Information" — Logs information and above.
   • "Microsoft.AspNetCore": "Warning" — Reduces noise from framework logs.
2. AllowedHosts
   • Accepts requests from any host. Used primarily for security in production environments.
   • "*" = Allow all (used in dev/test)
3. Serilog
   • Configuration for Serilog, a powerful .NET logging library.
4. Using
   • Tells Serilog which sinks (outputs) are being used.
   • Console: logs to terminal
   • Elasticsearch: sends logs to an Elasticsearch cluster
5. MinimumLevel
   • Controls log verbosity:
     • Default: Information and above
     • Override: Suppress verbose logs from Microsoft.* and System.*
6. WriteTo
   • Sinks (outputs) for Serilog.
   • Console: sends logs to terminal
   • Elasticsearch sink:
     • nodeUris: location of the Elasticsearch cluster
     • indexFormat: daily index naming (e.g., logstash-2025.04.10)
     • autoRegisterTemplate: allows Serilog to create an index mapping template
     • templateName: name of the template in Elasticsearch
7. Enrich
   • Adds contextual properties (like request ID, user info) to each log entry.
8. Properties
   • Adds static properties to every log, in this case:
   • "Application": "KibanaLoggingDemo" — great for filtering in Kibana.

Create a Controllers directory.

mkdir Controllers

Create a controller to simulate different log levels and errors.

nano Controllers/LoggingController.cs

Add the following code into it.

using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Logging;

namespace ElkLoggingDemo.Controllers;

[ApiController]
[Route("[controller]")]
public class LoggingController : ControllerBase
{
    private readonly ILogger<LoggingController> _logger;

    public LoggingController(ILogger<LoggingController> logger)
    {
        _logger = logger;
    }

    [HttpGet]
    public IActionResult Get()
    {
        _logger.LogInformation("This is an information log message");
        _logger.LogWarning("This is a warning log message");
        _logger.LogError("This is an error log message");

        return Ok(new {
            Message = "Logs generated successfully!",
            Timestamp = DateTime.UtcNow
        });
    }

    [HttpGet("test-exception")]
    public IActionResult TestException()
    {
        try
        {
            throw new Exception("This is a test exception");
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "An exception occurred in TestException");
            return StatusCode(500, new {
                Error = ex.Message,
                StackTrace = ex.StackTrace
            });
        }
    }
}

Explanation of the code:

1. Namespace & Imports
   • Microsoft.AspNetCore.Mvc: Provides attributes and types for defining Web API endpoints.
   • Microsoft.Extensions.Logging: Allows dependency injection of the ILogger<T> interface.
   • ElkLoggingDemo.Controllers: Custom namespace for this app, indicating it’s part of an ELK logging demo.
2. Controller Setup
   • [ApiController]: Enables automatic model validation, better error responses, and route binding.
   • [Route("[controller]")]: Sets the base route to /logging (it uses the controller name minus “Controller”).
   • Inherits from ControllerBase, a lightweight controller without view support.
3. Constructor Injection
   • Injects an instance of ILogger<LoggingController> via constructor.
   • This logger can be configured to send logs to Console, Elasticsearch, or any sink supported by Serilog or the default logger.
4. GET /logging
   • Sends three log entries: Info, Warning, and Error
   • Returns HTTP 200 with a simple message and UTC timestamp.
   • Great for testing log pipeline (Logstash/Elasticsearch ingestion).
5. GET /logging/test-exception
   • Simulates an unhandled exception
   • Catches it and logs it as an Error with exception details
   • Returns HTTP 500 with the exception message and stack trace in the response

Step #6:Run and Test the App

Build the application.

dotnet build

Run the application.

dotnet run

Access the application.

curl http://localhost:5291/rolldice/john

Step #7:Visualizing .Net App Logs in Kibana

Go to Menu bar from top-left corner and select Stack Management under the management section.

Under “Kibana” section, click on “Data views”.

Click on “Create data view”.

Enter logstash-* (the index name used in Logstash output) in the Index pattern name field. You can give any Name you wan’t like .Net logs and click on Save data view to Kibana.

Scroll down and click on the Logs option in Obeservability in the left-hand navigation menu. If the menu is collapsed, click the Expand icon to reveal the options.

Go to All logs as shown below.

Next go to the Data Views.

Select .Net logs as a data view.

Kibana displays .Net App logs data from the last 15 minutes, visualized as a histogram along with individual log messages below. (You may need to adjust the time range.)

• System logs (top 7): Tagged with "application": "KibanaLoggingDemo" likely formatted by Logstash and Come via Serilog HTTP Sink → Logstash → Elasticsearch.
• .Net App logs (bottom 7): Contain your app’s internal logging info, raw messages like "Request starting HTTP...". They are emitted by ASP.NET Core middleware or controllers.

Conclusion:

With the Elastic Stack and Serilog integration, monitoring your .NET application becomes easy and powerful. You can collect logs, search through them, and visualize patterns or errors using Kibana. This setup helps in debugging, improving performance, and understanding user behavior. It’s also flexible and can be scaled as your application grows. Once configured, it saves time and effort in managing logs.

Related Articles:

Monitor Kafka logs using Elastic Stack

Send Java Gradle App Logs to Elastic Stack

Send Java Maven App Logs to Elastic Stack

Reference:

Elastic Stack official page