Send Alerts to Slack Using Elastic Stack

In this article, we will learn Send Alerts to Slack Using Elastic Stack/how to Send Slack Alerts Using Elastic Stack. Monitoring systems in real time is important for keeping your applications healthy and secure. The Elastic Stack helps you collect and analyze logs easily. But just viewing logs is not enough, it’s important to get alerts when something goes wrong. One of the best ways to receive alerts is through Slack, where your team is already communicating. Today, we’ll show you how to connect Kibana to Slack so you can get instant alerts in your channel when issues occur. It’s quick, easy, and helps your team respond faster.

Prerequisites

• AWS Account with Ubuntu 24.04 LTS EC2 Instance.
• At least 2 CPU cores and 4 GB of RAM for smooth performance.
• Docker and Docker Compose installed.

Step #1:Install Docker and Docker Compose

First update the package list.

sudo apt update

If Docker and Docker Compose are not already installed, you can install them using the following command.

sudo apt install -y docker.io docker-compose

• docker.io: Installs the Docker engine.
• docker-compose: Installs Docker Compose for managing multi-container applications.

Step #2:Create Docker Compose File

Create a Docker Compose YAML file to define the Elastic Stack services.

sudo nano docker-compose.yml

Paste the following content into the file.

version: "3"
services:
  elasticsearch:
    image: "docker.elastic.co/elasticsearch/elasticsearch:8.17.2"
    environment:
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
      - "discovery.type=single-node"
    ports:
      - "9200:9200"
    volumes:
      - elasticsearch_data:/usr/share/elasticsearch/data

  kibana:
    image: "docker.elastic.co/kibana/kibana:8.17.2"
    ports:
      - "5601:5601"

  filebeat:
    image: "docker.elastic.co/beats/filebeat:8.17.2"
    user: root
    volumes:
      - /MY_WORKDIR/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
      - /var/lib/docker:/var/lib/docker:ro
      - /var/run/docker.sock:/var/run/docker.sock

  metricbeat:
    image: docker.elastic.co/beats/metricbeat:8.17.2
    user: root
    volumes:
      - /MY_WORKDIR/metricbeat.yml:/usr/share/metricbeat/metricbeat.yml:ro
      - /var/run/docker.sock:/var/run/docker.sock
      - /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
      - /proc:/hostfs/proc:ro
      - /:/hostfs:ro

volumes:
  elasticsearch_data:

Step #3:Prepare Configuration Files For Beats

Create a working directory for your configuration files.

sudo mkdir /MY_WORKDIR

Inside the directory create a filebeat.yml file.

sudo nano /MY_WORKDIR/filebeat.yml

Paste the following config in it.

filebeat.inputs:
- type: container
  paths:
    - '/var/lib/docker/containers/*/*.log'

processors:
- add_docker_metadata:
    host: "unix:///var/run/docker.sock"

- decode_json_fields:
    fields: ["message"]
    target: "json"
    overwrite_keys: true

output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  indices:
    - index: "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"

logging.json: true
logging.metrics.enabled: false

Create a metricbeat.yml file.

sudo nano /MY_WORKDIR/metricbeat.yml

Paste the following config in it.

metricbeat.modules:
- module: system
  metricsets:
    - cpu
    - memory
    - network
    - filesystem
    - process
    - process_summary
  enabled: true
  period: 10s
  processes: ['.*']
  hostfs: "/hostfs"

- module: docker
  metricsets:
    - container
    - cpu
    - diskio
    - healthcheck
    - info
    - memory
    - network
  hosts: ["unix:///var/run/docker.sock"]
  period: 10s
  enabled: true

output.elasticsearch:
  hosts: ["http://elasticsearch:9200"]

monitoring:
  enabled: true

Step #4:Start the Elastic Stack using Docker Composer

Start the services using Docker Compose.

sudo docker-compose up -d

Verify that the containers are running.

sudo docker ps -a

Step #5:Set Password for Elastic User

Elasticsearch 8.x enables security by default. Reset the password for the elastic user using following command.

sudo docker exec -it ubuntu_elasticsearch_1 /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic -i

Update the Filebeat configuration to include the Elasticsearch credentials.

sudo nano /MY_WORKDIR/filebeat.yml

Add the username and password fields under output.elasticsearch.

output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  username: "elastic"
  password: "devopshint"
  indices:
    - index: "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"

Save and exit the file.

Update the Metricbeat configuration to include the Elasticsearch credentials.

sudo nano /MY_WORKDIR/metricbeat.yml

Add the username and password fields under output.elasticsearch.

output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  username: "elastic"
  password: "devopshint"

Restart the containers. First shut down the containers.

sudo docker-compose down

Then start them again.

sudo docker-compose up -d

Step #6:Secure Kibana with Service Token

To securely connect Kibana to Elasticsearch, generate a service account token.

curl -u elastic:devopshint -X POST "http://localhost:9200/_security/service/elastic/kibana/credential/token/my-token"; echo

Save the value field from the response.

Generate an encryption key.

openssl rand -hex 32

Save the generated encryption key.

Update the docker-compose.yml file to include the service account token and encryption keys.

sudo nano docker-compose.yml

Add the following environment variables to the kibana service.

kibana:
  image: "docker.elastic.co/kibana/kibana:8.17.2"
  ports:
    - "5601:5601"
  environment:
    - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    - ELASTICSEARCH_SERVICEACCOUNTTOKEN=AAEAAWVsYXN0aWMva2liYW5hL215LXRva2VuOnEwby1YUGlyU1JhZnFxaEozS012aVE
    - XPACK_SECURITY_ENCRYPTIONKEY=2145ee3c3cc4e7853c77e96a7cd7e0fe102ea5ffa948e368132b525af018db0c
    - XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY=2145ee3c3cc4e7853c77e96a7cd7e0fe102ea5ffa948e368132b525af018db0c
    - XPACK_REPORTING_ENCRYPTIONKEY=2145ee3c3cc4e7853c77e96a7cd7e0fe102ea5ffa948e368132b525af018db0c
    - XPACK_SECURITY_SECURE_COOKIES=true
  depends_on:
    - elasticsearch

Save and exit the file.

Restart the services.

sudo docker-compose down

sudo docker-compose up -d

Verify that the containers are running.

sudo docker ps -a

Step #7:Access Kibana Observability

Open your browser and navigate to http://<EC2-PUBLIC-IP>:5601. Log in using the elastic username and the password you set earlier.

Click on Explore on my own.

Go to Observability.

You will see the Overview there.

Now lets go to Infrastructure Inventory to see the metrics.

Step #8:Create an Alert

Next lets set up alert for that go to Alerts as shown below and click on Manage Rules.

Click on Create rule to create a rule for alert.

We will create a Metric threshold rule.

Write a Name for your rule like Slack Alert. As conditions as

• WHEN Average
• OF system.cpu.total.pct
• IS ABOVE 0.9 %

And click on Save.

You can see the rule created successfully. Now go back to Alerts.

As you can see below alert is triggered.

Step #9:Create a Slack Channel

Open your Slack workspace and Add channels – Create a new channel.

Create a Blank channel.

Enter the channel name like elasticalerts and Change Visibility to Private. Then click on Create to create a channel.

As you can see our channel is created.

Step #10:Configure Slack Alert in Kibana

Now go to Apps – Manage – Browse apps.

Search for Incoming Webhooks and click on it.

Click on Add to Slack.

Select the channel, here it is elasticalert and click on Add Incoming Webhooks integration.

Copy the Webhook URL. We will use it in Kibana Connectors.

Now go back to kibana dashboard and select the Stack Management.

Go to the Connectors.

Click on Create connector.

To set up an Slack connector you need to upgrade your license so go to Manage license.

Select Start trial to Start a 30-day trial.

Click on Start my trial.

Now you can see Our Trial license is active.

Now again go back to connectors and select Slack as a connector.

Enter the Connector name like Slack Alert Connector and Enter the Webhook URL which we have copied. Click on Save and Test.

You need to enter the message CRITICAL: High CPU Usage !! then Run the Test and you will see the Test was successful message.

You can see the Alert Message in slack.

Now go back to Slack Alert and go to Actions. Click on Edit rule.

Scroll down and select Slack as an Action.

Click on Save.

You can see in below image our action has been added.

Now when alert is triggered you will get the message in format specified in Slack Alert Connector. Shown below.

Conclusion:

Sending alerts to Slack using the Elastic Stack is a great way to keep your team informed and ready to take action. By creating a Slack channel, setting up a webhook, and linking it to Kibana, you can get real-time notifications about errors or unusual activity. This setup improves your response time and makes your monitoring system more useful. It’s simple to configure and works well with your existing workflow. Try it out to make your alerting smarter and your team more connected.

Related Articles:

Send Email Alerts Using Elastic Stack

Observability Dashboard Overview in Elastic Stack

Observability Dashboard Overview in Elastic Stack (Logs and Infrastructure) – Part 2

Reference:

Elastic Stack official page