Secure Monitoring environment setup using Prometheus AlertManager and Grafana

In this tutorial I am going to demonstrate to setup end to end Monitoring environment using Prometheus AlertManager and Grafana, setup TLS/SSL for alertmanager.

Prerequisites

You need have one Linux servers installed with internet connectivity. I will be using Ubuntu 18.04 server for this demonstration.

IP – 192.168.56.107

Hostname – ubuntu-server1/ubuntu-server1.example.com

OS version – Ubuntu 18.04.4 LTS

Before we start practical demonstrations lets first understand what are these technologies.

What is Prometheus?

As per official documentation:

Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud. Since its inception in 2012, many companies and organizations have adopted Prometheus, and the project has a very active developer and user community. It is now a standalone open source project and maintained independently of any company.

Prometheus is a monitoring tool which uses time-series database to record the various real time metrices. It is open-source and written in Go language. The Prometheus metrics are collected using HTTP pulls, allowing for higher performance and scalability.

Main Features of Prometheus

Prometheus’s main features are:

• A multi-dimensional data model with time series data identified by metric name and key/value pairs
• No reliance on distributed storage; single server nodes are autonomous
• Time series collection happens via a pull model over HTTP
• Pushing time series is supported via an intermediary gateway
• Targets are discovered via service discovery or static configuration
• Multiple modes of graphing and dashboarding support

Following are the tools which makes Prometheus complete monitoring solution:

• Exporters: Libraries that help with exporting metrics from third-party systems as Prometheus metrics.
• PromQL: Also known as Prometheus Query Language, this allows you to filter multi-dimensional time series data.

What is AlertManager ?

The Alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integrations such as email, PagerDuty, or MS Teams. It also takes care of silencing and inhibition of alerts.

What is Grafana ?

Grafana is open source visualization and analytics software. It allows you to query, visualize, alert on, and explore your metrics no matter where they are stored. In plain English, it provides you with tools to turn your time-series database (TSDB) data into beautiful graphs and visualizations.

Prometheus Installation

Follow the step by step procedure described below to install Prometheus.

Update package repositories

root@ubuntu-server1:~# apt-get update

Install few required packages

root@ubuntu-server1:~# apt-get -y install wget curl vim

Create Prometheus system group

root@ubuntu-server1:~# groupadd --system prometheus

As a Linux user you should know that the GID < 1000 is a system group.

Create Prometheus system user and assign primary group created in last step

root@ubuntu-server1:~# useradd -s /sbin/nologin --system -g prometheus prometheus

Create data & configs directories for Prometheus

Prometheus stores its data on local system. I will need some directory to do the same. We will create this under /var/lib/prometheus.

root@ubuntu-server1:~# mkdir /var/lib/prometheus

Main configuration files of Prometheus will reside under /etc/prometheus/ and it will have some sub-directories as well:

root@ubuntu-server1:~# mkdir -p /etc/prometheus/{rules,rules.d,files_sd}
root@ubuntu-server1:~# cd /etc/prometheus/
root@ubuntu-server1:/etc/prometheus# ls
files_sd  rules  rules.d

Download Prometheus binary

We will be downloading the latest release of Prometheus archive and extract it to get binary files. Latest releases can be found at – Prometheus releases Github page or https://prometheus.io/download/.

root@ubuntu-server1:/opt# wget https://github.com/prometheus/prometheus/releases/download/v2.22.1/prometheus-2.22.1.linux-amd64.tar.gz

or to always download without mentioning the exact release/version number use following command –

root@ubuntu-server1:/opt# curl -s https://api.github.com/repos/prometheus/prometheus/releases/latest | grep browser_download_url | grep linux-amd64 | cut -d '"' -f 4 | wget -qi -

Extract the Prometheus binary file

root@ubuntu-server1:/opt# tar -xvzf prometheus-2.22.1.linux-amd64.tar.gz

root@ubuntu-server1:/opt# ls
app.py  app.pyc  prometheus-2.22.1.linux-amd64  prometheus-2.22.1.linux-amd64.tar.gz  VBoxGuestAdditions-6.1.10

Move binary files to /usr/local/bin/ directory

root@ubuntu-server1:/opt/prometheus-2.22.1.linux-amd64# ls
console_libraries  consoles  LICENSE  NOTICE  prometheus  prometheus.yml  promtool
root@ubuntu-server1:/opt/prometheus-2.22.1.linux-amd64# mv prometheus promtool /usr/local/bin/

That’s all! You are done with installation.

Verify the installed version

root@ubuntu-server1:~# prometheus --version
prometheus, version 2.22.1 (branch: HEAD, revision: 00f16d1ac3a4c94561e5133b821d8e4d9ef78ec2)
  build user:       root@516b109b1732
  build date:       20201105-14:02:25
  go version:       go1.15.3
  platform:         linux/amd64

Move Prometheus template file and consoles and console_libraries to /etc/prometheus directory:*

root@ubuntu-server1:/opt/prometheus-2.22.1.linux-amd64# mv prometheus.yml /etc/prometheus/prometheus.yml
root@ubuntu-server1:/opt/prometheus-2.22.1.linux-amd64# mv consoles/ console_libraries/ /etc/prometheus/
root@ubuntu-server1:/opt/prometheus-2.22.1.linux-amd64# cd /etc/prometheus/
root@ubuntu-server1:/etc/prometheus# ls
console_libraries  consoles  files_sd  prometheus.yml  rules  rules.d

Prometheus Configuration

It’s time to configure Prometheus. Open and start editing the configuration file for Prometheus – /etc/prometheus/prometheus.yml.

Here is default Prometheus template file. The file is written in YAML format.

root@ubuntu-server1:/etc/prometheus# cat prometheus.yml
# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']

This file can be configured as per our requirements. For now we will leave it as is.

Lets now understand the parameters of this file to get a clear cut understanding of Prometheus.

global The global configuration specifies parameters that are valid in all other configuration contexts. They also serve as defaults for other configuration sections.

scrape_configs

A scrape_config section specifies a set of targets and parameters describing how to scrape them. In the general case, one scrape configuration specifies a single job. In advanced configurations, this may change.

Targets may be statically configured via the static_configs parameter or dynamically discovered using one of the supported service-discovery mechanisms.

alerting

Alerting specifies settings related to the Alertmanager.

For now this is enough to cover the default template file. We will explore other parameters later in this tutorial.

Create a Prometheus systemd Service unit file

root@ubuntu-server1:/etc/prometheus# cat /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus
Documentation=https://prometheus.io/docs/introduction/overview/
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/local/bin/prometheus   --config.file=/etc/prometheus/prometheus.yml   --storage.tsdb.path=/var/lib/prometheus   --web.console.templates=/etc/prometheus/consoles   --web.console.libraries=/etc/prometheus/console_libraries   --web.listen-address=0.0.0.0:9090   --web.external-url=

SyslogIdentifier=prometheus
Restart=always

[Install]
WantedBy=multi-user.target

Change directory permissions

Change the ownership of Prometheus data and config directories to Prometheus system user and group.

root@ubuntu-server1:/etc/prometheus# for i in rules rules.d files_sd; do sudo chown -R prometheus:prometheus /etc/prometheus/${i}; done
root@ubuntu-server1:/etc/prometheus# for i in rules rules.d files_sd; do sudo chmod -R 775 /etc/prometheus/${i}; done
root@ubuntu-server1:/etc/prometheus# chown -R prometheus:prometheus /var/lib/prometheus/

Start Prometheus Service

First reload the systemd daemon:

root@ubuntu-server1:/etc/prometheus# systemctl daemon-reload

Start and enable the service.

root@ubuntu-server1:/etc/prometheus# systemctl start prometheus
root@ubuntu-server1:/etc/prometheus# systemctl enable prometheus
Created symlink /etc/systemd/system/multi-user.target.wants/prometheus.service → /etc/systemd/system/prometheus.service.

Check the status of service running.

root@ubuntu-server1:/etc/prometheus# systemctl status prometheus.service
● prometheus.service - Prometheus
   Loaded: loaded (/etc/systemd/system/prometheus.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2020-11-10 10:31:55 UTC; 36s ago
     Docs: https://prometheus.io/docs/introduction/overview/
 Main PID: 13564 (prometheus)
    Tasks: 8 (limit: 2344)
   CGroup: /system.slice/prometheus.service
           └─13564 /usr/local/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus --web.console

Nov 10 10:31:55 ubuntu-server1 prometheus[13564]: level=info ts=2020-11-10T10:31:55.421Z caller=head.go:642 component=tsdb msg="Replaying on-di
Nov 10 10:31:55 ubuntu-server1 prometheus[13564]: level=info ts=2020-11-10T10:31:55.421Z caller=head.go:656 component=tsdb msg="On-disk memory
Nov 10 10:31:55 ubuntu-server1 prometheus[13564]: level=info ts=2020-11-10T10:31:55.421Z caller=head.go:662 component=tsdb msg="Replaying WAL,
Nov 10 10:31:55 ubuntu-server1 prometheus[13564]: level=info ts=2020-11-10T10:31:55.421Z caller=head.go:714 component=tsdb msg="WAL segment loa
Nov 10 10:31:55 ubuntu-server1 prometheus[13564]: level=info ts=2020-11-10T10:31:55.421Z caller=head.go:719 component=tsdb msg="WAL replay comp
Nov 10 10:31:55 ubuntu-server1 prometheus[13564]: level=info ts=2020-11-10T10:31:55.422Z caller=main.go:732 fs_type=EXT4_SUPER_MAGIC
Nov 10 10:31:55 ubuntu-server1 prometheus[13564]: level=info ts=2020-11-10T10:31:55.422Z caller=main.go:735 msg="TSDB started"
Nov 10 10:31:55 ubuntu-server1 prometheus[13564]: level=info ts=2020-11-10T10:31:55.422Z caller=main.go:861 msg="Loading configuration file" fi
Nov 10 10:31:55 ubuntu-server1 prometheus[13564]: level=info ts=2020-11-10T10:31:55.423Z caller=main.go:892 msg="Completed loading of configura
Nov 10 10:31:55 ubuntu-server1 prometheus[13564]: level=info ts=2020-11-10T10:31:55.423Z caller=main.go:684 msg="Server is ready to receive web

Enable firewall ports

If your ubuntu server has is configured to use firewall service, you’ll need to open port 9090.

root@ubuntu-server1:/etc/prometheus# ufw allow 9090/tcp
Rules updated
Rules updated (v6)

Access the Prometheus UI at port 9090

Configure Prometheus with secure reverse proxy

Install nginx and configure it.

root@ubuntu-server1:/etc/prometheus# apt-get install nginx

Now generate self-signed certificates. To do this, log into your server and issue the following command:

root@ubuntu-server1:/etc/nginx/ssl# openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/nginx/ssl/ubuntu-server1.key -out /etc/nginx/ssl/ubuntu-server1.pem
Can't load /root/.rnd into RNG
140342748676544:error:2406F079:random number generator:RAND_load_file:Cannot open file:../crypto/rand/randfile.c:88:Filename=/root/.rnd
Generating a RSA private key
..............................................+++++
.................................+++++
writing new private key to '/etc/nginx/ssl/ubuntu-server1.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:IN
State or Province Name (full name) [Some-State]:Karnataka
Locality Name (eg, city) []:Bangalore
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Fosstechnix
Organizational Unit Name (eg, section) []:DevOps
Common Name (e.g. server FQDN or YOUR name) []:ubuntu-server1
Email Address []:

Create nginx configuration file for Prometheus.

root@ubuntu-server1:/etc/nginx/sites-enabled# pwd
/etc/nginx/sites-enabled

root@ubuntu-server1:/etc/nginx/sites-enabled# cat ubuntu-server1.conf
upstream ubuntu-server1 {
server localhost:9090;
}
server {
server_name ubuntu-server1.example.com;
listen 80;
access_log /var/log/nginx/sites/$server_name.http.log;
rewrite ^ https://$server_name$request_uri? permanent;
}
server {
server_name ubuntu-server1.example.com;
listen 443 ssl;
access_log /var/log/nginx/sites/$server_name.https.log;
ssl_certificate /etc/nginx/ssl/ubuntu-server1.pem;
ssl_certificate_key /etc/nginx/ssl/ubuntu-server1.key;
charset utf-8;
client_max_body_size 16m;
keepalive_timeout 70;
client_body_timeout 10;
client_header_timeout 10;
client_header_buffer_size 128;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://ubuntu-server1;
}
}

Update Nginx main configuration file at the end to include files under sites-enabled directory.

root@ubuntu-server1:/etc/nginx/sites-enabled# cat /etc/nginx/nginx.conf
include /etc/nginx/sites-enabled/*.conf;

Allow though ufw

root@ubuntu-server1:/etc/nginx/sites-enabled# ufw allow 'Apache Full'
root@ubuntu-server1:/etc/nginx/sites-enabled# ufw allow 'Apache Secure'

root@ubuntu-server1:/etc/nginx/sites-enabled# ufw status
Status: active

To                         Action      From
--                         ------      ----
9090/tcp                   ALLOW       Anywhere
Apache Full                ALLOW       Anywhere
Apache Secure              ALLOW       Anywhere
9090/tcp (v6)              ALLOW       Anywhere (v6)
Apache Full (v6)           ALLOW       Anywhere (v6)
Apache Secure (v6)         ALLOW       Anywhere (v6)

Verify the Nginx config and restart the service.

root@ubuntu-server1:/etc/nginx/sites-enabled# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

root@ubuntu-server1:/etc/nginx/sites-enabled# systemctl restart nginx

root@ubuntu-server1:/etc/nginx/sites-enabled# journalctl -f -u nginx.service
-- Logs begin at Thu 2020-04-02 09:59:29 UTC. --
Nov 10 11:09:27 ubuntu-server1 systemd[1]: Started nginx - high performance web server.
Nov 10 11:10:05 ubuntu-server1 systemd[1]: Stopping nginx - high performance web server...
Nov 10 11:10:05 ubuntu-server1 systemd[1]: Stopped nginx - high performance web server.
Nov 10 11:10:05 ubuntu-server1 systemd[1]: Starting nginx - high performance web server...
Nov 10 11:10:05 ubuntu-server1 systemd[1]: Started nginx - high performance web server.
Nov 10 11:15:58 ubuntu-server1 systemd[1]: Stopping nginx - high performance web server...
Nov 10 11:15:58 ubuntu-server1 systemd[1]: Stopped nginx - high performance web server.
Nov 10 11:15:58 ubuntu-server1 systemd[1]: Starting nginx - high performance web server...
Nov 10 11:15:58 ubuntu-server1 systemd[1]: nginx.service: Can't open PID file /var/run/nginx.pid (yet?) after start: No such file or directory
Nov 10 11:15:58 ubuntu-server1 systemd[1]: Started nginx - high performance web server.

Verify on the browser – Access it at https://192.168.56.107/ or https://ubuntu-server1.example.com/

Proceed to access the server.

The https icon red out and crossed means it isn’t signed by one of the root CA certificates that the browser implicitly trusts and that’s fine.

Till now we are done with configuring Prometheus.

Install and Configure Alert Manager

In this part, we are going to take a look at the AlertManager installation and configuration with Prometheus.

Alertmanager runs as a standalone server aside from Prometheus though its part of the Prometheus stack.

AlertManager also will be configured as one of the targets with Prometheus configuration and Prometheus will take care of sending alerts directly to the AlertManager.

In case you use different clients then Prometheus itself, the AlertManager exposes a set of REST endpoints that you can use to fire alerts to users via various mediums.

Follow the step by step procedure described below to install AlertManager.

Download the AlertManager

root@ubuntu-server1:/opt# curl -s https://api.github.com/repos/prometheus/alertmanager/releases/latest | grep browser_download_url | grep linux-amd64 | cut -d '"' -f 4 | wget -qi -

root@ubuntu-server1:/opt# ls
alertmanager-0.21.0.linux-amd64.tar.gz  prometheus-2.22.1.linux-amd64.tar.gz  prometheus-2.22.1.linux-amd64

Extract the tarball

root@ubuntu-server1:/opt# tar -xvzf alertmanager-0.21.0.linux-amd64.tar.gz

root@ubuntu-server1:/opt# cd alertmanager-0.21.0.linux-amd64
root@ubuntu-server1:/opt/alertmanager-0.21.0.linux-amd64# ls
alertmanager  alertmanager.yml  amtool  LICENSE  NOTICE

In the folder where you extracted your files, you should find the following entries:

• amtool: amtool binary allows you to view or to modify the current state of the AlertManager and helps in silence alerts, expire silences, as well as import silences or query them.
• alertmanager: the executable for the alertmanager. This is the executable that you will run in order to start an AlertManager server on your instance.
• alertmanager.yml: This is the configuration file for the AlertManager and defines some example routes, but I will create our own alert file.

Move binary files to /usr/local/bin/ directory

root@ubuntu-server1:/opt/alertmanager-0.21.0.linux-amd64# mv amtool alertmanager /usr/local/bin/

Verify the installed version

root@ubuntu-server1:/opt/alertmanager-0.21.0.linux-amd64# alertmanager --version
alertmanager, version 0.21.0 (branch: HEAD, revision: 4c6c03ebfe21009c546e4d1e9b92c371d67c021d)
  build user:       root@dee35927357f
  build date:       20200617-08:54:02
  go version:       go1.14.4

Alertmanager Configuration

Create a user for our AlertManager service

root@ubuntu-server1:~# useradd -rs /bin/false alertmanager

Create AlertManager data and config directories

root@ubuntu-server1:~# mkdir -p /etc/alertmanager
root@ubuntu-server1:~# mkdir -p /data/alertmanager

Move AlertManager template config file to config directory created above

root@ubuntu-server1:/opt/alertmanager-0.21.0.linux-amd64# cp alertmanager.yml /etc/alertmanager/

Change the ownerships of above directories

root@ubuntu-server1:~# chown -R alertmanager:alertmanager /data/alertmanager /etc/alertmanager/*

Create an AlertManager systemd Service unit file

root@ubuntu-server1:/etc/prometheus# cat /etc/systemd/system/alertmanager.service
[Unit]
Description=Alert Manager
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=alertmanager
Group=alertmanager
ExecStart=/usr/local/bin/alertmanager \
  --config.file=/etc/alertmanager/alertmanager.yml \
  --storage.path=/data/alertmanager

Restart=always

[Install]
WantedBy=multi-user.target

Start and enable the alertmanager service

root@ubuntu-server1:~# systemctl start alertmanager.service
root@ubuntu-server1:~# systemctl status alertmanager.service
● alertmanager.service - Alert Manager
   Loaded: loaded (/etc/systemd/system/alertmanager.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2020-11-11 05:29:46 UTC; 4s ago
 Main PID: 14680 (alertmanager)
    Tasks: 9 (limit: 2344)
   CGroup: /system.slice/alertmanager.service
           └─14680 /usr/local/bin/alertmanager --config.file=/etc/alertmanager/alertmanager.yml --storage.path=/data/alertmanager

Nov 11 05:29:46 ubuntu-server1 systemd[1]: Started Alert Manager.
Nov 11 05:29:46 ubuntu-server1 alertmanager[14680]: level=info ts=2020-11-11T05:29:46.572Z caller=main.go:216 msg="Starting Alertmanager" versi
Nov 11 05:29:46 ubuntu-server1 alertmanager[14680]: level=info ts=2020-11-11T05:29:46.574Z caller=main.go:217 build_context="(go=go1.14.4, user
Nov 11 05:29:46 ubuntu-server1 alertmanager[14680]: level=info ts=2020-11-11T05:29:46.578Z caller=cluster.go:161 component=cluster msg="setting
Nov 11 05:29:46 ubuntu-server1 alertmanager[14680]: level=info ts=2020-11-11T05:29:46.592Z caller=cluster.go:623 component=cluster msg="Waiting
Nov 11 05:29:46 ubuntu-server1 alertmanager[14680]: level=info ts=2020-11-11T05:29:46.614Z caller=coordinator.go:119 component=configuration ms
Nov 11 05:29:46 ubuntu-server1 alertmanager[14680]: level=info ts=2020-11-11T05:29:46.614Z caller=coordinator.go:131 component=configuration ms
Nov 11 05:29:46 ubuntu-server1 alertmanager[14680]: level=info ts=2020-11-11T05:29:46.616Z caller=main.go:485 msg=Listening address=:9093
Nov 11 05:29:48 ubuntu-server1 alertmanager[14680]: level=info ts=2020-11-11T05:29:48.593Z caller=cluster.go:648 component=cluster msg="gossip

root@ubuntu-server1:~# systemctl enable alertmanager
Created symlink /etc/systemd/system/multi-user.target.wants/alertmanager.service → /etc/systemd/system/alertmanager.service.

Configure AlertManager with secure reverse proxy

Creating proxy configuration file for alertmanager

root@ubuntu-server1:/etc/nginx/sites-enabled# touch alertmanager.conf

Generate self-signed certificates

This time we will use another method (with certtool) to create self signed certs.

root@ubuntu-server1:/etc/nginx/ssl# apt-get install gnutls-bin

root@ubuntu-server1:/etc/nginx/ssl# certtool --generate-self-signed --load-privkey alertmanager-private-key.pem --outfile alertmanager-cert.pemGenerating a self signed certificate...
Please enter the details of the certificate's distinguished name. Just press enter to ignore a field.
Common name: Alertmanager cert
UID:
Organizational unit name: DevOps
Organization name: Fosstechnix
Locality name: Bangalore
State or province name: Karnataka
Country name (2 chars): IN
Enter the subject's domain component (DC):
This field should not be used in new certificates.
E-mail:
Enter the certificate's serial number in decimal (default: 6893737989104015219):


Activation/Expiration time.
The certificate will expire in (days): 3650

Extensions.
Does the certificate belong to an authority? (y/N): y
Will the certificate be used to sign other certificates? (y/N): y
Will the certificate be used to sign CRLs? (y/N): y

Is the above information ok? (y/N): y


Signing certificate...
root@ubuntu-server1:/etc/nginx/ssl# ls
alertmanager-cert.pem  alertmanager-private-key.pem  ubuntu-server1.key  ubuntu-server1.pem

Configure Nginx file

root@ubuntu-server1:/etc/nginx/sites-enabled# cat alertmanager.conf
upstream ubuntu-server1.example.com {
server localhost:9093;
}

server {
server_name ubuntu-server1.example.com;
listen 3456 ssl;
ssl_certificate /etc/nginx/ssl/alertmanager-cert.pem;
ssl_certificate_key /etc/nginx/ssl/alertmanager-private-key.pem;
charset utf-8;
client_max_body_size 16m;
keepalive_timeout 70;
client_body_timeout 10;
client_header_timeout 10;
client_header_buffer_size 128;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://ubuntu-server1.example.com;
}
}

Test nginx configuration and restart the service

root@ubuntu-server1:/etc/nginx/sites-enabled# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

root@ubuntu-server1:/etc/nginx/sites-enabled# systemctl restart nginx

root@ubuntu-server1:/etc/nginx/sites-enabled# journalctl -f -u nginx.service

Configure firewall to allow connections on port 3456/9093

root@ubuntu-server1:/etc/nginx/sites-enabled# ufw allow 3456
Rule added
Rule added (v6)
root@ubuntu-server1:/etc/nginx/sites-enabled# ufw allow 9093
Rule added
Rule added (v6)

Modify alertmanager service

Add the following parameter – –web.external-url=https://localhost:3456

root@ubuntu-server1:/etc/nginx/sites-enabled# cat /etc/systemd/system/alertmanager.service
[Unit]
Description=Alert Manager
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=alertmanager
Group=alertmanager
ExecStart=/usr/local/bin/alertmanager \
  --config.file=/etc/alertmanager/alertmanager.yml \
  --storage.path=/data/alertmanager \
  --web.external-url=https://localhost:3456

Restart=always

[Install]
WantedBy=multi-user.target

Reload the systemd daemon and restart alertmanager service

root@ubuntu-server1:/etc/nginx/sites-enabled# systemctl daemon-reload
root@ubuntu-server1:/etc/nginx/sites-enabled# systemctl restart alertmanager

root@ubuntu-server1:/etc/nginx/sites-enabled# cat /var/log/syslog | grep -i alert
Nov 11 05:29:46 ubuntu-server1 systemd[1]: Started Alert Manager.

Verify the access

Go to advanced and proceed with https url.

Great! Its working as expected and we are able to access the AlertManager UI.

Binding AlertManager with Prometheus

It’s time to bind AlertManager with Prometheus to do that go to your Prometheus configuration directory, and make following changes in /etc/prometheus/prometheus.yml file.

# Alertmanager configuration
alerting:
  alertmanagers:
  - scheme: https
    static_configs:
    - targets:
       - ubuntu-server1.example.com

Restart Prometheus server and verify if everything is working fine.

root@ubuntu-server1:/etc/prometheus# systemctl restart prometheus.service

Monitor alertmanager instance (in Our case its on the same host) with prometheus

Update prometheus.yml file and add following section of code under scrape_configs:

- file_sd_configs:
    - files:
      - /etc/prometheus/files_sd/alertmanager.yml
    job_name: alertmanager

Here we have introduced another Prometheus config file parameter that is file_sd_configs. Lets understand that.

file_sd_config

File-based service discovery provides a more generic way to configure static targets and serves as an interface to plug in custom service discovery mechanisms.

It reads a set of files containing a list of zero or more <static_config>s. Changes to all defined files are detected via disk watches and applied immediately. Files may be provided in YAML or JSON format. Only changes resulting in well-formed target groups are applied.

Create alertmanager.yml target file under files_sd directory

root@ubuntu-server1:/etc/prometheus/files_sd# ls
alertmanager.yml
root@ubuntu-server1:/etc/prometheus/files_sd# cat alertmanager.yml
- targets:
  - ubuntu-server1.example.com:9093

Restart the Prometheus service

root@ubuntu-server1:/etc/prometheus# systemctl restart prometheus.service

Verify the status over UI

We can clearly see a new target has been added to our monitoring list.

Verify the metrices scrapped

Till now we are done with setting up the AlertManager with Prometheus.

Install and Configure Grafana

Follow the step by step procedure described below to install Grafana.

Update the package repo

root@ubuntu-server1:~# apt-get update
root@ubuntu-server1:~# apt-get install -y apt-transport-https
root@ubuntu-server1:~# apt-get install -y software-properties-common wget

Add the gpg key to install signed packages

root@ubuntu-server1:~# wget -q -O - https://packages.grafana.com/gpg.key | apt-key add -
OK

Add this repository for stable releases:

For OSS release –

root@ubuntu-server1:~# echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
deb https://packages.grafana.com/oss/deb stable main

For Enterprise version –

root@ubuntu-server1:~# echo "deb https://packages.grafana.com/enterprise/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
deb https://packages.grafana.com/enterprise/deb stable main

Update repositories again and install Grafana

root@ubuntu-server1:~# apt-get update

root@ubuntu-server1:~# apt-get install grafana -y

Verify the version

root@ubuntu-server1:~# grafana-server -v
Version 7.3.1 (commit: 6668161a88, branch: HEAD)

Verify the Grafana systemd service file

root@ubuntu-server1:~# cat /usr/lib/systemd/system/grafana-server.service
[Unit]
Description=Grafana instance
Documentation=http://docs.grafana.org
Wants=network-online.target
After=network-online.target
After=postgresql.service mariadb.service mysql.service

[Service]
EnvironmentFile=/etc/default/grafana-server
User=grafana
Group=grafana
Type=simple
Restart=on-failure
WorkingDirectory=/usr/share/grafana
RuntimeDirectory=grafana
RuntimeDirectoryMode=0750
ExecStart=/usr/sbin/grafana-server                                                  \
                            --config=${CONF_FILE}                                   \
                            --pidfile=${PID_FILE_DIR}/grafana-server.pid            \
                            --packaging=deb                                         \
                            cfg:default.paths.logs=${LOG_DIR}                       \
                            cfg:default.paths.data=${DATA_DIR}                      \
                            cfg:default.paths.plugins=${PLUGINS_DIR}                \
                            cfg:default.paths.provisioning=${PROVISIONING_CFG_DIR}


LimitNOFILE=10000
TimeoutStopSec=20
UMask=0027

[Install]
WantedBy=multi-user.target

Understand this file a bit more.

• The Grafana server binary is available at /usr/sbin/grafana-server.
• Environment variables file is located at /etc/default/grafana-server
• CONF_FILE environment variable mention the config file.
• The PID of the file is also determined by the PID_FILE_DIR environment variable.
• Logging, data, plugins and provisioning paths are given by environment variables.

Enable and Start Grafana service

root@ubuntu-server1:~# systemctl enable grafana-server
Synchronizing state of grafana-server.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable grafana-server
Created symlink /etc/systemd/system/multi-user.target.wants/grafana-server.service → /usr/lib/systemd/system/grafana-server.service.

root@ubuntu-server1:~# systemctl start grafana-server

root@ubuntu-server1:~# systemctl status grafana-server
● grafana-server.service - Grafana instance
   Loaded: loaded (/usr/lib/systemd/system/grafana-server.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2020-11-11 07:06:47 UTC; 4s ago
     Docs: http://docs.grafana.org
 Main PID: 18505 (grafana-server)
    Tasks: 8 (limit: 2344)
   CGroup: /system.slice/grafana-server.service
           └─18505 /usr/sbin/grafana-server --config=/etc/grafana/grafana.ini --pidfile=/var/run/grafana/grafana-server.pid --packaging=deb cfg

Nov 11 07:06:51 ubuntu-server1 grafana-server[18505]: t=2020-11-11T07:06:51+0000 lvl=info msg="Executing migration" logger=migrator id="create
Nov 11 07:06:51 ubuntu-server1 grafana-server[18505]: t=2020-11-11T07:06:51+0000 lvl=info msg="Executing migration" logger=migrator id="add uni
Nov 11 07:06:51 ubuntu-server1 grafana-server[18505]: t=2020-11-11T07:06:51+0000 lvl=info msg="Executing migration" logger=migrator id="add uni
Nov 11 07:06:51 ubuntu-server1 grafana-server[18505]: t=2020-11-11T07:06:51+0000 lvl=info msg="Executing migration" logger=migrator id="add ind
Nov 11 07:06:51 ubuntu-server1 grafana-server[18505]: t=2020-11-11T07:06:51+0000 lvl=info msg="Executing migration" logger=migrator id="create
Nov 11 07:06:51 ubuntu-server1 grafana-server[18505]: t=2020-11-11T07:06:51+0000 lvl=info msg="Executing migration" logger=migrator id="add uni
Nov 11 07:06:51 ubuntu-server1 grafana-server[18505]: t=2020-11-11T07:06:51+0000 lvl=info msg="Executing migration" logger=migrator id="create
Nov 11 07:06:51 ubuntu-server1 grafana-server[18505]: t=2020-11-11T07:06:51+0000 lvl=info msg="Executing migration" logger=migrator id="add ind
Nov 11 07:06:52 ubuntu-server1 grafana-server[18505]: t=2020-11-11T07:06:52+0000 lvl=info msg="Created default admin" logger=sqlstore user=admi
Nov 11 07:06:52 ubuntu-server1 grafana-server[18505]: t=2020-11-11T07:06:52+0000 lvl=info msg="Starting plugin search" logger=plugins

Allow firewall to access port 3000

root@ubuntu-server1:~# ufw allow 3000
Rule added
Rule added (v6)

Verify the GUI for Grafana

Open the url in browser – http://ubuntu-server1.example.com:3000/login

NOTE: The default login for Grafana is ‘admin’ and the default password is also ‘admin’.

After inserting those credentials, you will be asked to change your password.

Add Prometheus Data Source

So now that your Grafana server is up and running, you can start playing with it by adding/creating different dashboards.

Lets import some default sample dashboards and visualize the data we are getting from Prometheus server.

We will be importing following two dashboards.

https://grafana.com/grafana/dashboards/3662

https://grafana.com/grafana/dashboards/2

With 3662 dashboard ID Overview of metrics from Prometheus 2.0. Useful for using prometheus to monitor your prometheus.

More dashboards can be found here – https://grafana.com/grafana/dashboards

That’s all!

It’s a bit lengthy article. Isn’t it! But that covers end to end concepts and tools when it comes to monitoring.

In coming articles I will cover how we can monitor Windows servers, Applications, Databases with different Prometheus exporters.

I hope you like the article.

Please let us know your feedback below in the comment section.

Related Articles:

Kubernetes cluster Monitoring with Prometheus and Grafana

Reference:

Prometheus official site

Grafana official site