In this article we are going to cover How to Install Prometheus and Grafana on Linux with Node Exporter/Install Prometheus and Grafana on Linux (CentOS, RedHat and Amazon Linux 2), Monitoring Linux server with Prometheus and Grafana with Node Exporter.
What is Prometheus ?
- Prometheus is a open source Linux Server Monitoring tool mainly used for metrics monitoring, event monitoring, alert management, etc.
- Prometheus has changed the way of monitoring systems and that is why it has become the Top-Level project of Cloud Native Computing Foundation (CNCF).
- Prometheus uses a powerful query language i.e. “PromQL”.
- In Prometheus tabs are on and handles hundreds of services and microservices.
- Prometheus use multiple modes used for graphing and dashboarding support.
- As above we can see an architecture of Prometheus monitoring tool.
- We made a basic design to understand it easily for you people.
Now lets understand the Prometheus components one-by-one
1. Prometheus Server
- Prometheus server is a first component of Prometheus architecture.
- Prometheus server is a core of Prometheus architecture which is divided into several parts like Storage, PromQL, HTTP server, etc.
- In Prometheus server data is scraped from the target nodes and then stored int the database.
- Storage in Prometheus server has a local on disk storage.
- Prometheus has many interfaces that allow integrating with remote storage systems.
- Prometheus uses its own query language i.e. PromQL which is very powerful querying language.
- PromQL allows the user to select and aggregate the data.
2. Service Discovery
- Next and very important component of Prometheus Server is the Service Discovery.
- With the help of Service discovery the services are identified which are need to scraped.
- To Pull metrics, identification of services and finding the targets are compulsory needed.
- Through Service discovery we monitor the entities and can also locate its targets.
3. Scrape Target
- Once the services are identified and the targets are ready then we can pull metrics from it and can scrape the target.
- We can export the data of end point using node exporters.
- Once the metrics or other data is pulled, Prometheus stores it in a local storage.
4. Alert Manager
- Alert Manager handles the alerts which may occurs during the session.
- Alert manager handles all the alerts which are sent by Prometheus server.
- Alert manager is one of the very useful component of Prometheus tool.
- If in case any big error or any issue occurs, alert manager manage those alerts and contact with human via E-mail, Text Messages, On-call, or any other chat application service.
5. User Interface
- User interface is also a important component as it builds a bridge between the user and the system.
- In Prometheus, user interface are note that much user friendly and can be used till graph queries.
- For good exclusive dashboards Prometheus works together with Grafana (visualization tool).
- Using Grafana over Prometheus to visualize properly we can use custom dashboards.
- Grafana dashboards displays via pie charts, line charts, tables, good data graphs of CPU usage, RAM utilization, network load, etc with indicators.
- Grafana supports and run with Prometheus by querying language i.e. PromQL.
- To fetch data from Prometheus and to display the results on Grafana dashboards PromQL is used.
What is Grafana ?
- Grafana is a free and open source visualization tool mostly used with Prometheus to which monitor metrics.
- Grafana provides various dashboards, charts, graphs, alerts for the particular data source.
- Grafana allows us to query, visualize, explore metrics and set alerts for the data source which can be a system, server, nodes, cluster, etc.
- We can also create our own dynamic dashboard for visualization and monitoring.
- We can save the dashboard and can even share with our team members which is one of the main advantage of Grafana.
What is Node Exporter ?
- Node exporter is one of the Prometheus exporters which is used to expose servers or system OS metrics.
- With the help of Node exporter we can expose various resources of the system like RAM, CPU utilization, Memory Utilization, disk space.
- Node exporter runs as a system service which gathers the metrics of your system and that gathered metrics is displayed with the help of Grafana visualization tool.
- Linux Server (CentOS, RedHat, Amazon Linux 2)
- Root user account with sudo privilege.
- Prometheus system user and group.
- Sufficient storage on your system and good internet connectivity.
- Ports Required- 9090 (Prometheus), 3000 (Grafana), 9100 (Node Exporter)
We will update the system repository index by using the following command.
sudo yum update -y
switch to root user
sudo su -
Export the release of Prometheus
#1. Creating Prometheus System Users and Directory
We will have to create a Prometheus user named Prometheus and a Prometheus directory named as Prometheus.
Using below commands we can create a user and directory.
sudo useradd --no-create-home --shell /bin/false prometheus
sudo useradd --no-create-home --shell /bin/false node_exporter
sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
#2. Update Prometheus user
As user groups and directories are created successfully which store the Prometheus data and files.
Now we will have to update the group and user ownership on the newly created directories.
By using the below command we update the ownership.
sudo chown prometheus:prometheus /etc/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus
#3. Download Prometheus Binary File
Now we will download the latest version of Prometheus. We can copy the download link as per our Operating System from Prometheus download page
Using below command we can download Prometheus, here we are downloading Prometheus 2.26 version, you use above link to download specific version.
Navigate to /opt directory
Download the Prometheus setup using wget
Now we have successfully downloaded the Prometheus file and now we will extract that file.
#4.Install Prometheus and Grafana on Linux
We can use sha256sum command line to generate a checksum of the Prometheus downloaded file.
We will also extract the downloaded file using the tar command.
We have verify that the output from above command with checksum matches the sha256sum checksum which is on official Prometheus download page.
It will ensure that our downloaded file is not a corrupted file.
Now we will extract the Prometheus setup file using the following commands.
tar -xvf prometheus-2.26.0.linux-amd64.tar.gz
navigate to prometheus extracted folder
To check list of setup files
#5. Copy Prometheus Binary files
Now we have two libraries in our directory i.e. Prometheus and promtool. We will have to copy that both libraries to our /usr/local/bin directory.
By following below commands we will perform the copy operation.
sudo cp /opt/prometheus-2.26.0.linux-amd64/prometheus /usr/local/bin/
sudo cp /opt/prometheus-2.26.0.linux-amd64/promtool /usr/local/bin/
#6. Update Prometheus user ownership on Binaries
Now we will update the user and group ownership on the binaries of Prometheus.
Using following commands we will update the user and group ownership.
sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool
#7. Copy Prometheus Console Libraries
We will need to copy the console and console_libraries directories to /etc/Prometheus/.
Use below commands to copy console and console_libraries.
sudo cp -r /opt/prometheus-2.26.0.linux-amd64/consoles /etc/prometheus
sudo cp -r /opt/prometheus-2.26.0.linux-amd64/console_libraries /etc/prometheus
sudo cp -r /opt/prometheus-2.26.0.linux-amd64/prometheus.yml /etc/prometheus
#8. Update Prometheus ownership on Directories
Now we will update the user and group ownership on the directories to Prometheus user using -R.
By executing this commands ownership is set on is ensured. Execute the following commands.
sudo chown -R prometheus:prometheus /etc/prometheus/consoles
sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries
sudo chown -R prometheus:prometheus /etc/prometheus/prometheus.yml
#9. Check Prometheus Version
Now the Prometheus is successfully installed on our system. We will check the version of Prometheus and promtool to configure it.
Follow the commands to verify prometheus version.
#10. Prometheus configuration file
We have already copied /opt/prometheus-2.26.0.linux-amd64/prometheus.yml file /etc/prometheus directory, verify if it present and should look like below and modify it as per your requirement.
# my global config global: scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute. evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute. # scrape_timeout is set to the global default (10s). # Alertmanager configuration alerting: alertmanagers: - static_configs: - targets: # - alertmanager:9093 # Load rules once and periodically evaluate them according to the global 'evaluation_interval'. rule_files: # - "first_rules.yml" # - "second_rules.yml" # A scrape configuration containing exactly one endpoint to scrape: # Here it's Prometheus itself. scrape_configs: # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config. - job_name: 'prometheus' # metrics_path defaults to '/metrics' # scheme defaults to 'http'. static_configs: - targets: ['localhost:9090']
#11. Creating Prometheus Systemd file
To run Prometheus as service we have to setting up prometheus, We will provide a path for both configuration file and data directory. We will start it with the Prometheus user using the following command.
sudo -u prometheus /usr/local/bin/prometheus \ --config.file /etc/prometheus/prometheus.yml \ --storage.tsdb.path /var/lib/prometheus/ \ --web.console.templates=/etc/prometheus/consoles \ --web.console.libraries=/etc/prometheus/console_libraries
Now we will create a system service file in /etc/systemd/system location.
sudo nano /etc/systemd/system/prometheus.service
After creating file successfully, copy the below files and it to the newly created file. /etc/systemd/system/prometheus.service
[Unit] Description=Prometheus Wants=network-online.target After=network-online.target [Service] User=prometheus Group=prometheus Type=simple ExecStart=/usr/local/bin/prometheus \ --config.file /etc/prometheus/prometheus.yml \ --storage.tsdb.path /var/lib/prometheus/ \ --web.console.templates=/etc/prometheus/consoles \ --web.console.libraries=/etc/prometheus/console_libraries [Install] WantedBy=multi-user.target
After adding the program save the file with Ctrl+O and exit with Ctrl+X.
To use the newly created service we will have to reload the daemon services, Use the below command to reload daemon services.
sudo systemctl daemon-reload
start and enable prometheus service using below commands
sudo systemctl start prometheus
sudo systemctl enable prometheus
We will check the Prometheus status weather it is running or not
sudo systemctl status prometheus
Prometheus installation and configuration is set up, We can see status Active: active(running)
#12. Accessing Prometheus
Now as Prometheus installation and configuration is set up and it is ready to use we can access its services via web interface.Also check weather port 9090 is UP in firewall.
Use below command to enable prometheus service in firewall
sudo ufw allow 9090/tcp
Now Prometheus service is ready to run and we can access it from any web browser.
As we can see the Prometheus dashboards, we can also check the target.As we can observe Current state is UP and we can also see the last scrape.
Here are some snapshots of Prometheus dashboards and graphs.
#13.Install Grafana on Linux (CentOS, RedHat, Amazon Linux 2)
Now lets Install Grafana for wonderful dashboards and data visualization for monitoring systems, servers, services, etc
Add the Grafana YUM repo using nano
sudo nano /etc/yum.repos.d/grafana.repo
Add the below Grafana OSS repo into it, if you want use Enterprise version then visit Grafana Official Page
[grafana] name=grafana baseurl=https://packages.grafana.com/enterprise/rpm repo_gpgcheck=1 enabled=1 gpgcheck=1 gpgkey=https://packages.grafana.com/gpg.key sslverify=1 sslcacert=/etc/pki/tls/certs/ca-bundle.crt
update the system packages to take effect
sudo yum update
Now lets Install Grafana using below command
sudo yum install grafana
Now start the Grafana service using below command
sudo systemctl start grafana-server
Verify the Grafana Service Status using below command
sudo systemctl status grafana-server
Now finally enable the Grafana service which will automatically start the Grafana on boot
sudo systemctl enable grafana-server.service
To access Grafana Dashboard open your favorite browser, type server IP or Name followed by grafana default port 3000.
Here you can see Login page of Grafana now you will have to login with below Grafana default UserName and Password.
Username – admin
Password – admin
It is always a good practice to change your login credentials.
Provide your New Password and click on Change Password
Now here you can see Home Dashboard page of Grafana
#14. Configure Prometheus as Grafana DataSource
Once you logged into Grafana Now first Navigate to Settings Icon ->> Configuration ->> data sources
Now lets click on Add Data sources and select Prometheus
Now configure Prometheus data source by providing Prometheus URL
As per your requirement you can do other changes or you can also keep remaining configuration as default.
Now click on Save & test so it will prompt a message Data Source is working.
#15. Install Node Exporter on Linux (CentOS, RedHat, Amazon Linux 2)
Node Exporter collects the metrics of your system such as Memory usage, CPU usage, RAM, disk space, etc.
To install Node Exporter first navigate to Prometheus official download page, Scroll down and you will get node_exporter section and then select Linux OS for amd64.
Now right click on node exporter and copy link address
Now lets run the copied URL with wget command
Unzip the downloaded the file using below command
sudo tar xvzf node_exporter-1.2.0.linux-amd64.tar.gz
Now do ls and your can see node_exporter binary file.
Go to that file and move this file to your /usr/local/bin directory using below command
sudo cp node_exporter /usr/local/bin
As early at the time of Prometheus installation we have created node_exporter user and also updated the ownership permission so we will not repeat it again
#16. Creating Node Exporter Systemd service
Now lets create a node_exporter service in /lib/systemd/system directory named node_exporter.service using below commands
sudo nano node_exporter.service
Paste the below content in your service file
[Unit] Description=Node Exporter Wants=network-online.target After=network-online.target [Service] Type=simple User=node_exporter Group=node_exporter ExecStart=/usr/local/bin/node_exporter \ — collector.mountstats \ — collector.logind \ — collector.processes \ — collector.ntp \ — collector.systemd \ — collector.tcpstat \ — collector.wifi Restart=always RestartSec=10s [Install] WantedBy=multi-user.target
Now lets start and enable the node_exporter service using below commands
sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter
sudo systemctl status node_exporter
We have covered How to Install Prometheus and Grafana on Linux with Node Exporter.
#17. Configure the Node Exporter as a Prometheus target
Now to scrape the node_exporter lets instruct the Prometheus by making a minor change in prometheus.yml file
So go to etc/prometheus and open prometheus.yml
sudo nano prometheus.yml
Now in static_configs in your configuration file replace the target line with the below one
- targets: [‘localhost:9090’, ‘localhost:9100’]
Now restart the Prometheus Service
sudo systemctl restart prometheus
Hit the URL in your web browser to check weather our target is successfully scraped by Prometheus or not
We have covered How to Install Prometheus and Grafana on Linux with Node Exporter.
#18. Creating Grafana Dashboard to Monitor Linux Server
Now lets build a dashboard in Grafana so then it will able to reflect the metrics of the Linux system.
So we will use 14513 to import Grafana.com, Lets come to Grafana Home page and you can see a “+” icon. Click on that and select “Import”
Now provide the Grafana.com Dashboard ID which is 14513 and click on Load
Now provide the name and select the Prometheus Datasource and click on Import.
There you are done with the setup. Now your Dashboard is running up!.
We have covered How to Install Prometheus and Grafana on Linux with Node Exporter/Install Prometheus and Grafana on Linux (CentOS, RedHat and Amazon Linux 2), Monitoring Linux server with Prometheus and Grafana with Node Exporter.