In this article, We are going to perform, How to Install ELK Stack on Ubuntu 18.04/16.04 LTS or any Cloud Instance like Amazon EC2, Azure VM, Google Compute Engine,etc. Now ELK Stack renamed as Elastic Stack with the addition of FileBeats.
Table of Contents
Introduction
ELK is the combination of three open source projects: Elasticsearch, Logstash, Kibana and Filebeat. Elasticsearch is a search and analytics engine. Logstash is a server‑side logs processing pipeline that transport logs from multiple sources simultaneously, transforms it, and then sends it to a “stash” like Elasticsearch. Kibana is to visualize logs with charts and graphs from Elasticsearch.
WorkFlow = ElasticSearch —> Kibana —> Logstash —> Filebeat
Prerequisites
- Ubuntu Server with 18.04/16.04 LTS
- jdk – Java 8
- 2 CPU and 4 GB RAM
- Open Ports 9200, 5601, 5044
Install JDK on Ubuntu
Please follow below article to download and install Oracle JAVA 8 on Ubuntu 18.04/16.04 LTS Manually.
Download and Install Oracle Java 8 on Ubuntu 18.04/16.04 LTS
Step #1: Install and Configure ElasticSearch
Here, We are adding ElasticSearch official apt package and this is signed with GPG keys
sudo wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
Next add the repository in /etc/apt/sources.list.d/elastic-6.x.list using below command.
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
To install elasticsearch enter below command
sudo apt-get update
sudo apt-get install elasticsearch
Output:
Reading package lists... Done Building dependency tree Reading state information... Done The following NEW packages will be installed: elasticsearch 0 upgraded, 1 newly installed, 0 to remove and 50 not upgraded. Need to get 149 MB of archives. After this operation, 238 MB of additional disk space will be used. Get:1 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 elasticsearch all 6.8.4 [149 MB] Fetched 149 MB in 3s (38.7 MB/s) . . . OK Unpacking elasticsearch (6.8.4) ... Processing triggers for systemd (229-4ubuntu21.22) ... Processing triggers for ureadahead (0.100.0-19.1) ... Setting up elasticsearch (6.8.4) ... Created elasticsearch keystore in /etc/elasticsearch Processing triggers for systemd (229-4ubuntu21.22) ... Processing triggers for ureadahead (0.100.0-19.1) ..
Let’s make changes in below configuration file
sudo vi /etc/elasticsearch/elasticsearch.yml
uncommnet network.host and make changes as below
network.host: 0.0.0.0
To start elacticsearch services
sudo systemctl start elasticsearch
To enable elacticsearch at system startup
sudo systemctl enable elasticsearch
Output:
Synchronizing state of elasticsearch.service with SysV init with /lib/systemd/systemd-sysv-install... Executing /lib/systemd/systemd-sysv-install enable elasticsearch Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.
To check the status of elasticsearch
sudo systemctl status elasticsearch
Output:
● elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2019-11-13 09:43:43 IST; 6min ago Docs: http://www.elastic.co Main PID: 112878 (java) Tasks: 41 Memory: 1.2G CPU: 39.173s CGroup: /system.slice/elasticsearch.service ├─112878 /usr/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networ └─113029 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller Nov 13 09:43:43 master systemd[1]: Started Elasticsearch. Nov 13 09:43:43 master elasticsearch[112878]: warning: Falling back to java on path. This behavior is deprecated. Specify JAVA_HOME lines 1-14/14 (END)
To stop elasticsearch
sudo systemctl stop elasticsearch
Enter below command to check elasticsearch is running or not
curl -X GET "localhost:9200"
Output:
{ "name" : "-8VPh1P", "cluster_name" : "elasticsearch", "cluster_uuid" : "KpBdinybQdGUJH7KKQzRYg", "version" : { "number" : "6.8.4", "build_flavor" : "default", "build_type" : "deb", "build_hash" : "bca0c8d", "build_date" : "2019-10-16T06:19:49.319352Z", "build_snapshot" : false, "lucene_version" : "7.7.2", "minimum_wire_compatibility_version" : "5.6.0", "minimum_index_compatibility_version" : "5.0.0" }, "tagline" : "You Know, for Search" }
Step #2: Install and Configure Kibana
To install Kibana
sudo apt-get install kibana
Output:
Reading package lists... Done Building dependency tree Reading state information... Done The following NEW packages will be installed: kibana 0 upgraded, 1 newly installed, 0 to remove and 50 not upgraded. Need to get 193 MB of archives. After this operation, 442 MB of additional disk space will be used. Get:1 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 kibana amd64 6.8.4 [193 MB] Fetched 193 MB in 5s (33.7 MB/s) Selecting previously unselected package kibana. (Reading database ... 114535 files and directories currently installed.) Preparing to unpack .../kibana_6.8.4_amd64.deb ... Unpacking kibana (6.8.4) ... Processing triggers for systemd (229-4ubuntu21.22) ... Processing triggers for ureadahead (0.100.0-19.1) ... Setting up kibana (6.8.4) ... sent invalidate(group) request, exiting sent invalidate(passwd) request, exiting . . . . Processing triggers for systemd (229-4ubuntu21.22) ... Processing triggers for ureadahead (0.100.0-19.1) ...
Now let’s make changes in below configuration file for to access kibana
sudo vi /etc/kibana/kibana.yml
uncomment server.host at line 7 and make changes as mentioned below
server.host: "0.0.0.0"
To start kibana service
sudo systemctl start kibana
To enable kibana at system startup
sudo systemctl enable kibana
Output:
Synchronizing state of kibana.service with SysV init with /lib/systemd/systemd-sysv-install... Executing /lib/systemd/systemd-sysv-install enable kibana
To check the status of kibana service
sudo systemctl status kibana
Output:
● kibana.service - Kibana Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2019-11-13 09:49:13 IST; 51s ago Main PID: 116609 (node) Tasks: 11 Memory: 519.8M CPU: 38.257s CGroup: /system.slice/kibana.service └─116609 /usr/share/kibana/bin/../node/bin/node --no-warnings --max-http-header-size=65536 /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml Nov 13 09:50:01 sivasai kibana[116609]: {"type":"log","@timestamp":"2019-11-13T09:50:01Z","tags":["warning","elasticsearch","admin"],"pid":116609,"message":"Unable to revive connection: http Nov 13 09:50:01 sivasai kibana[116609]: {"type":"log","@timestamp":"2019-11-13T09:50:01Z","tags":["warning","elasticsearch","admin"],"pid":116609,"message":"No living connections"} Nov 13 09:50:02 sivasai kibana[116609]: {"type":"log","@timestamp":"2019-11-13T09:50:02Z","tags":["warning","elasticsearch","admin"],"pid":116609,"message":"Unable to revive connection: http Nov 13 09:50:02 sivasai kibana[116609]: {"type":"log","@timestamp":"2019-11-13T09:50:02Z","tags":["warning","elasticsearch","admin"],"pid":116609,"message":"No living connections"} Nov 13 09:50:02 sivasai kibana[116609]: {"type":"log","@timestamp":"2019-11-13T09:50:02Z","tags":["warning","task_manager"],"pid":116609,"message":"PollError No Living connections"} Nov 13 09:50:04 sivasai kibana[116609]: {"type":"log","@timestamp":"2019-11-13T09:50:04Z","tags":["warning","elasticsearch","admin"],"pid":116609,"message":"Unable to revive connection: http Nov 13 09:50:04 sivasai kibana[116609]: {"type":"log","@timestamp":"2019-11-13T09:50:04Z","tags":["warning","elasticsearch","admin"],"pid":116609,"message":"No living connections"} Nov 13 09:50:05 sivasai kibana[116609]: {"type":"log","@timestamp":"2019-11-13T09:50:05Z","tags":["warning","elasticsearch","admin"],"pid":116609,"message":"Unable to revive connection: http Nov 13 09:50:05 sivasai kibana[116609]: {"type":"log","@timestamp":"2019-11-13T09:50:05Z","tags":["warning","elasticsearch","admin"],"pid":116609,"message":"No living connections"} Nov 13 09:50:05 sivasai kibana[116609]: {"type":"log","@timestamp":"2019-11-13T09:50:05Z","tags":["warning","task_manager"],"pid":116609,"message":"PollError No Living connections"} lines 1-20/20 (END)
To stop kibana service
sudo systemctl stop kibana
Step #3: Install and Configure Logstash
To install logstash
sudo apt-get install logstash
Output:
The following NEW packages will be installed: logstash 0 upgraded, 1 newly installed, 0 to remove and 56 not upgraded. Need to get 174 MB of archives. After this operation, 304 MB of additional disk space will be used. Get:1 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 logstash all 1:6.8.4-1 [174 MB] Fetched 174 MB in 9s (19.0 MB/s) Selecting previously unselected package logstash. (Reading database ... 241519 files and directories currently installed.) Preparing to unpack .../logstash_1%3a6.8.4-1_all.deb ... sent invalidate(passwd) request, exiting sent invalidate(group) request, exiting sent invalidate(group) request, exiting sent invalidate(passwd) request, exiting sent invalidate(group) request, exiting sent invalidate(passwd) request, exiting sent invalidate(group) request, exiting Unpacking logstash (1:6.8.4-1) ... Setting up logstash (1:6.8.4-1) ... Using provided startup.options file: /etc/logstash/startup.options /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.30/lib/pleaserun/platform/base.rb:112: warning: constant ::Fixnum is deprecated Successfully created system startup script for Logstash
To load logstash beat create the below config file and insert below lines.
sudo nano /etc/logstash/conf.d/02-beats-input.conf
input { beats { port => 5044 } }
save and close the file.
Create the configuration file and insert below lines
sudo nano /etc/logstash/conf.d/30-elasticsearch-output.conf
output { elasticsearch { hosts => ["localhost:9200"] manage_template => false index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" } }
To start logstash services
sudo systemctl start logstash
To enable logstash at system startup
sudo systemctl enable logstash
Output:
Created symlink from /etc/systemd/system/multi-user.target.wants/logstash.service to /etc/systemd/system/logstash.service.
To stop logstash services
sudo systemctl stop logstash
To check status of logstash
sudo systemctl status logstash
Step #4: Install and Configure Filebeat
To install filebeat
sudo apt-get install filebeat
Now lets make changes in below configuration file
sudo vi /etc/filebeat/filebeat.yml
In the configuration file go to Filebeat Section change false to true as shown below :
#=========================== Filebeat inputs ============================= # Change to true to enable this input configuration. enabled: true # (change false to true)
In the configuration file go to Kibana Section as shown below:
#============================== Kibana ===================================== # uncomment the host and change "localhost to IP" host: "<IP of server>:5601"
In the configuration file got ElasticSearch Section as shown below:
#-------------------------- Elasticsearch output ------------------------------ # uncommnet the hosts section hosts: ["localhost:9200"]
To start filebeat services:
sudo systemctl start filebeat
To enable filebeat at system startup
sudo systemctl enable filebeat
output:
Synchronizing state of filebeat.service with SysV init with /lib/systemd/systemd-sysv-install... Executing /lib/systemd/systemd-sysv-install enable filebeat
To check status of filebeat services
sudo systemctl status filebeat
Output:
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch. Loaded: loaded (/lib/systemd/system/filebeat.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2019-11-19 09:26:00 IST; 3min 10s ago Docs: https://www.elastic.co/products/beats/filebeat Main PID: 4513 (filebeat) CGroup: /system.slice/filebeat.service └─4513 /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/f Nov 19 09:26:00 master systemd[1]: Started Filebeat sends log files to Logstash or directly to Elasticsearch.. l
Now lets check that ElasticSearch is receiving datalog from filebeat using below command
sudo curl -XGET 'https://localhost:9200/filebeat-*/_search?pretty'
output:
{ "took" : 7, "timed_out" : false, "_shards" : { "total" : 18, "successful" : 18, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 2887, "max_score" : 1.0, "hits" : [ { "_index" : "filebeat-6.8.4-2019.11.15", "_type" : "doc", "_id" : "YgcZbm4Bq0FM-JrvEymu", "_score" : 1.0, "_source" : { "offset" : 556, "log" : { "file" : { "path" : "/var/log/mysql/error.log" } }, "prospector" : { "type" : "log" }, "source" : "/var/log/mysql/error.log", "fileset" : { "module" : "mysql", "name" : "error" }, "input" : { "type" : "log" }, . . . ...
Finally lets login into kibana portal using http:/<ip of the server>:5601
We have covered Install ELK stack on ubuntu.
Conclusion:
In this article, We have performed ,How to Install ELK Stack on Ubuntu 18.04/16.04 LTS and any other cloud platform like Azure, EC2, Compute Engine System.
- How to Install Elastic Stack on Ubuntu 20.04 LTS
- How to Install Elastic Stack on CentOS 7
- 5 Simple Steps to Download & Install Jenkins on Windows