How to Install ELK Stack on Ubuntu 18.04/16.04 LTS

by

In this article, We are going to perform, How to Install ELK Stack on Ubuntu 18.04/16.04 LTS or any Cloud Instance like Amazon EC2, Azure VM, Google Compute Engine,etc. Now ELK Stack renamed as Elastic Stack with the addition of FileBeats.

Introduction

ELK is the combination of  three open source projects: Elasticsearch, Logstash, Kibana and Filebeat. Elasticsearch is a search and analytics engine. Logstash is a server‑side logs processing pipeline that transport logs from multiple sources simultaneously, transforms it, and then sends it to a “stash” like Elasticsearch. Kibana is to visualize logs with charts and graphs from Elasticsearch.

WorkFlow  =  ElasticSearch  —> Kibana  —> Logstash —> Filebeat

Prerequisites

  • Ubuntu Server with 18.04/16.04 LTS
  • jdk – Java 8
  • 2 CPU and 4 GB RAM
  • Open Ports 9200, 5601, 5044

Install JDK on Ubuntu

Please follow below article to download and install Oracle JAVA 8 on Ubuntu 18.04/16.04 LTS Manually.

Download and Install Oracle Java 8 on Ubuntu 18.04/16.04 LTS

Step #1: Install and Configure ElasticSearch

Here, We are adding ElasticSearch official apt package and this is signed with GPG keys

sudo wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Next add the repository in /etc/apt/sources.list.d/elastic-6.x.list using below command.

echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list

To install elasticsearch enter below command

sudo apt-get update
sudo apt-get install elasticsearch

Output:

Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  elasticsearch
0 upgraded, 1 newly installed, 0 to remove and 50 not upgraded.
Need to get 149 MB of archives.
After this operation, 238 MB of additional disk space will be used.
Get:1 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 elasticsearch all 6.8.4 [149 MB]
Fetched 149 MB in 3s (38.7 MB/s)
.
.
.
 OK
Unpacking elasticsearch (6.8.4) ...
Processing triggers for systemd (229-4ubuntu21.22) ...
Processing triggers for ureadahead (0.100.0-19.1) ...
Setting up elasticsearch (6.8.4) ...
Created elasticsearch keystore in /etc/elasticsearch
Processing triggers for systemd (229-4ubuntu21.22) ...
Processing triggers for ureadahead (0.100.0-19.1) ..

Let’s make changes in below configuration file

sudo vi /etc/elasticsearch/elasticsearch.yml

uncommnet network.host and make changes as below

network.host: 0.0.0.0

To start elacticsearch services

sudo systemctl start elasticsearch

To enable elacticsearch at system startup

sudo systemctl enable elasticsearch

Output:

Synchronizing state of elasticsearch.service with SysV init with /lib/systemd/systemd-sysv-install...
Executing /lib/systemd/systemd-sysv-install enable elasticsearch
Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.

To check the status of elasticsearch

sudo systemctl status elasticsearch

Output:

● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2019-11-13 09:43:43 IST; 6min ago
     Docs: http://www.elastic.co
 Main PID: 112878 (java)
    Tasks: 41
   Memory: 1.2G
      CPU: 39.173s
   CGroup: /system.slice/elasticsearch.service
           ├─112878 /usr/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networ
           └─113029 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Nov 13 09:43:43 master systemd[1]: Started Elasticsearch.
Nov 13 09:43:43 master elasticsearch[112878]: warning: Falling back to java on path. This behavior is deprecated. Specify JAVA_HOME
lines 1-14/14 (END)

To stop elasticsearch

sudo systemctl stop elasticsearch

Enter below command to check elasticsearch is running or not

curl -X GET "localhost:9200"

Output:

{
  "name" : "-8VPh1P",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "KpBdinybQdGUJH7KKQzRYg",
  "version" : {
    "number" : "6.8.4",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "bca0c8d",
    "build_date" : "2019-10-16T06:19:49.319352Z",
    "build_snapshot" : false,
    "lucene_version" : "7.7.2",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

Step #2: Install and Configure Kibana

To install Kibana

sudo apt-get install kibana

Output:

Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  kibana
0 upgraded, 1 newly installed, 0 to remove and 50 not upgraded.
Need to get 193 MB of archives.
After this operation, 442 MB of additional disk space will be used.
Get:1 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 kibana amd64 6.8.4 [193 MB]
Fetched 193 MB in 5s (33.7 MB/s)
Selecting previously unselected package kibana.
(Reading database ... 114535 files and directories currently installed.)
Preparing to unpack .../kibana_6.8.4_amd64.deb ...
Unpacking kibana (6.8.4) ...
Processing triggers for systemd (229-4ubuntu21.22) ...
Processing triggers for ureadahead (0.100.0-19.1) ...
Setting up kibana (6.8.4) ...
sent invalidate(group) request, exiting
sent invalidate(passwd) request, exiting
.
.
.
.
Processing triggers for systemd (229-4ubuntu21.22) ...
Processing triggers for ureadahead (0.100.0-19.1) ...

Now let’s make changes in below configuration file for to access kibana

sudo vi /etc/kibana/kibana.yml

uncomment  server.host at line 7 and make changes as mentioned below

server.host: "0.0.0.0"

To start kibana service

sudo systemctl start kibana

To enable kibana at system startup

sudo systemctl enable kibana

Output:

Synchronizing state of kibana.service with SysV init with /lib/systemd/systemd-sysv-install...
Executing /lib/systemd/systemd-sysv-install enable kibana

To check the status of kibana service

sudo systemctl status kibana

Output:

● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2019-11-13 09:49:13 IST; 51s ago
 Main PID: 116609 (node)
    Tasks: 11
   Memory: 519.8M
      CPU: 38.257s
   CGroup: /system.slice/kibana.service
           └─116609 /usr/share/kibana/bin/../node/bin/node --no-warnings --max-http-header-size=65536 /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Nov 13 09:50:01 sivasai kibana[116609]: {"type":"log","@timestamp":"2019-11-13T09:50:01Z","tags":["warning","elasticsearch","admin"],"pid":116609,"message":"Unable to revive connection: http
Nov 13 09:50:01 sivasai kibana[116609]: {"type":"log","@timestamp":"2019-11-13T09:50:01Z","tags":["warning","elasticsearch","admin"],"pid":116609,"message":"No living connections"}
Nov 13 09:50:02 sivasai kibana[116609]: {"type":"log","@timestamp":"2019-11-13T09:50:02Z","tags":["warning","elasticsearch","admin"],"pid":116609,"message":"Unable to revive connection: http
Nov 13 09:50:02 sivasai kibana[116609]: {"type":"log","@timestamp":"2019-11-13T09:50:02Z","tags":["warning","elasticsearch","admin"],"pid":116609,"message":"No living connections"}
Nov 13 09:50:02 sivasai kibana[116609]: {"type":"log","@timestamp":"2019-11-13T09:50:02Z","tags":["warning","task_manager"],"pid":116609,"message":"PollError No Living connections"}
Nov 13 09:50:04 sivasai kibana[116609]: {"type":"log","@timestamp":"2019-11-13T09:50:04Z","tags":["warning","elasticsearch","admin"],"pid":116609,"message":"Unable to revive connection: http
Nov 13 09:50:04 sivasai kibana[116609]: {"type":"log","@timestamp":"2019-11-13T09:50:04Z","tags":["warning","elasticsearch","admin"],"pid":116609,"message":"No living connections"}
Nov 13 09:50:05 sivasai kibana[116609]: {"type":"log","@timestamp":"2019-11-13T09:50:05Z","tags":["warning","elasticsearch","admin"],"pid":116609,"message":"Unable to revive connection: http
Nov 13 09:50:05 sivasai kibana[116609]: {"type":"log","@timestamp":"2019-11-13T09:50:05Z","tags":["warning","elasticsearch","admin"],"pid":116609,"message":"No living connections"}
Nov 13 09:50:05 sivasai kibana[116609]: {"type":"log","@timestamp":"2019-11-13T09:50:05Z","tags":["warning","task_manager"],"pid":116609,"message":"PollError No Living connections"}
lines 1-20/20 (END)

To stop kibana service

sudo systemctl stop kibana

Step #3: Install and Configure Logstash

To install logstash

sudo apt-get install logstash

Output:

The following NEW packages will be installed:
  logstash
0 upgraded, 1 newly installed, 0 to remove and 56 not upgraded.
Need to get 174 MB of archives.
After this operation, 304 MB of additional disk space will be used.
Get:1 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 logstash all 1:6.8.4-1 [174 MB]
Fetched 174 MB in 9s (19.0 MB/s)
Selecting previously unselected package logstash.
(Reading database ... 241519 files and directories currently installed.)
Preparing to unpack .../logstash_1%3a6.8.4-1_all.deb ...
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
sent invalidate(group) request, exiting
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
Unpacking logstash (1:6.8.4-1) ...
Setting up logstash (1:6.8.4-1) ...
Using provided startup.options file: /etc/logstash/startup.options
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.30/lib/pleaserun/platform/base.rb:112: warning: constant ::Fixnum is deprecated
Successfully created system startup script for Logstash

To load logstash beat create the below config file and insert below lines.

sudo nano /etc/logstash/conf.d/02-beats-input.conf
input {

  beats {

    port => 5044

  }

}

save and close the file.

Create the configuration file and insert below lines

 sudo nano /etc/logstash/conf.d/30-elasticsearch-output.conf
output {

  elasticsearch {

    hosts => ["localhost:9200"]

    manage_template => false

    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"

  }

}

To start logstash services

sudo systemctl start logstash

To enable logstash at system startup

 sudo systemctl enable logstash

Output:

Created symlink from /etc/systemd/system/multi-user.target.wants/logstash.service to /etc/systemd/system/logstash.service.

To stop logstash services

sudo systemctl stop logstash

To check status of logstash

sudo systemctl status logstash

Step #4: Install and Configure Filebeat

To install filebeat

sudo apt-get install filebeat

Now lets make changes in below configuration file

sudo vi /etc/filebeat/filebeat.yml

In the configuration file go to Filebeat Section change false to true as shown below :

#=========================== Filebeat inputs =============================
# Change to true to enable this input configuration.
  enabled: true                            #  (change false to true)

In the configuration file go to Kibana Section as shown below:

#============================== Kibana =====================================
# uncomment the host and change "localhost to IP"
host: "<IP of server>:5601"

In the configuration file got ElasticSearch Section as shown below:

#-------------------------- Elasticsearch output ------------------------------
# uncommnet the hosts section
hosts: ["localhost:9200"]

To start filebeat services:

sudo systemctl start filebeat

To enable filebeat at system startup

sudo systemctl enable filebeat

output:

Synchronizing state of filebeat.service with SysV init with /lib/systemd/systemd-sysv-install...
Executing /lib/systemd/systemd-sysv-install enable filebeat

To check status of filebeat services

sudo systemctl status filebeat

Output:

● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
   Loaded: loaded (/lib/systemd/system/filebeat.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-11-19 09:26:00 IST; 3min 10s ago
     Docs: https://www.elastic.co/products/beats/filebeat
 Main PID: 4513 (filebeat)
   CGroup: /system.slice/filebeat.service
           └─4513 /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/f

Nov 19 09:26:00 master systemd[1]: Started Filebeat sends log files to Logstash or directly to Elasticsearch..
l

Now lets check that ElasticSearch is receiving datalog from filebeat using below command

sudo curl -XGET 'https://localhost:9200/filebeat-*/_search?pretty'

output:

{
  "took" : 7,
  "timed_out" : false,
  "_shards" : {
    "total" : 18,
    "successful" : 18,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 2887,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "filebeat-6.8.4-2019.11.15",
        "_type" : "doc",
        "_id" : "YgcZbm4Bq0FM-JrvEymu",
        "_score" : 1.0,
        "_source" : {
          "offset" : 556,
          "log" : {
            "file" : {
              "path" : "/var/log/mysql/error.log"
            }
          },
          "prospector" : {
            "type" : "log"
          },
          "source" : "/var/log/mysql/error.log",
          "fileset" : {
            "module" : "mysql",
            "name" : "error"
          },
          "input" : {
            "type" : "log"
          },
.
.
.
...

Finally lets login into kibana portal using http:/<ip of the server>:5601

How to Install ELK Stack on Ubuntu 18.04/16.04 LTS 1

We have covered Install ELK stack on ubuntu.

Conclusion:

In this article, We have performed ,How to Install ELK Stack on Ubuntu 18.04/16.04 LTS and any other cloud platform like Azure, EC2, Compute Engine System.

Related Articles:

Reference:

https://www.elastic.co/guide

0 Shares
Sivasai Sagar

I am working as DevOps Engineer and having 5 years of Experience. Likes to share knowledge.

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Share via
Facebook
X (Twitter)
LinkedIn
Mix
Email
Print
Copy Link
Powered by Social Snap
Copy link
CopyCopied
Powered by Social Snap