How to install Logstash with Kibana interface??

We will need to install below stuffs to start with our activity:

1) Nginx

2) Logstash - You can get from the from this link : https://download.elasticsearch.org/logstash/logstash/logstash-1.4.1.tar.gz

3) Elasticsearch

4) Kibana

Step 1: Add the official ElasticSearch repository for Version 1.1.x

$ sudo vi /etc/yum.repos.d/elasticsearch.repo

Step 2: Add the following content to this file

[elasticsearch-1.1]

name=Elasticsearch repository for 1.1.x packages

baseurl=http://packages.elasticsearch.org/elasticsearch/1.1/centos

gpgcheck=1

gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch

enabled=1

Step 3: Install required packages. You may need to install more if packages are not installed earlier.

sudo yum install nginx elasticsearch policycoreutils-python 

Step 4: Create the following directories:

$ sudo mkdir -p /var/log/nginx/kibana

$ sudo mkdir -p /usr/share/nginx/kibana/public

$ sudo mkdir -p /etc/nginx/sites-available

$ sudo mkdir -p /etc/nginx/sites-enabled

Step 5: Go to the nginx directory [ /etc/nginx/] and replace nginx.conf with the below one:

$ sudo wget https://github.com/kulshresht/nginx/blob/master/nginx.conf

Step 6: Go to the nginx vhost directory

$ cd /etc/nginx/sites-available/

You can get the official kibana vhost that kibana provides from below link:

https://github.com/elasticsearch/kibana/blob/master/sample/nginx.conf

But I had made changes and disabled password protected endpoints. You can get the mofidied version at below link.

$ sudo wget https://github.com/kulshresht/nginx/blob/master/kibana

P.S: Server name needs to be replaced with variable {$hostname}

Step 7: Symbolic link the vhost file so nginx will load it

$ sudo ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/kibana

Step 8: Go to the kibana folder

$ cd /usr/share/nginx/kibana/public

Step 9: Download the latest Kibana version

$ sudo wget https://download.elasticsearch.org/kibana/kibana/kibana-latest.tar.gz

Step 10: Untar Kibana and fix directory stucture

$ sudo tar xzfv kibana-latest.tar.gz

$ sudo mv kibana-latest/* .

$ sudo rm -rf kibana-latest.tar.gz

$ sudo rm -rf kibana-latest

Step 11: Open the config.js file

$ sudo vi config.js

Step 12: Change the file a bit

Change the following line:

default_route : '/dashboard/file/default.json',

To the following:

default_route : '/dashboard/file/kulshresht.json',

Step 13: Go to the dashboard directory

$ cd /usr/share/nginx/kibana/public/app/dashboards

Step 14: Download the following version of kulshresht.json

$ sudo wget https://github.com/kulshresht/nginx/blob/master/kulshresht.json

The downloaded version is identical to the official file with little bit of changes in the graph size and how the logs show.

Step 15: Create a nologin user called kibana

$ sudo useradd -s /sbin/nologin kibana

Step 16: Chown the web dir to kibana:nginx

$ sudo chown -R kibana:nginx /usr/share/nginx/kibana/

Step 17: Start Logstash, ElasticSearch and Nginx

$ sudo service elasticsearch start

$ sudo service logstash start

$ sudo service nginx start

If you now go to your website for example for me “http://{$hostname}” you will see the awesome UI.

For feeding logs to logstash/redis server:

How to setup Logstash:

Step 1:

• cd /opt
• sudo wget https://download.elasticsearch.org/logstash/logstash/logstash-1.4.1.tar.gz
• sudo tar -xvf logstash-1.4.1.tar.gz
• cd logstash-1.4.1
• sudo vim redis.conf - > Add below content inside redis.conf (This is server side settings where kibana and redis and running). Losgtash config file settings for shipper will be different.

***********************************

input {
  redis {
    host => "{Server-IP}"
    type => "redis"
    data_type => "list"
    key => "logstash"
  }
}

output {
  elasticsearch
        {
                host => localhost
         }
  stdout { codec => rubydebug }
  }
***********************************


Step 2: Change the ownership of the Logstash config file
$ sudo chown logstash:logstash logstash.conf

Step 3: Start logstash -> cd logstash-1.4.1
# bin/logstash -f redis.conf -vv


How to setup Redis server:

Step 1: sudo wget http://download.redis.io/releases/redis-2.8.10.tar.gz
Step 2: sudo tar -xvf redis-2.8.10.tar.gz
Step 3: cd redis-2.8.10
Step 4: sudo make
Step 5: sudo make install
Step 6: cd /opt/redis-2.8.10/src
Step 7: Start redis server: # sudo ./redis-server