Dr Sambit Patra mind blowing speech

https://www.youtube.com/watch?v=VzUyib-MJRo

What you want in life

Had written a lot of Technical blogs . So now it's turn for Life again.

Read it somewhere and I totally agree on this:
“ The quality of your life is in direct proportion to the amount of uncertainty you can comfortably deal with.” ~Tony Robbins

There are many occasion in our life when we just don’t know what we want. These are the awkward moments where we feel uncertain and unsure, and perhaps even question our existence.

Always remember. You don't earn name or fame because you are beautiful and attractive. People get attracted or inspired only by your work and not look. Example Shahrukh khan, Kajol etc. There are thousands of beautiful girls in kashmir and other part of India who are more fairer & attractive than kajol but it's because of her work people recognize and admire her, and not for her complexion.I always underestimated the sentence "There is no substitute to hard work". But as life moved forward, I realized the importance of these lines. Life teaches us a lot.

Many people (including me ;)) these days have a lot of goals in our life but we actually don't realize what we truly want. We should not think what we want "from" life, but think what we want "in" life. And rightly said by Modi jee  (Narendra Modi ) too :: "Kuch banne mein biswas maat rakho, kuch karne mein biswas rakho" (Don't try to become something, try to do something).

I had changed 4 companies in last 5 years. Friends and colleague thinks money is the main motive behind my frequent hops. To be very honest, that can be counted of one of the reason but practically that has a very small fraction of contribution in my decision to switch companies frequently. The actual reason is still unknown to me but there is something I am looking for but I am not able to get things in right direction for myself and I think thats what we call life. And this time I went one step ahead, apart from changing company I moved to a different city too, in search of **SOMETHING**. And I am being very honest here. I really don't know what that **SOMETHING** is. Leaving Bangalore was a pain and I have plan in future to move back to Bangalore. But let's see where does life takes me.

Even at some sort I thought to start my own business but I didn't had that much courage or will power to do that. We indians will never become a great entrepreneur as westerns do for obvious reasons. Our parents set our goal for our life (I am not complaining here, they do that for our good future only).....good grade.....good college....good job....car...house....family.....etc..... and by the time we realize our actual needs and our purpose of existence,it's time to leave this world. We then think that if we have the power to go back to older days and rectify our mistakes then things would have been different. But that's what life is, we can't change things as per our convenience and who knows the things would  been more worse after that.

Zindagi Gulzar Hai

This is perhaps the first Pakistani serial I had watched and probably the best hindi serial (I am referring to romantic genre only) I had watched till date.  A great acting by all star cast but mainly Fawad Afzal Khan (Zaroon) and Sanam Saeed (kashaf) . 

Below is the full lyrics of one of the songs which I liked the most.......

Zindagi Khaaq Na Thi, Khaaq Udate Guzri
Tujh Se Kya Kehte, Tere Paas Jo Aate Guzri
Zindagi……….

Din Jo Guzra To Kisi Yaad Ki Ro Mein Guzra
Shaam Aayi To Koi Khwaab Dikhate Guzri
Tujh Se Kya Kehte, Tere Paas Jo Aatey Guzri
Zindagi……….

Acche Waqtoon Ki Tamanna Mein Rahi Umar-e-Rawaan
Waqt Aisa Tha Ki Bus Naaz Uthate Guzri
Tujh Se Kya Kehte, Tere Paas Jo Aate Guzri
Zindagi………..

Raat Kya Aayi Ki Tanhai Ke Sargoshi Mein
Ghum Ka Aalam Tha Magar Sunte Sunate Guzri
Tujh Se Kya Kehte, Tere Paas Jo Aate Guzri
Zindagi………. Zindagi………… Zindagi……………

How to install Logstash with Kibana interface??

We will need to install below stuffs to start with our activity:

1) Nginx

2) Logstash - You can get from the from this link : https://download.elasticsearch.org/logstash/logstash/logstash-1.4.1.tar.gz

3) Elasticsearch

4) Kibana

Step 1: Add the official ElasticSearch repository for Version 1.1.x

$ sudo vi /etc/yum.repos.d/elasticsearch.repo

Step 2: Add the following content to this file

[elasticsearch-1.1]

name=Elasticsearch repository for 1.1.x packages

baseurl=http://packages.elasticsearch.org/elasticsearch/1.1/centos

gpgcheck=1

gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch

enabled=1

Step 3: Install required packages. You may need to install more if packages are not installed earlier.

sudo yum install nginx elasticsearch policycoreutils-python 

Step 4: Create the following directories:

$ sudo mkdir -p /var/log/nginx/kibana

$ sudo mkdir -p /usr/share/nginx/kibana/public

$ sudo mkdir -p /etc/nginx/sites-available

$ sudo mkdir -p /etc/nginx/sites-enabled

Step 5: Go to the nginx directory [ /etc/nginx/] and replace nginx.conf with the below one:

$ sudo wget https://github.com/kulshresht/nginx/blob/master/nginx.conf

Step 6: Go to the nginx vhost directory

$ cd /etc/nginx/sites-available/

You can get the official kibana vhost that kibana provides from below link:

https://github.com/elasticsearch/kibana/blob/master/sample/nginx.conf

But I had made changes and disabled password protected endpoints. You can get the mofidied version at below link.

$ sudo wget https://github.com/kulshresht/nginx/blob/master/kibana

P.S: Server name needs to be replaced with variable {$hostname}

Step 7: Symbolic link the vhost file so nginx will load it

$ sudo ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/kibana

Step 8: Go to the kibana folder

$ cd /usr/share/nginx/kibana/public

Step 9: Download the latest Kibana version

$ sudo wget https://download.elasticsearch.org/kibana/kibana/kibana-latest.tar.gz

Step 10: Untar Kibana and fix directory stucture

$ sudo tar xzfv kibana-latest.tar.gz

$ sudo mv kibana-latest/* .

$ sudo rm -rf kibana-latest.tar.gz

$ sudo rm -rf kibana-latest

Step 11: Open the config.js file

$ sudo vi config.js

Step 12: Change the file a bit

Change the following line:

default_route : '/dashboard/file/default.json',

To the following:

default_route : '/dashboard/file/kulshresht.json',

Step 13: Go to the dashboard directory

$ cd /usr/share/nginx/kibana/public/app/dashboards

Step 14: Download the following version of kulshresht.json

$ sudo wget https://github.com/kulshresht/nginx/blob/master/kulshresht.json

The downloaded version is identical to the official file with little bit of changes in the graph size and how the logs show.

Step 15: Create a nologin user called kibana

$ sudo useradd -s /sbin/nologin kibana

Step 16: Chown the web dir to kibana:nginx

$ sudo chown -R kibana:nginx /usr/share/nginx/kibana/

Step 17: Start Logstash, ElasticSearch and Nginx

$ sudo service elasticsearch start

$ sudo service logstash start

$ sudo service nginx start

If you now go to your website for example for me “http://{$hostname}” you will see the awesome UI.

For feeding logs to logstash/redis server:

How to setup Logstash:

Step 1:

• cd /opt
• sudo wget https://download.elasticsearch.org/logstash/logstash/logstash-1.4.1.tar.gz
• sudo tar -xvf logstash-1.4.1.tar.gz
• cd logstash-1.4.1
• sudo vim redis.conf - > Add below content inside redis.conf (This is server side settings where kibana and redis and running). Losgtash config file settings for shipper will be different.

***********************************

input {
  redis {
    host => "{Server-IP}"
    type => "redis"
    data_type => "list"
    key => "logstash"
  }
}

output {
  elasticsearch
        {
                host => localhost
         }
  stdout { codec => rubydebug }
  }
***********************************


Step 2: Change the ownership of the Logstash config file
$ sudo chown logstash:logstash logstash.conf

Step 3: Start logstash -> cd logstash-1.4.1
# bin/logstash -f redis.conf -vv


How to setup Redis server:

Step 1: sudo wget http://download.redis.io/releases/redis-2.8.10.tar.gz
Step 2: sudo tar -xvf redis-2.8.10.tar.gz
Step 3: cd redis-2.8.10
Step 4: sudo make
Step 5: sudo make install
Step 6: cd /opt/redis-2.8.10/src
Step 7: Start redis server: # sudo ./redis-server

Mobile redirection

Assumption: There is a site kulshresht.com and it supports both version, Desktop and mobile site but with different code base on each of them.

Desktop version -->  www.kulshresht.com

Mobile site --> m.kulshresht.com

The below rule redirect visitors from different mobile devices to mobile site.We have also added a support for a cookie, so that mobile users can still opt to visit the full site, in case they don't like mobile site.

The first block checks to see if it is a mobile device and there is neither a cookie nor a url parameter set. If those conditions are met, redirect the visitor to the mobile site. The second block looks to see if the url parameter is set and there is no cookie. If those conditions are met, set a cookie for the user.

In practice, this will automatically redirect mobile users to a mobile site. On the mobile site, add a link going to www.kulshresht.com?fullbrowser=1. When the visitor uses that link, they will get back to the original site, a cookie will be set, and they can now bypass the redirect while the cookie is still valid. The point is to redirect mobile visitors, but not permanently if they still want to visit the full version of the site.

Download Apache config from http://detectmobilebrowsers.com/ and we can use that for detecting mobile in mod rewrite.

RewriteEngine On RewriteCond %{HTTP_HOST} ^www.kulshresht.com [NC,OR] RewriteCond %{HTTP_HOST} ^kulshresht.com [NC] RewriteCond %{HTTP_USER_AGENT} (android|bb\d+|meego).+mobile|avantgo|bada\/|blackberry|blazer|compal|elaine|fennec|hiptop|iemobile|ip(hone|od)|iris|kindle|lge\ |maemo|midp|mmp|mobile.+firefox|netfront|opera\ m(ob|in)i|palm(\ os)?|phone|p(ixi|re)\/|plucker|pocket|psp|series(4|6)0|symbian|treo|up\.(browser|link)|vodafone|wap|windows\ (ce|phone)|xda|xiino [NC,OR] RewriteCond %{HTTP_USER_AGENT} ^(1207|6310|6590|3gso|4thp|50[1-6]i|770s|802s|a\ wa|abac|ac(er|oo|s\-)|ai(ko|rn)|al(av|ca|co)|amoi|an(ex|ny|yw)|aptu|ar(ch|go)|as(te|us)|attw|au(di|\-m|r\ |s\ )|avan|be(ck|ll|nq)|bi(lb|rd)|bl(ac|az)|br(e|v)w|bumb|bw\-(n|u)|c55\/|capi|ccwa|cdm\-|cell|chtm|cldc|cmd\-|co(mp|nd)|craw|da(it|ll|ng)|dbte|dc\-s|devi|dica|dmob|do(c|p)o|ds(12|\-d)|el(49|ai)|em(l2|ul)|er(ic|k0)|esl8|ez([4-7]0|os|wa|ze)|fetc|fly(\-|_)|g1\ u|g560|gene|gf\-5|g\-mo|go(\.w|od)|gr(ad|un)|haie|hcit|hd\-(m|p|t)|hei\-|hi(pt|ta)|hp(\ i|ip)|hs\-c|ht(c(\-|\ |_|a|g|p|s|t)|tp)|hu(aw|tc)|i\-(20|go|ma)|i230|iac(\ |\-|\/)|ibro|idea|ig01|ikom|im1k|inno|ipaq|iris|ja(t|v)a|jbro|jemu|jigs|kddi|keji|kgt(\ |\/)|klon|kpt\ |kwc\-|kyo(c|k)|le(no|xi)|lg(\ g|\/(k|l|u)|50|54|\-[a-w])|libw|lynx|m1\-w|m3ga|m50\/|ma(te|ui|xo)|mc(01|21|ca)|m\-cr|me(rc|ri)|mi(o8|oa|ts)|mmef|mo(01|02|bi|de|do|t(\-|\ |o|v)|zz)|mt(50|p1|v\ )|mwbp|mywa|n10[0-2]|n20[2-3]|n30(0|2)|n50(0|2|5)|n7(0(0|1)|10)|ne((c|m)\-|on|tf|wf|wg|wt)|nok(6|i)|nzph|o2im|op(ti|wv)|oran|owg1|p800|pan(a|d|t)|pdxg|pg(13|\-([1-8]|c))|phil|pire|pl(ay|uc)|pn\-2|po(ck|rt|se)|prox|psio|pt\-g|qa\-a|qc(07|12|21|32|60|\-[2-7]|i\-)|qtek|r380|r600|raks|rim9|ro(ve|zo)|s55\/|sa(ge|ma|mm|ms|ny|va)|sc(01|h\-|oo|p\-)|sdk\/|se(c(\-|0|1)|47|mc|nd|ri)|sgh\-|shar|sie(\-|m)|sk\-0|sl(45|id)|sm(al|ar|b3|it|t5)|so(ft|ny)|sp(01|h\-|v\-|v\ )|sy(01|mb)|t2(18|50)|t6(00|10|18)|ta(gt|lk)|tcl\-|tdg\-|tel(i|m)|tim\-|t\-mo|to(pl|sh)|ts(70|m\-|m3|m5)|tx\-9|up(\.b|g1|si)|utst|v400|v750|veri|vi(rg|te)|vk(40|5[0-3]|\-v)|vm40|voda|vulc|vx(52|53|60|61|70|80|81|83|85|98)|w3c(\-|\ )|webc|whit|wi(g\ |nc|nw)|wmlb|wonu|x700|yas\-|your|zeto|zte\-) [NC] RewriteCond %{QUERY_STRING} !^.*fullbrowser=1.*$ [NC] RewriteCond %{HTTP_COOKIE} !fullbrowser=1 [NC] RewriteRule ^(.*)$ http://m.kulshresht.com$1 [R=302,L] # Set cookie, if user doesn't want to be redirected RewriteCond %{QUERY_STRING} ^.*fullbrowser=1.*$ RewriteCond %{HTTP_COOKIE} !^.*fullbrowser.*$ [NC] RewriteRule .* - [co=fullbrowser:1:www.kulshresht.com]

References:

http://www.phillipadsmith.com/2011/11/note-to-self-a-simple-way-to-accomplish-mobile-site-redirection-using-mod-rewrite.html
http://david.narrabilis.com/node/40

Apache Performance Tunning / Apache Optimization

Apache Optimization

Apache (HTPD) is  most popular and widely used web server around the globe. It comes with multiple modules. The term MPM is used for multiprocessing module. We can check for default mpm by running this command “ httpd -l ”

[kulshresht@dev server ~]#  httpd -l

Compiled in modules:

  core.c

  prefork.c

  http_core.c

  mod_so.c

There could be n number of reasons for bad performance of your website and one of the possibility can be that Apache is not able to handle load. Apache server performance can be improved/enhanced by adding additional hardware resources such as RAM, faster CPU, etc. But most of the time it's custom server configuration which will play a bigger role rather than just adding extra hardware. We will discuss here about those configuration so that we can make Apache performance better using the Apache MPM prefork module.

Apache 2 is available with following 2 MPM modules: Prefork & Worker

To check which MPM module it is running on your server, use below command:

[kulshresht@dev server~]$ /usr/sbin/httpd -l | egrep "prefork|worker"

prefork.c

The most important configuration file for Apache is httpd.conf which is located at /etc/httpd/conf [for Centos].We will try to tune this configuration file.For example:

vim /etc/httpd/conf/httpd.conf

Will write a sample configuration first and then try to explain all parameters and how to tune them one by one. 

# prefork MPM # StartServers: number of server processes to start # MinSpareServers: minimum number of server processes which are kept spare # MaxSpareServers: maximum number of server processes which are kept spare # ServerLimit: maximum value for MaxClients for the lifetime of the server # MaxClients: maximum number of server processes allowed to start # MaxRequestsPerChild: maximum number of requests a server process serves <IfModule prefork.c> StartServers       20 MinSpareServers    20 MaxSpareServers   50 ServerLimit      2000 MaxClients       2000 MaxRequestsPerChild  2000 </IfModule>

StartServers : StartServers directive sets how many server processes are created upon start up. Since web server dynamically kills and creates server processes based on traffic load, it is not necessary to change this parameter.

MinSpareServers and MaxSpareServers:
These values are only used with the prefork MPM. They adjust how the Apache HTTP Server dynamically adapts to the perceived load by maintaining an appropriate number of spare server processes based on the number of incoming requests. The server checks the number of servers waiting for a request and kills some if there are more than MaxSpareServers or creates some if the number of servers is less than MinSpareServers.
The default MinSpareServers value is 5; the default MaxSpareServers value is 20. These default settings should be appropriate for most situations. Be careful not to increase the MinSpareServers to a large number as doing so creates a heavy processing load on the server even when traffic is light.

MinSpareThreads and MaxSpareThreads:
These values are only used with the worker MPM. They adjust how the Apache HTTP Server dynamically adapts to the perceived load by maintaining an appropriate number of spare server threads based on the number of incoming requests. The server checks the number of server threads waiting for a request and kills some if there are more than MaxSpareThreads or creates some if the number of servers is less than MinSpareThreads.
The default MinSpareThreads value is 25; the default MaxSpareThreads value is 75. These default settings should be appropriate for most situations. The value for MaxSpareThreads must be greater than or equal to the sum of MinSpareThreads and ThreadsPerChild, else the Apache HTTP Server automatically corrects it.

ThreadsPerChild: This value is only used with the worker MPM. It sets the number of threads within each child process. The default value for this directive is 25.

MaxClients: MaxClients sets a limit on total number of concurrent connections. Will let you know in a while about how to determine that value.The server's default is set to 150 regardless of the MPM in use.There is no point in keeping this value very high until and unless you have huge traffic serving site. A high value can lead to a complete server hang in case of a DOS attack. A value too low can create timeout problems for your clients if the limit is reached.So take appropriate decision after seeing production behavior during high load by using below technique.

Use below shell script to determine an average amount of memory consumed by one Apache process. In addition to that it will show total amount of memory consumed by all Apache processes.

[kulshresht@server1 ~]$ ps -ylC httpd | awk '{x += $8;y += 1} END {print "Apache Memory Usage (MB): "x/1024; print "Average Proccess Size (MB): "x/((y-1)*1024)}'

Apache Memory Usage (MB): 821.441

Average Proccess Size (MB): 6.08475

**Use httpd for CentOS in above shell script. Use apache2 otherwise.

Execute the above script several times during high load to get optimum result.Now we know average amount of memory consumed by Apache and total amount of memory of your server, it is possible to calculate value to be used for MaxClients setting. For example, if in average one of your Apache process consumes 6MB RAM and server RAM is 16GB, and you want to leave 8GB for the rest processes, then:
MaxClients = (16GB – 8GB)/6MB = 1365

ServerLimit : This value should be same as MaxClients. But MaxClient setting can be changed on the fly without restarting Apache whereas ServerLimit new value needs Apache restart.

MaxRequestsPerChild: MaxRequestsPerChild sets the total number of requests each child server process serves before the child dies. The main reason for setting MaxRequestsPerChild is to avoid long-lived process induced memory leaks. The default MaxRequestsPerChild for the prefork MPM is 4000 and for the worker MPM is 0.We should definitely set some value for this parameter (pls do not ignore this ) to avoid memory leak by long lived process in worst case scenario.

If you still wish to switch to "worker" module, it is sufficient to uncomment this directive in the /etc/sysconfig/httpd file:

# grep HTTPD= /etc/sysconfig/httpd
#HTTPD=/usr/sbin/httpd.worker

and restart Apache:

/etc/init.d/httpd restart

References:
http://httpd.apache.org/docs/current/mod/prefork.html
http://www.centos.org/docs/4/4.5/Reference_Guide/s3-apache-minmaxsparethreads.html
http://kb.parallels.com/en/113007
https://code.google.com/p/check-httpd-limits/
http://www.zarafa.com/wiki/index.php/Apache_tuning

Redis cache monitoring / Automating telnet session using bash scripts

Below is a combination of shell & expect script which helps to monitor Redis cache.

Get your shell script [redistest.sh] embedded in Nagios/Zenoss [ or whichever monitoring tool you are using ] to set up monitoring for redis cache.

redistest.sh #!/bin/bash set -x var=`/tmp/redistest| awk '{print $1}'|grep PONG|cut -c2-5` echo $var if [ "$var"  == "PONG" ] then echo "Redis is now running fine |" exit 0 else echo "Redis is having some issue |" exit 2 fi echo "Unknown error|" exit 3

redistest #!/usr/bin/expect set timeout 5 set serverip "10.50.16.10" set port 6379 spawn telnet $serverip $port send "ping\r" expect "+PONG" exit

Reference:

http://stackoverflow.com/questions/7013137/automating-telnet-session-using-bash-scripts