Apache Performance Tunning / Apache Optimization

Apache Optimization

Apache (HTPD) is  most popular and widely used web server around the globe. It comes with multiple modules. The term MPM is used for multiprocessing module. We can check for default mpm by running this command “ httpd -l ”

[kulshresht@dev server ~]#  httpd -l

Compiled in modules:

  core.c

  prefork.c

  http_core.c

  mod_so.c

There could be n number of reasons for bad performance of your website and one of the possibility can be that Apache is not able to handle load. Apache server performance can be improved/enhanced by adding additional hardware resources such as RAM, faster CPU, etc. But most of the time it's custom server configuration which will play a bigger role rather than just adding extra hardware. We will discuss here about those configuration so that we can make Apache performance better using the Apache MPM prefork module.

Apache 2 is available with following 2 MPM modules: Prefork & Worker

To check which MPM module it is running on your server, use below command:

[kulshresht@dev server~]$ /usr/sbin/httpd -l | egrep "prefork|worker"

prefork.c

The most important configuration file for Apache is httpd.conf which is located at /etc/httpd/conf [for Centos].We will try to tune this configuration file.For example:

vim /etc/httpd/conf/httpd.conf

Will write a sample configuration first and then try to explain all parameters and how to tune them one by one. 

# prefork MPM # StartServers: number of server processes to start # MinSpareServers: minimum number of server processes which are kept spare # MaxSpareServers: maximum number of server processes which are kept spare # ServerLimit: maximum value for MaxClients for the lifetime of the server # MaxClients: maximum number of server processes allowed to start # MaxRequestsPerChild: maximum number of requests a server process serves <IfModule prefork.c> StartServers       20 MinSpareServers    20 MaxSpareServers   50 ServerLimit      2000 MaxClients       2000 MaxRequestsPerChild  2000 </IfModule>

StartServers : StartServers directive sets how many server processes are created upon start up. Since web server dynamically kills and creates server processes based on traffic load, it is not necessary to change this parameter.

MinSpareServers and MaxSpareServers:
These values are only used with the prefork MPM. They adjust how the Apache HTTP Server dynamically adapts to the perceived load by maintaining an appropriate number of spare server processes based on the number of incoming requests. The server checks the number of servers waiting for a request and kills some if there are more than MaxSpareServers or creates some if the number of servers is less than MinSpareServers.
The default MinSpareServers value is 5; the default MaxSpareServers value is 20. These default settings should be appropriate for most situations. Be careful not to increase the MinSpareServers to a large number as doing so creates a heavy processing load on the server even when traffic is light.

MinSpareThreads and MaxSpareThreads:
These values are only used with the worker MPM. They adjust how the Apache HTTP Server dynamically adapts to the perceived load by maintaining an appropriate number of spare server threads based on the number of incoming requests. The server checks the number of server threads waiting for a request and kills some if there are more than MaxSpareThreads or creates some if the number of servers is less than MinSpareThreads.
The default MinSpareThreads value is 25; the default MaxSpareThreads value is 75. These default settings should be appropriate for most situations. The value for MaxSpareThreads must be greater than or equal to the sum of MinSpareThreads and ThreadsPerChild, else the Apache HTTP Server automatically corrects it.

ThreadsPerChild: This value is only used with the worker MPM. It sets the number of threads within each child process. The default value for this directive is 25.

MaxClients: MaxClients sets a limit on total number of concurrent connections. Will let you know in a while about how to determine that value.The server's default is set to 150 regardless of the MPM in use.There is no point in keeping this value very high until and unless you have huge traffic serving site. A high value can lead to a complete server hang in case of a DOS attack. A value too low can create timeout problems for your clients if the limit is reached.So take appropriate decision after seeing production behavior during high load by using below technique.

Use below shell script to determine an average amount of memory consumed by one Apache process. In addition to that it will show total amount of memory consumed by all Apache processes.

[kulshresht@server1 ~]$ ps -ylC httpd | awk '{x += $8;y += 1} END {print "Apache Memory Usage (MB): "x/1024; print "Average Proccess Size (MB): "x/((y-1)*1024)}'

Apache Memory Usage (MB): 821.441

Average Proccess Size (MB): 6.08475

**Use httpd for CentOS in above shell script. Use apache2 otherwise.

Execute the above script several times during high load to get optimum result.Now we know average amount of memory consumed by Apache and total amount of memory of your server, it is possible to calculate value to be used for MaxClients setting. For example, if in average one of your Apache process consumes 6MB RAM and server RAM is 16GB, and you want to leave 8GB for the rest processes, then:
MaxClients = (16GB – 8GB)/6MB = 1365

ServerLimit : This value should be same as MaxClients. But MaxClient setting can be changed on the fly without restarting Apache whereas ServerLimit new value needs Apache restart.

MaxRequestsPerChild: MaxRequestsPerChild sets the total number of requests each child server process serves before the child dies. The main reason for setting MaxRequestsPerChild is to avoid long-lived process induced memory leaks. The default MaxRequestsPerChild for the prefork MPM is 4000 and for the worker MPM is 0.We should definitely set some value for this parameter (pls do not ignore this ) to avoid memory leak by long lived process in worst case scenario.

If you still wish to switch to "worker" module, it is sufficient to uncomment this directive in the /etc/sysconfig/httpd file:

# grep HTTPD= /etc/sysconfig/httpd
#HTTPD=/usr/sbin/httpd.worker

and restart Apache:

/etc/init.d/httpd restart

References:
http://httpd.apache.org/docs/current/mod/prefork.html
http://www.centos.org/docs/4/4.5/Reference_Guide/s3-apache-minmaxsparethreads.html
http://kb.parallels.com/en/113007
https://code.google.com/p/check-httpd-limits/
http://www.zarafa.com/wiki/index.php/Apache_tuning

Redis cache monitoring / Automating telnet session using bash scripts

Below is a combination of shell & expect script which helps to monitor Redis cache.

Get your shell script [redistest.sh] embedded in Nagios/Zenoss [ or whichever monitoring tool you are using ] to set up monitoring for redis cache.

redistest.sh #!/bin/bash set -x var=`/tmp/redistest| awk '{print $1}'|grep PONG|cut -c2-5` echo $var if [ "$var"  == "PONG" ] then echo "Redis is now running fine |" exit 0 else echo "Redis is having some issue |" exit 2 fi echo "Unknown error|" exit 3

redistest #!/usr/bin/expect set timeout 5 set serverip "10.50.16.10" set port 6379 spawn telnet $serverip $port send "ping\r" expect "+PONG" exit

Reference:

http://stackoverflow.com/questions/7013137/automating-telnet-session-using-bash-scripts

Puppet - Bit Advanced setup and configuration

In continuation to Basic puppet setup explained in my earlier post . Here is the advanced setup.

Advanced puppet configuration:

Agenda: What exactly I am trying to replicate here.I will be trying to keep 'mod_jk.conf' file which I had placed inside '/etc/puppet/files' folder of master server to keep in sync. Whenever I make any changes to mod_jk.conf file on puppet master,the daemon running on puppet agent will pick up the changes on master and bring agent in sync with puppet master. It will also restart Apache webserver where puppet agent is installed.

----------------------------------Master side configuration changes-----------------------------------

Create  a file inside '/etc/puppet/manifests'. Touch site.pp.

site.pp

import 'classes/*.pp' class toolbox {         file { '/tmp/gautam.txt':         owner => root, group => root, mode => 0755,         content => "Hi gautam....how are you",         }         }         node 'beta01.hs18.lan' {         include toolbox         include apache         include mysite         }

Create a folder 'classes' inside '/etc/puppet/manifests'. Now create two files inside classes folder.
apache.pp
mysite.pp

apache.pp

class apache {         package { 'mod_ssl-2.2.15-29.el6.centos.x86_64':         ensure => installed         }         service { 'httpd':                         ensure => running,                         hasstatus => true,                         hasrestart => true,                 } }

mysite.pp

class mysite {                 file { '/etc/httpd/conf.d/mod_jk.conf':         owner => root, group => root, mode => 0644,         source => "puppet:///files/mod_jk.conf",         } }

fileserver.conf [Please modify the content as per your requirement]

# This file consists of arbitrarily named sections/modules # defining where files are served from and to whom # Define a section 'files' # Adapt the allow/deny settings to your needs. Order # for allow/deny does not matter, allow always takes precedence # over deny [files] path /etc/puppet/files/ allow * #  allow *.hs18.lan #  deny *.evil.example.com #  allow 192.168.0.0/24 # [mount_point]         path /etc/puppet/files/     allow *

make a folder 'files' inside '/etc/puppet' and place our file [mod_jk.conf in my case]

Now start puppet agent as explained in my previous blog ( http://kulshresht-gautam.blogspot.in/2013/12/getting-started-with-puppet-basic-setup.html ) and play around with it.

Please watch the below video for better understanding.
http://www.youtube.com/watch?v=Hiu_ui2nZa0

Getting started with Puppet - Basic setup

In this blog I will try to explain about basic setup of puppet. Assuming that you are already aware about need/benefits of puppet.

Prerequisites:

1. Need two servers for this activity.Will make one of the server as puppet master and the other as puppet agent.
2. The other main prerequisites for installing puppet on redhat/centos is that we need to have the following.

• Ruby Language
• Ruby Libraries
• Shadow Ruby Libraries 

      [root@kulshresht1~]# yum install ruby-shadow ruby ruby-libs

In this example the name of two server is 'kulshresht1.home.lan' & 'kulshresht2.home.lan'

For understanding well:
kulshresht1.home.lan --> puppetmaster.example.org --> 10.50.20.19
kulshresht2.home.lan --> puppetagent.example.org   --> 10.50.20.30

Map the server name as 'puppetmaster.example.org' and 'puppetagent.example.org' respectively  in '/etc/hosts' file. It's better if you can get this registered in your local DNS for lookup.

Install puppet server on master server

[root@kulshresht1~]# yum install puppet puppet-server facter

puppet master server must contain the following packages:

1. Pupppet :: contains the puppet agent
2. Puppet-server :: contains the puppet master server
3. facter :: contain the tool which will act as fetching information about the node

Install puppet on agent server

[root@kulshresht2~]# yum install puppet facter

For testing basic setup and get puppet working , make "Agent side configuration changes" only, as stated below, and run the below command on puppet agent server. This is basic setup of puppet.

#puppet agent --no-daemonize --onetime --verbose

Now go on puppet master screen and sign the certificate using below commands

puppet cert list
puppet cert sign "puppetagent.example.org"

Few useful commands:

puppet cert clean puppetmaster.example.org

puppet cert clean puppetagent.example.org

START PUPPET AGENT:
puppet agent --no-daemonize --onetime --verbose

TEST AGENT:
puppet agent --test

CREATE/GENERATE CERTIFICATE:
puppet certificate generate puppetagent.example.org --ca-location  remote

HOW TO SIGN CERTIFICATE:
puppet cert list
puppet cert sign "puppetagent.example.org"

----------------------------------Agent side configuration changes-------------------------------------

Add below settings in "puppet.conf" on agent side.
server= kulshresht1.home.lan / puppetmaster.example.org
puppet.conf

[main]     # The Puppet log directory.     # The default value is '$vardir/log'.     logdir = /var/log/puppet     # Where Puppet PID files are kept.     # The default value is '$vardir/run'.     rundir = /var/run/puppet     # Where SSL certificates are kept.     # The default value is '$confdir/ssl'.     ssldir = $vardir/ssl     server=puppetmaster.example.org

Please go through http://kulshresht-gautam.blogspot.in/2013/12/puppet-bit-advanced-setup-and.html for advance puppet setup.

References:
http://www.slashroot.in/puppet-tutorial-installing-puppet-master-and-puppet-agent
http://blog.adityapatawari.com/2012/02/puppet-and-common-errors.html
http://www.youtube.com/watch?v=Hiu_ui2nZa0

How to find out what exactly is consuming space on my hard drive

Working nowadays a lot on linux had made me extremely week in windows environment. Hahahahahaha...;)

Was unable to figure out for a long time about what exactly is consuming space on my hard drive (Windows).

Googled a bit and came across a good software ( WizTree ) which resolved my issue within couple of minutes. Thought to share the same. Snapshot is attached below.

Link to download the software:

http://antibody-software.com/web/software/software/wiztree-finds-the-files-and-folders-using-the-most-disk-space-on-your-hard-drive/

Merry Christmas and have a great day ahead. GOD bless you all.

Kryo Serialization Strategy

Kryo is a fast and efficient object graph serialization framework for Java which is much faster and efficient compared to java serialization.The project is useful any time objects need to be persisted, whether to a file, database, or over the network.Kryo can also perform automatic deep and shallow copying/cloning. This is direct copying from object to object, not object->bytes->object

In one of my previous post I had explained about how to integrate tomcat with memcache for session clustering: http://kulshresht-gautam.blogspot.in/2013/07/integrating-tomcat-with-memcahed-for.html

Now in addition to do that post here I would focus on how to override java serialization with kryo serialization which is more faster and efficient compared to the earlier one [http://code.google.com/p/thrift-protobuf-compare/wiki/Benchmarking ].The aim is still to implement memcached-session-manager but we would use kryo serialization instead of normal java serialization.

First Option :

• As msm is available in maven central (under groupId de.javakaffee.msm) you can just pull it in using the dependency management of your build system. With maven you can use this dependency definition for the kryo-serializer:

<dependency>
    <groupId>de.javakaffee.msm</groupId>
    <artifactId>msm-kryo-serializer</artifactId>
    <version>1.6.5</version>
    <scope>runtime</scope>
</dependency>

Second Option :

• If you're not using a dependency management based on maven repositories then below are the jars you need for kryo serializers. Please download the appropriate jars and put them in $CATALINA_HOME/lib/ .

• memcached-session-manager-${version}.jar
• memcached-session-manager-tc7-${version}.jar
• kryo
• kryo-serializers-0.10.jar
• minlog
• msm-kryo-serializer-1.6.5.jar
• reflectasm-1.01.jar
• asm-3.2.jar
• spymemcached-2.8.12.jar
• couchbase-client-1.1.4.jar

The next task would be to add below entries in $CATALINA_HOME/conf/context.xml

<Context>

..................

        <Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"

                 memcachedNodes="n1:hostname1:11211,n2:hostname2:11211"

                 failoverNodes="n2"

                 requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"

                 transcoderFactoryClass="de.javakaffee.web.msm.serializer.kryo.KryoTranscoderFactory"

        />

</Context>

Restart tomcat and you are done.

References:

http://code.google.com/p/kryo/
https://code.google.com/p/memcached-session-manager/

https://code.google.com/p/memcached-session-manager/wiki/SerializationStrategyBenchmark

https://code.google.com/p/memcached-session-manager/wiki/SetupAndConfiguration

Simple Spring Memcached [ Memcached + Spring Caching ]

Introduction

Memcached is undoubtedly one of the most popular distributed caching system used across applications. Through his post I will try to make you understand on how to integrate memcache with a Spring enabled applications. Since Spring directly supports only Ehcache therefore we will use google's SSM (Simple Spring Memcache ) to use spring caching abstraction.

Getting started

1. Dependencies - For downloading SSM dependencies add following to your POM. 

 
  com.google.code.simple-spring-memcached
  spring-cache
  3.2.0
 
 
  com.google.code.simple-spring-memcached
  spymemcached-provider
  3.2.0
 
 

2. Enable Caching – To enable caching in your spring application add following to your spring memcached.xml.

<cache:annotation-driven/>

3. Configure Spring to enable Memcached based caching  – Add following to your application memcached.xml.

 
 
    
      
        
          
          
          
          
          
        
         
           // "applicationData1" is 1st cache Name for your code
          
          
          
          
        
         
           // "applicationData2" is 2nd cache Name for your code
          
          
          
          
        
        
    
  

 
  
  
   
  
  
   
    
   
  
  
   
    
   
      
      
  
  
   
     
 

  
    
  

  
    
  
 

** Note: You have to keep on adding "applicationData1", "applicationData2" ......."applicationData.....N" if you have N number of cache name for your application. 

Limitation: You are having common key for two different value as mentioned below.

@Cacheable(value = "applicationData1", key = #EmployeeId")

@Cacheable(value = "applicationData2", key = #EmployeeId")

In above scenario two  same key are trying to put some value therefore one key will overwrite the other key value and will lead to data corruption. To avoid this we have to make our key unique. This can be achieved by adding a string in-front of the key as done below. 

@Cacheable(value = "applicationData1", key = "'applicationData1'+#EmployeeId")

@Cacheable(value = "applicationData2", key = "'applicationData2'+#EmployeeId")

If you have no constraint about overwriting cache value then there is no need for making your key unique and everything can be put in single memcache. 

As stated earlier in my different post "phpMemCachedAdmin" is a great tool to Monitor And Debug Memcached Server [ http://kulshresht-gautam.blogspot.in/2013/08/how-to-monitor-and-debug-memcached.html ]. In the snap below you can see that the key and it's corresponding size. Expiration time for these keys are infinite which is clearly visible from snapshot below.

How to install memcache : I had explained it here: http://kulshresht-gautam.blogspot.in/2013/07/memcached-installation.html


References:
https://code.google.com/p/simple-spring-memcached/wiki/Getting_Started
https://code.google.com/p/simple-spring-memcached/wiki/UserGuide#Cache_zone