카테고리 없음

nginx + tomcat load balancer

[czar] 2013. 8. 22. 10:38

http://ae.koroglu.org/using-nginx-as-load-balancer-for-tomcat/


Using Nginx as Load Balancer for Tomcat




You may ask why Nginx? There are many reasons which I’ll not going to write down here, but you can read the nginx testimonials here..! So lets take a look at the big picture, what we want is a server on the internet side will load the balance for the servers on LAN side. Easy, right :P We’ll have 1 load-balancer, 2 application and 1 database server. Well, let’s get started..

nginx


Let’s get rid of not necessary extra memory killer TTYs, I always use 2 :)

/etc/sysconfig/init
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

# color => new RH6.0 bootup
# verbose => old-style bootup
# anything else => new style bootup without ANSI colors or positioning
BOOTUP=color
# column to start "[  OK  ]" label in
RES_COL=60
# terminal sequence to move to that column. You could change this
# to something like "tput hpa ${RES_COL}" if your terminal supports it
MOVE_TO_COL="echo -en \\033[${RES_COL}G"
# terminal sequence to set color to a 'success' color (currently: green)
SETCOLOR_SUCCESS="echo -en \\033[0;32m"
# terminal sequence to set color to a 'failure' color (currently: red)
SETCOLOR_FAILURE="echo -en \\033[0;31m"
# terminal sequence to set color to a 'warning' color (currently: yellow)
SETCOLOR_WARNING="echo -en \\033[0;33m"
# terminal sequence to reset to the default color.
SETCOLOR_NORMAL="echo -en \\033[0;39m"
# Set to anything other than 'no' to allow hotkey interactive startup...
PROMPT=yes
# Set to 'yes' to allow probing for devices with swap signatures
AUTOSWAP=no
# What ttys should gettys be started on?
ACTIVE_CONSOLES=/dev/tty[1-2]
# Set to '/sbin/sulogin' to prompt for password on single-user mode
# Set to '/sbin/sushell' otherwise
SINGLE=/sbin/sushell

We are going to use CentOS 6 for our system infrastructure, so let us add EPEL and nginx repositories to /etc/yum/repos.d. For EPEL repository please readread this.

 nginx.repo
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=0
enabled=1

After adding the repositories

yum clean all
yum upgrade

Installation

Let’s upgrade the system for each server

yum update

Load-balancer :

yum install nginx

Application Servers :

yum install nginx tomcat6-jsp-2.1-api tomcat6-lib tomcat6 tomcat6-admin-webapps tomcat6-webapps tomcat6-servlet-2.5-api tomcat6-el-2.1-api

I’ll not cover the database side, you can choose anything you like :)

Configuration: Load-balancer

Two configuration file is important here, /etc/nginx/nginx.conf and /etc/nginx/conf.d/default.conf

 /etc/nginx/nginx.conf
user  nginx;
worker_processes  4;
 
error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;
 
events {
    worker_connections  1024;
}
 
http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
 
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
 
    access_log  /var/log/nginx/access.log  main;
 
    sendfile        on;
    #tcp_nopush     on;
 
    # --- Size Limits & Buffer Overflows --- #
    client_body_buffer_size  1K;
    client_header_buffer_size 1k;
    client_max_body_size 1k;
    large_client_header_buffers 2 1k;
 
    ## Start: Timeouts ##
    client_body_timeout   10;
    client_header_timeout 10;
    keepalive_timeout     5 5;
    send_timeout          10;
 
    tcp_nodelay on;
    gzip on;
    gzip_http_version 1.1;
    gzip_vary on;
    gzip_comp_level 6;
    gzip_proxied any;
    gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript application/javascript text/x-js;
    gzip_buffers 16 8k;
    gzip_disable "MSIE [1-6]\.(?!.*SV1)";
 
    include /etc/nginx/conf.d/*.conf;
    server_names_hash_bucket_size  64;
    server_tokens off;
}
 /etc/nginx/conf.d/default.conf
upstream backend {
        ip_hash;
        server 192.168.1.11:80;
        server 192.168.1.12:80;
}
 
server {
        listen 80;
        location / {
                proxy_set_header Host $http_host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_pass  http://backend;
        }
}

Important note: If you forget to add “ip_hash” into upstream when you start the load-balancer, your visitors will bounce from one application server to another. It’s not good if you’re deploying a session based mechanism into your application.

Configuration: Application Servers

Nginx first..

 /etc/nginx/nginx.conf
user  nginx;
worker_processes  4;
 
error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;
 
events {
    worker_connections  1024;
}
 
http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
 
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
 
    access_log  /var/log/nginx/access.log  main;
 
    sendfile        on;
    #tcp_nopush     on;
 
    keepalive_timeout  65;
 
    #gzip  on;
    server_tokens off;
 
    include /etc/nginx/conf.d/*.conf;
}
 /etc/nginx/conf.d/aekoroglu.conf
server {
        listen 80;
        server_name test.koroglu.org;
        root /usr/share/tomcat6/webapps/aekoroglu;
        access_log /var/log/nginx/aekoroglu.access.log main;
        error_log /var/log/nginx/aekoroglu.error error;
 
    location / {
        index index.jsp;
    }
 
    location ~ \.do$ {
        proxy_pass http://localhost:8080;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
    }
 
    location ~ \.jsp$ {
        proxy_pass http://localhost:8080;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
    }
    location ^~/servlets/* {
        proxy_pass http://localhost:8080;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
    }
}

What I did here is if any do,jsp or servlets requested they will come from Tomcat and the rest (other files except do,jsp or servlets) will be coming via Nginx. Cause it’s not Tomcat’s job to serve static files. Better way is to have an extra nginx server to serve statics files (css,js,jpg,png etc..)

And Tomcat

 /etc/tomcat/server.xml
<?xml version='1.0' encoding='utf-8'?>
<Server port="8005" shutdown="SHUTDOWN">
  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
  <Listener className="org.apache.catalina.core.JasperListener" />
  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
  <Listener className="org.apache.catalina.mbeans.ServerLifecycleListener" />
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
  <GlobalNamingResources>
    <Resource name="UserDatabase" auth="Container"
              type="org.apache.catalina.UserDatabase"
              description="User database that can be updated and saved"
              factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
              pathname="conf/tomcat-users.xml" />
  </GlobalNamingResources>
 
  <Service name="Catalina">
    <Connector URIEncoding="UTF-8" port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />
    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
    <Engine name="Catalina" defaultHost="localhost">
      <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
             resourceName="UserDatabase"/>
      <Valve className="org.apache.catalina.valves.RemoteIpValve"  />
      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true"
            xmlValidation="false" xmlNamespaceAware="false">
      </Host>
 
        <!-- Virtual Hosts -->
        <Host name="test.koroglu.org" debug="0" unpackWARs="true">
        <Logger className="org.apache.catalina.logger.FileLogger" directory="logs" prefix="aekoroglu_log." suffix=".txt" timestamp="true"/>
        <Context path="" docBase="/usr/share/tomcat6/webapps/aekoroglu" debug="0" reloadable="true"/>
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" prefix="aekoroglu_log." suffix=".txt" pattern="common"/>
        </Host>
 
    </Engine>
  </Service>
</Server>

Optimization

I’ll write a documentation about network and system optimization soon..

Let’s run..

Load-balancer

/etc/init.d/nginx start

Application Servers

/etc/init.d/nginx start
/etc/init.d/tomcat6 start

If you want to start those services automatically in each reboot, you can add those services into startup services with chkconfig

chkconfig --level 3 nginx on
chkconfig --level 3 tomcat6 on

Or you can use ntsysv but its not coming with CentOS minimal installation, you can install with

yum install ntsysv

So that’s it, bon appetit.. :)

PS: I recommended all of you to choose minimal CentOS installation