Home > Linux > High availability(cluster) with Load balancing by using Heartbeat and Haproxy

High availability(cluster) with Load balancing by using Heartbeat and Haproxy

This document describes how to set up a two-node load balancer in an active/passive configuration with HAProxy and heartbeat on Fedora 7. The load balancer acts between the user and two (or more) Apache web servers that hold the same content. The load balancer passes the requests to the web servers and it also checks their health. If one of them is down, all requests will automatically be redirected to the remaining web server(s). In addition to that, the two load balancer nodes monitor each other using heartbeat. If the master fails, the slave becomes the master – users won’t notice any disruption of the service.

This how to is a practical guide without any warranty – it doesn’t cover the theoretical backgrounds. There are many ways to set up such a system – this is the way which I tested.

Preparation

For this how to I set up five Fedora 7 systems (minimal installation without gui etc.) with the following configuration:

Load Balancer 1

Hostname: lb1.example.com
IP: 192.168.10.54
Shared IP: 192.168.10.206

Load Balancer 2

Hostname: lb2.example.com
IP: 192.168.10.91
Shared IP: 192.168.10.206

Web Server 1

Hostname: http1.example.com
IP: 192.168.10.100

Web Server 2

Hostname: http2.example.com
IP: 192.168.10.72

Database Server

Hostname: db.example.com

IP: 192.168.10.65

Overview

+—————–+
| 192.168.10.206 |
| Shared IP |
+——–+——–+
|
+———————-+
| |
+——–+——–+ +——–+——–+
| 192.168.10.54 | | 192.168.10.91 |
| Load Balancer 1 | | Load Balancer 2 |
+——–+——–+ +——–+——–+

+——–+——–+ +——–+——–+
| 192.168.10.100 | | 192.168.10.72 |
| Web Server 1 | | Web Server 2 |
+—————–+ +—————–+

+—————–+
| 192.168.10.65 |
| Database IP |
+——–+——–+
|

Make sure to turn off the httpd server in both Load balancers

Make sure to turn on the Firewall in all the load balancers and web servers

Apache Configuration

HAProxy will work as a transparent proxy – so the user’s IP address will be passed in the field “X-Forwarded-For” to the web servers. In order that the web servers will log the user’s IP address and not the IP addresses of the load balancers we have to modify the log format within the apache configuration file on both web servers.

vi /etc/httpd/conf/httpd.conf

Search the Lines that begin with “LogFormat”

[...]
LogFormat “%h %l %u %t \”%r\” %>s %b \”%{Referer}i\” \”%{User-Agent}i\”" combined
LogFormat “%h %l %u %t \”%r\” %>s %b” common
LogFormat “%{Referer}i -> %U” referer
LogFormat “%{User-agent}i” agent
[...]
… and replace “%h” with “%{X-Forwarded-For}i“. The content should look like this:
[...]
LogFormat “%{X-Forwarded-For}i %l %u %t \”%r\” %>s %b \”%{Referer}i\” \”%{User-Agent}i\”" combined
LogFormat “%{X-Forwarded-For}i %l %u %t \”%r\” %>s %b” common
LogFormat “%{Referer}i -> %U” referer
LogFormat “%{User-agent}i” agent
[...]

We’ll configure HAProxy to check the web servers’ health by continuously requesting the file “check.txt” from the web servers. To keep the logs small, we’ll customize the first vhost on each web server (HAProxy will use the web servers’ IP adresses to request the file – so the first vhost will answer the request) to ensure that the access to “check.txt” won’t be logged. In this example the vhosts are configured in

/etc/httpd/conf.d/vhosts.conf“.

Add the following line to the configuration of your first vhost …

SetEnvIf Request_URI “^/check\.txt$” dontlog

… and add the exception (env=!dontlog) to the line for the CustomLog. For example, the configuration for the first vhost could look like this:

NameVirtualHost 192.168.0.112:80
 
<VirtualHost 192.168.0.112:80>
ServerName health.example.com
ServerAdmin admin@example.com
DocumentRoot /var/www/haproxy
 
SetEnvIf Request_URI "^/check\.txt$" dontlog
LogLevel warn
ErrorLog /var/log/httpd/vhost_error.log
CustomLog /var/log/httpd/vhost_access.log combined env=!dontlog
</VirtualHost>

Now we have to create the file “check.txt” (this file can be empty) within the document root of the first vhost.

touch /var/www/haproxy/check.txt

Afterwards the configuration of the web servers is completed – restart the web servers.

/etc/init.d/httpd restart

Firewall Configuration In LB1 and LBb2

In order that HTTP & HTTPS connections can be forwarded to the web servers and the heartbeat daemons can communicate with each other you have to open the corresponding ports on both load balancers.

system-config-securitylevel-tui

Set HTTP & HTTPS as trusted service and insert the heartbeat-port (694 udp) into the section “Other Ports” as shown on the screenshot below. After that save the settings.

Needed Packages On Both Load Balancers

Install the needed packages via:

yum -y install haproxy heartbeat

3.3 HAProxy Configuration

cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg_orig
cat /dev/null > /etc/haproxy/haproxy.cfg
vi /etc/haproxy/haproxy.cfg

The content should look like this on both load balancers.

global
log 127.0.0.1   local0
log 127.0.0.1   local1 notice
#log loghost    local0 info
maxconn 4096
#debug
#quiet
user haproxy
group haproxy
 
defaults
log     global
mode    http
option  httplog
option  dontlognull
retries 3
redispatch
maxconn 2000
contimeout      5000
clitimeout      50000
srvtimeout      50000
 
listen webfarm 192.168.10.206:80
mode http
stats enable
stats auth someuser:somepassword
balance roundrobin
cookie JSESSIONID prefix
option httpclose
option forwardfor
option httpchk HEAD /check.txt HTTP/1.0
server webA 192.168.10.100:80 cookie A check
server webB 192.168.10.72:80 cookie B check

Note: If you want to know more about the available options to configure HAProxy, you should take a look at http://haproxy.1wt.eu/download/1.3/doc/haproxy-en.txt and http://haproxy.1wt.eu/download/1.2/doc/architecture.txt.

Heartbeat Configuration

On Both Load Balancers

Heartbeat will tell LB1 & LB2 that they should listen on the shared IP (192.168.10.206). First we have to allow HAProxy to bind to the shared IP.

vi /etc/sysctl.conf

Add the following lines to the file …

# Allow HAProxy shared IP
net.ipv4.ip_nonlocal_bind = 1

… and run:

sysctl -p

Now we have to create three configuration files for heartbeat.

vi /etc/ha.d/authkeys

The content should look like this – replace %auth_password% with a password of your choice. The heartbeat daemons on the both load balancers will use this password to authenticate against each other (so it should be a very secure password).

auth 3
3 md5 authpassword

Change the rights so that only root is allowed to access the file.

chmod 600 /etc/ha.d/authkeys

vi /etc/ha.d/haresources

The content should look like this (on both load balancers!) – the first word is the output of

uname -n

on load balancer 1.

lb1.example.com 192.168.10.206
 

On Load Balancer 1 (LB1)

vi /etc/ha.d/ha.cf

The content should look like this – the last two lines contain the output of uname -n from both load balancers!:

#
#       keepalive: how many seconds between heartbeats
#
keepalive 2
#
#       deadtime: seconds-to-declare-host-dead
#
deadtime 10
#
#       What UDP port to use for udp or ppp-udp communication?
#
udpport        694
bcast  eth0
mcast eth0 225.0.0.1 694 1 0
ucast eth0 192.168.10.54
#       What interfaces to heartbeat over?
udp     eth0
#
#       Facility to use for syslog()/logger (alternative to log/debugfile)
#
logfacility     local0
#
#       Tell what machines are in the cluster
#       node    nodename ...    -- must match uname -n
node    lb1.example.com
node    lb2.example.com
 

On Load Balancer 2 (LB2)

vi /etc/ha.d/ha.cf

The content should look like this – the last two lines contain the output of uname -n from both load balancers!:

#
#       keepalive: how many seconds between heartbeats
#
keepalive 2
#
#       deadtime: seconds-to-declare-host-dead
#
deadtime 10
#
#       What UDP port to use for udp or ppp-udp communication?
#
udpport        694
bcast  eth0
mcast eth0 225.0.0.1 694 1 0
ucast eth0 192.168.10.91
#       What interfaces to heartbeat over?
udp     eth0
#
#       Facility to use for syslog()/logger (alternative to log/debugfile)
#
logfacility     local0
#
#       Tell what machines are in the cluster
#       node    nodename ...    -- must match uname -n
node    lb1.example.com
node    lb2.example.com

Afterwards start heartbeat on both load balancers.

/etc/init.d/heartbeat start

Check Heartbeat On LB1

If all went well, the output of …

ip addr sh eth0

… should also contain the shared IP – it’s the active load balancer.

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:0c:29:02:ae:eb brd ff:ff:ff:ff:ff:ff
inet 192.168.10.54/24 brd 192.168.10.255 scope global eth0
inet 192.168.10.206/24 brd 192.168.10.255 scope global secondary eth0:0
inet6 fe80::20c:29ff:fe02:aeeb/64 scope link
valid_lft forever preferred_lft forever

Check Heartbeat On LB2

If all went well, the output of …

ip addr sh eth0

… should not contain the shared IP as long as load balancer 1 is up – it’s the passive load balancer.

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:0c:29:e6:66:18 brd ff:ff:ff:ff:ff:ff
inet 192.168.10.91/24 brd 192.168.0.255 scope global eth1
inet6 fe80::20c:29ff:fee6:6618/64 scope link
valid_lft forever preferred_lft forever

Now we can add HAProxy to autostart and start HAProxy on both load balancers.

chkconfig –level 3 haproxy on
/etc/init.d/haproxy start

Web Server

Shut down one of the both web servers and make a HTTP request to the shared IP 192.168.10.206 (or to any domain/hostname that is pointing to the shared IP) – you should get content from the remaining web server.

Load Balancer

Shut down the active load balancer (LB1) – the passive loadbalancer (LB2) should take over immediately. The output of …

ip addr sh eth0

… on the second load balancer (LB) should now also contain the shared ip.

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:0c:29:e6:66:18 brd ff:ff:ff:ff:ff:ff
inet 192.168.10.91/24 brd 192.168.10.255 scope global eth1
inet 192.168.10.206/24 brd 192.168.10.255 scope global secondary eth1:0
inet6 fe80::20c:29ff:fee6:6618/64 scope link
valid_lft forever preferred_lft forever

When the first load balancer (LB1) is up again, it will take over the active role again.

HAProxy Statistics

HAProxy provides a webinterface for statistics. You can access it via http://192.168.10.206/haproxy?stats within your preferred browser. Log in with the data you configured in the HAProxy configuration file (in this example you can log in with the username someuser and the password somepassword (both without the quotes). If you don’t want/need statistics, simply remove the lines that begin with “stats” within the HAProxy configuration file on both load balancers.

For more technical articles, news and forum discussion logon

Advertisements
Categories: Linux
  1. ji
    January 29, 2009 at 6:28 am

    you not mentioned this ip “NameVirtualHost 192.168.0.112:80 ” in overview

  2. Dr. Kenneth Noisewater
    September 8, 2010 at 10:01 pm

    Alternatively, if you have hardware beefy enough for haproxy to run on 2 separate IPs, you can have an active-active cluster, whereby one haproxy IP is preferred on each host, and can fail over to its cluster partner (which then handle both IPs until the failed partner is brought back up). Given that haproxy is single-threaded, any modern quad-core single CPU, or even the Atom dual-cores, should be able to handle this pretty easily up into the hundreds of megabits/sec.

    I had success doing this and simply having round-robin A records (with www. being a CNAME to the domain A recs), just using round-robin DNS I got a roughly 50/50 spread on the 2 haproxy instances.

  3. January 9, 2012 at 6:19 pm

    Hi,
    heartbeat implementation only works if master node goes down. Can we implement heartbeat solution for if haproxy processes goes down on master node then slave node automatically pickup?

  1. January 24, 2010 at 3:30 pm

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: