Archive

Archive for May, 2008

Taking Backup and restore by using System rescue cd

May 1, 2008 1 comment

This Post has been moved to http://opensupportindia.com

For more technical articles, news and forum discussion logon

Categories: Linux

How to setup an NTP Server?

May 1, 2008 1 comment

The Network Time Protocol is defined in RFC1305 and allows the transfer and maintenance of time functions over distributed network systems. One of the most widely used NTP servers is ntpd (ntp.isc.org), which also provides Simple NTP (RFC2030) and is a common package of most Linux distributions. The NTP server figures out how much the system clock drifts and smoothly corrects it with delicate accuracy, as opposed to large adjustments once every few hours.

The following chapter details how to configure the local NTP daemon to access external time servers and to provide the clients on the internal network the ability to synchronise from the server.

Basic Configuration

Just like any time piece, the NTP server maintains a time service where the synchronisation and maintenance of time is paramount. Before any time services are configured, its important that the server should have an initial state which has been synchronised with another time source reference. The following command will synchronise the local system time against another server, ensuring nearest possible time is available before configuring the NTP server.

[bash]# ntpdate -b pool.ntp.org

Before we adjust any configuration files, its always recommended that we make a backup of the original in case things go wrong, then we can edit the file and make changes as required.

[bash]# cp /etc/ntp.conf /etc/ntp.conf.original
[bash]# vi /etc/ntp.conf

Finding a Time Source

One of the most difficult issues that people face with NTP is finding a time server that allows home and small office users to publicly synchronise off them. Enter the NTP Pooling Project located at http://www.pool.ntp.org. The NTP Pool is a collection of over 220 publicly accessible NTP servers distributed throughout different regions of the world. The DNS records for the NTP Pool are rotated hourly with different servers being allocated into each pool and region.

The advantages of the NTP Pool are:

  • that all the available servers will load balance,
  • you don’t need to spend hours hunting for any public servers, and
  • you only need to remember one set of records for all the servers.

The default configuration for ntpd servers after version 4.2 uses the NTP Pool for the default server sources.

server 0.pool.ntp.org
server 1.pool.ntp.org
server 2.pool.ntp.org

If you have access to other NTP servers which are geographically closer and you can synchronise from them, you should substitute the server values above. Alternately the NTP Pools are also broken into geographical Pools which may serve as a quicker time source, see here: http://ntp.isc.org/bin/view/Servers/NTPPoolServers

Access Controls

The NTP server is a network application that provides a resource to other networked systems and clients, as such we need to ensure that some security measures are enforced. The NTP configuration has adjustable access controls that define all the default controls and those for allowable clients and remote servers.

The following restrict statement defines the suggested access controls for all default connections.

restrict default kod nomodify notrap noquery nopeer

The following table lists and defines some of the more commonly used access control parameters.

Parameters Definitions
ignore Deny all packets and queries
kod Send Kiss-Of-Death packet on access violation
nomodify Deny ntpq / ntpdc queries that attempt to modify the server
notrap Deny control message trap service
noquery Deny all ntpq / ntpdc queries
noserve Deny all queries – except ntpq / ntpdc
notrust Deny access unless cryptographically authenticated (ver 4.2 onwards)
nopeer Deny all packets that attempt to establish a peer association
Caution !! In NTP versions prior to 4.2, the notrust option meant not to trust a server/host for time. In NTP versions 4.2 and later, the notrust option means cryptographic authentication is required before believing the server/host. Unless using cryptography, do not use the notrust option, your client requests will fail.

To allow full control to the localhost, add the following entry to the configuration.

restrict 127.0.0.1

The NTP Pool servers have been listed as a time source already (ver 4.2 onwards), and they too need restrictions applied so the local server can synchronise from them. Ensure the access control parameters are strict enough that the remote servers can only be used for queries.

restrict 0.pool.ntp.org mask 255.255.255.255 nomodify notrap noquery
restrict 1.pool.ntp.org mask 255.255.255.255 nomodify notrap noquery
restrict 2.pool.ntp.org mask 255.255.255.255 nomodify notrap noquery

To allow all the workstations inside the internal private network to be able to query the time from your server, use the following access control rule (adjust subnet if needed).

restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap

Finally we need the following declarations in the /etc/ntp.conf file.

server 127.127.1.0 # local clock
fudge 127.127.1.0 stratum 10
driftfile /var/lib/ntp/drift
broadcastdelay 0.008
keys /etc/ntp/keys

The above configuration parameters are as follows:

server Specifies that a server is running on the host (own local clock)
fudge Passes additional information to the clock driver
stratum 10 Manually sets the Stratum the server should operate at (1-15)
driftfile Specifies the location of the frequency file
broadcastdelay Sets the propagation delay from the server when broadcasting
keys Store a list of keys needed for any cryptographic links

The Strata

The world of NTP is hierarchical with the primary servers at the top keeping the master time, and distributing the time down to the secondary servers and so forth until your little workstation synchronises in the corner office. Each server participating in the hierarchy are allocated a stratum, with stratum 1 being the master servers, stratum 2 the secondary servers, down to the lower end of stratum 15. A stratum 1 server uses an external time source (GPS, etc..) which is introduced into the server and then used to propagate the time signals. Stratum 2 servers draw their time from the higher (1) stratum servers.

When determining the stratum of your server, firstly consider who you are providing time to? If you are only using the system for yourself and passing it on to a few workstations, then your stratum can be safely left at 10. If you are using the system for a large scale network, then plan your time servers and strata effectively.

Starting NTP

The server is now fully configured and ready to start. If you have not already done an initial synchronisation of time (before running the daemon), you should do so now. The initial sync only needs to be done once before the server is started for the first time, not each time it starts.

[bash]# ntpdate -b pool.ntp.org

You should now set the runlevels required for the ntpd service, then restart it.

[bash]# chkconfig –level 2345 ntpd on
[bash]# /etc/init.d/ntpd restart
Note !! The NTP server uses UDP packets to query time servers on port 123. Depending on your Linux configuration, the initscripts for the ntpd service may have iptables commands to allow ntpd to access the external time servers.

You can check which runlevels the service will be active with the following command.

[bash]# chkconfig –list ntpd

To see if the service started successfully, you should check the system log file.

[bash]# grep ntpd /var/log/messages
galaxy ntpd[1110]: ntpd 4.2.0a@1.1196-r Thu Feb 23 04:42:00 EST 2006 (1)
galaxy ntpd[1110]: precision = 2.000 usec
galaxy ntpd[1110]: Listening on interface wildcard, 0.0.0.0#123
galaxy ntpd[1110]: Listening on interface wildcard, ::#123
galaxy ntpd[1110]: Listening on interface lo, 127.0.0.1#123
galaxy ntpd[1110]: Listening on interface eth0, 192.168.1.1#123
galaxy ntpd[1110]: kernel time sync status 0040
galaxy ntpd[1110]: frequency initialized 0.000 PPM from /var/lib/ntp/drift

You can now query the NTP server with the ntpq (query) tool. The output display after ntpd has been (re)started will be similar to the first table. As ntpd is allowed to run for a while, the table will start to fill with synchronisation details.

[bash]# ntpq -pn
remote refid st t when poll reach delay offset jitter
==============================================================================
80.26.104.184 .INIT. 16 u – 64 0 0.000 0.000 4000.00
128.95.231.7 .INIT. 16 u – 64 0 0.000 0.000 4000.00
64.112.189.11 .INIT. 16 u – 64 0 0.000 0.000 4000.00
127.127.1.0 LOCAL(0) 10 l – 64 0 0.000 0.000 4000.00
remote refid st t when poll reach delay offset jitter
==============================================================================
*80.26.104.184 217.127.32.90 2 u 66 256 377 470.247 32.058 33.497
+128.95.231.7 140.142.2.8 3 u 254 256 377 217.646 -3.832 2.734
+64.112.189.11 128.10.252.6 2 u 2 256 377 258.208 2.395 47.246
127.127.1.0 LOCAL(0) 10 l 56 64 377 0.000 0.000 0.002

The above output shows a properly synchronised time server drawing from NTP Pool allocated sources. You will notice our server is running at a stratum of 10.

Note !! Your internal workstation computers will not be able to use the server as a synchronisation source until the LOCAL(0) clock has stable time. This may take up to 15 minutes after starting the NTP daemon.

The server can be tested from another Linux workstation by issuing the following synchronisation command.

[bash]# ntpdate 192.168.1.1

If the client computer does not get the time from the server, check that the server and client have access through any firewall settings.

Client Configuration

Before any client can successfully synchronise with the NTP server, the server’s time must be stable. It may take a server up to 15 minutes before it can be used as a time source, after ntpd has been (re)started.

Linux Client

To configure a Linux client to use the new server as a time source, the configuration file for the client should at least contain the following entries.

[bash]# vi /etc/ntp.conf
server 192.168.1.1
restrict default ignore
restrict 127.0.0.1
restrict 192.168.1.1 mask 255.255.255.255 nomodify notrap noquery
driftfile /var/lib/ntp/drift

The Linux client will also need to have the ntpd service started.

Microsoft Client

To configure a Microsoft Windows (XP) client, follow the sequence of commands below.

Note !! These commands are tested on Windows XP, and may not be specific to every version of Windows.
C:\>net time /setsntp:192.168.1.1
The command completed successfully.
C:\>net time /querysntp
The current SNTP value is: 192.168.1.1The command completed successfully.
C:\>net stop w32time && net start w32time
The Windows Time service is stopping.
The Windows Time service was stopped successfully.The Windows Time service is starting.
The Windows Time service was started successfully.

The standard time query interval for Windows (XP) is one query every 7 days, which for time critical applications and environments is ineffective. To adjust the time interval for Windows (XP), a registry value needs to be adjusted.

Windows (XP) stores the following registry key in a hexadecimal format, which converted to decimal will amount to the time in seconds between time queries. Select the new time (in seconds) that you require the Windows (XP) system to poll the server, then convert it to hexadecimal (86400 seconds is 1 day). This should be the “DWORD” value.

Warning !! Adjusting the Windows registry may cause your computer system to become unstable, do so at your own risk.
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\w32time\TimeProviders\NtpClient]
“SpecialPollInterval”=dword:00093a80

The “Windows Time” service in Windows (XP) should be set to start automatically on each system boot. The event log should be viewed for any errors that may occur. See this article for more information: http://support.microsoft.com/kb/323621

For more technical articles, news and forum discussion logon

Note !! If the service does not appear to be synchronising with the Linux server, ensure that the firewall is allowing any required connections.
Categories: Linux

High availability(cluster) with Load balancing by using Heartbeat and Haproxy

May 1, 2008 4 comments

This document describes how to set up a two-node load balancer in an active/passive configuration with HAProxy and heartbeat on Fedora 7. The load balancer acts between the user and two (or more) Apache web servers that hold the same content. The load balancer passes the requests to the web servers and it also checks their health. If one of them is down, all requests will automatically be redirected to the remaining web server(s). In addition to that, the two load balancer nodes monitor each other using heartbeat. If the master fails, the slave becomes the master – users won’t notice any disruption of the service.

This how to is a practical guide without any warranty – it doesn’t cover the theoretical backgrounds. There are many ways to set up such a system – this is the way which I tested.

Preparation

For this how to I set up five Fedora 7 systems (minimal installation without gui etc.) with the following configuration:

Load Balancer 1

Hostname: lb1.example.com
IP: 192.168.10.54
Shared IP: 192.168.10.206

Load Balancer 2

Hostname: lb2.example.com
IP: 192.168.10.91
Shared IP: 192.168.10.206

Web Server 1

Hostname: http1.example.com
IP: 192.168.10.100

Web Server 2

Hostname: http2.example.com
IP: 192.168.10.72

Database Server

Hostname: db.example.com

IP: 192.168.10.65

Overview

+—————–+
| 192.168.10.206 |
| Shared IP |
+——–+——–+
|
+———————-+
| |
+——–+——–+ +——–+——–+
| 192.168.10.54 | | 192.168.10.91 |
| Load Balancer 1 | | Load Balancer 2 |
+——–+——–+ +——–+——–+

+——–+——–+ +——–+——–+
| 192.168.10.100 | | 192.168.10.72 |
| Web Server 1 | | Web Server 2 |
+—————–+ +—————–+

+—————–+
| 192.168.10.65 |
| Database IP |
+——–+——–+
|

Make sure to turn off the httpd server in both Load balancers

Make sure to turn on the Firewall in all the load balancers and web servers

Apache Configuration

HAProxy will work as a transparent proxy – so the user’s IP address will be passed in the field “X-Forwarded-For” to the web servers. In order that the web servers will log the user’s IP address and not the IP addresses of the load balancers we have to modify the log format within the apache configuration file on both web servers.

vi /etc/httpd/conf/httpd.conf

Search the Lines that begin with “LogFormat”

[...]
LogFormat “%h %l %u %t \”%r\” %>s %b \”%{Referer}i\” \”%{User-Agent}i\”" combined
LogFormat “%h %l %u %t \”%r\” %>s %b” common
LogFormat “%{Referer}i -> %U” referer
LogFormat “%{User-agent}i” agent
[...]
… and replace “%h” with “%{X-Forwarded-For}i“. The content should look like this:
[...]
LogFormat “%{X-Forwarded-For}i %l %u %t \”%r\” %>s %b \”%{Referer}i\” \”%{User-Agent}i\”" combined
LogFormat “%{X-Forwarded-For}i %l %u %t \”%r\” %>s %b” common
LogFormat “%{Referer}i -> %U” referer
LogFormat “%{User-agent}i” agent
[...]

We’ll configure HAProxy to check the web servers’ health by continuously requesting the file “check.txt” from the web servers. To keep the logs small, we’ll customize the first vhost on each web server (HAProxy will use the web servers’ IP adresses to request the file – so the first vhost will answer the request) to ensure that the access to “check.txt” won’t be logged. In this example the vhosts are configured in

/etc/httpd/conf.d/vhosts.conf“.

Add the following line to the configuration of your first vhost …

SetEnvIf Request_URI “^/check\.txt$” dontlog

… and add the exception (env=!dontlog) to the line for the CustomLog. For example, the configuration for the first vhost could look like this:

NameVirtualHost 192.168.0.112:80
 
<VirtualHost 192.168.0.112:80>
ServerName health.example.com
ServerAdmin admin@example.com
DocumentRoot /var/www/haproxy
 
SetEnvIf Request_URI "^/check\.txt$" dontlog
LogLevel warn
ErrorLog /var/log/httpd/vhost_error.log
CustomLog /var/log/httpd/vhost_access.log combined env=!dontlog
</VirtualHost>

Now we have to create the file “check.txt” (this file can be empty) within the document root of the first vhost.

touch /var/www/haproxy/check.txt

Afterwards the configuration of the web servers is completed – restart the web servers.

/etc/init.d/httpd restart

Firewall Configuration In LB1 and LBb2

In order that HTTP & HTTPS connections can be forwarded to the web servers and the heartbeat daemons can communicate with each other you have to open the corresponding ports on both load balancers.

system-config-securitylevel-tui

Set HTTP & HTTPS as trusted service and insert the heartbeat-port (694 udp) into the section “Other Ports” as shown on the screenshot below. After that save the settings.

Needed Packages On Both Load Balancers

Install the needed packages via:

yum -y install haproxy heartbeat

3.3 HAProxy Configuration

cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg_orig
cat /dev/null > /etc/haproxy/haproxy.cfg
vi /etc/haproxy/haproxy.cfg

The content should look like this on both load balancers.

global
log 127.0.0.1   local0
log 127.0.0.1   local1 notice
#log loghost    local0 info
maxconn 4096
#debug
#quiet
user haproxy
group haproxy
 
defaults
log     global
mode    http
option  httplog
option  dontlognull
retries 3
redispatch
maxconn 2000
contimeout      5000
clitimeout      50000
srvtimeout      50000
 
listen webfarm 192.168.10.206:80
mode http
stats enable
stats auth someuser:somepassword
balance roundrobin
cookie JSESSIONID prefix
option httpclose
option forwardfor
option httpchk HEAD /check.txt HTTP/1.0
server webA 192.168.10.100:80 cookie A check
server webB 192.168.10.72:80 cookie B check

Note: If you want to know more about the available options to configure HAProxy, you should take a look at http://haproxy.1wt.eu/download/1.3/doc/haproxy-en.txt and http://haproxy.1wt.eu/download/1.2/doc/architecture.txt.

Heartbeat Configuration

On Both Load Balancers

Heartbeat will tell LB1 & LB2 that they should listen on the shared IP (192.168.10.206). First we have to allow HAProxy to bind to the shared IP.

vi /etc/sysctl.conf

Add the following lines to the file …

# Allow HAProxy shared IP
net.ipv4.ip_nonlocal_bind = 1

… and run:

sysctl -p

Now we have to create three configuration files for heartbeat.

vi /etc/ha.d/authkeys

The content should look like this – replace %auth_password% with a password of your choice. The heartbeat daemons on the both load balancers will use this password to authenticate against each other (so it should be a very secure password).

auth 3
3 md5 authpassword

Change the rights so that only root is allowed to access the file.

chmod 600 /etc/ha.d/authkeys

vi /etc/ha.d/haresources

The content should look like this (on both load balancers!) – the first word is the output of

uname -n

on load balancer 1.

lb1.example.com 192.168.10.206
 

On Load Balancer 1 (LB1)

vi /etc/ha.d/ha.cf

The content should look like this – the last two lines contain the output of uname -n from both load balancers!:

#
#       keepalive: how many seconds between heartbeats
#
keepalive 2
#
#       deadtime: seconds-to-declare-host-dead
#
deadtime 10
#
#       What UDP port to use for udp or ppp-udp communication?
#
udpport        694
bcast  eth0
mcast eth0 225.0.0.1 694 1 0
ucast eth0 192.168.10.54
#       What interfaces to heartbeat over?
udp     eth0
#
#       Facility to use for syslog()/logger (alternative to log/debugfile)
#
logfacility     local0
#
#       Tell what machines are in the cluster
#       node    nodename ...    -- must match uname -n
node    lb1.example.com
node    lb2.example.com
 

On Load Balancer 2 (LB2)

vi /etc/ha.d/ha.cf

The content should look like this – the last two lines contain the output of uname -n from both load balancers!:

#
#       keepalive: how many seconds between heartbeats
#
keepalive 2
#
#       deadtime: seconds-to-declare-host-dead
#
deadtime 10
#
#       What UDP port to use for udp or ppp-udp communication?
#
udpport        694
bcast  eth0
mcast eth0 225.0.0.1 694 1 0
ucast eth0 192.168.10.91
#       What interfaces to heartbeat over?
udp     eth0
#
#       Facility to use for syslog()/logger (alternative to log/debugfile)
#
logfacility     local0
#
#       Tell what machines are in the cluster
#       node    nodename ...    -- must match uname -n
node    lb1.example.com
node    lb2.example.com

Afterwards start heartbeat on both load balancers.

/etc/init.d/heartbeat start

Check Heartbeat On LB1

If all went well, the output of …

ip addr sh eth0

… should also contain the shared IP – it’s the active load balancer.

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:0c:29:02:ae:eb brd ff:ff:ff:ff:ff:ff
inet 192.168.10.54/24 brd 192.168.10.255 scope global eth0
inet 192.168.10.206/24 brd 192.168.10.255 scope global secondary eth0:0
inet6 fe80::20c:29ff:fe02:aeeb/64 scope link
valid_lft forever preferred_lft forever

Check Heartbeat On LB2

If all went well, the output of …

ip addr sh eth0

… should not contain the shared IP as long as load balancer 1 is up – it’s the passive load balancer.

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:0c:29:e6:66:18 brd ff:ff:ff:ff:ff:ff
inet 192.168.10.91/24 brd 192.168.0.255 scope global eth1
inet6 fe80::20c:29ff:fee6:6618/64 scope link
valid_lft forever preferred_lft forever

Now we can add HAProxy to autostart and start HAProxy on both load balancers.

chkconfig –level 3 haproxy on
/etc/init.d/haproxy start

Web Server

Shut down one of the both web servers and make a HTTP request to the shared IP 192.168.10.206 (or to any domain/hostname that is pointing to the shared IP) – you should get content from the remaining web server.

Load Balancer

Shut down the active load balancer (LB1) – the passive loadbalancer (LB2) should take over immediately. The output of …

ip addr sh eth0

… on the second load balancer (LB) should now also contain the shared ip.

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:0c:29:e6:66:18 brd ff:ff:ff:ff:ff:ff
inet 192.168.10.91/24 brd 192.168.10.255 scope global eth1
inet 192.168.10.206/24 brd 192.168.10.255 scope global secondary eth1:0
inet6 fe80::20c:29ff:fee6:6618/64 scope link
valid_lft forever preferred_lft forever

When the first load balancer (LB1) is up again, it will take over the active role again.

HAProxy Statistics

HAProxy provides a webinterface for statistics. You can access it via http://192.168.10.206/haproxy?stats within your preferred browser. Log in with the data you configured in the HAProxy configuration file (in this example you can log in with the username someuser and the password somepassword (both without the quotes). If you don’t want/need statistics, simply remove the lines that begin with “stats” within the HAProxy configuration file on both load balancers.

For more technical articles, news and forum discussion logon

Categories: Linux