High availability(cluster) with Load balancing by using Heartbeat and Haproxy

May 1, 2008 4 comments

This document describes how to set up a two-node load balancer in an active/passive configuration with HAProxy and heartbeat on Fedora 7. The load balancer acts between the user and two (or more) Apache web servers that hold the same content. The load balancer passes the requests to the web servers and it also checks their health. If one of them is down, all requests will automatically be redirected to the remaining web server(s). In addition to that, the two load balancer nodes monitor each other using heartbeat. If the master fails, the slave becomes the master – users won’t notice any disruption of the service.

This how to is a practical guide without any warranty – it doesn’t cover the theoretical backgrounds. There are many ways to set up such a system – this is the way which I tested.

Preparation

For this how to I set up five Fedora 7 systems (minimal installation without gui etc.) with the following configuration:

Load Balancer 1

Hostname: lb1.example.com
IP: 192.168.10.54
Shared IP: 192.168.10.206

Load Balancer 2

Hostname: lb2.example.com
IP: 192.168.10.91
Shared IP: 192.168.10.206

Web Server 1

Hostname: http1.example.com
IP: 192.168.10.100

Web Server 2

Hostname: http2.example.com
IP: 192.168.10.72

Database Server

Hostname: db.example.com

IP: 192.168.10.65

Overview

+—————–+
| 192.168.10.206 |
| Shared IP |
+——–+——–+
|
+———————-+
| |
+——–+——–+ +——–+——–+
| 192.168.10.54 | | 192.168.10.91 |
| Load Balancer 1 | | Load Balancer 2 |
+——–+——–+ +——–+——–+

+——–+——–+ +——–+——–+
| 192.168.10.100 | | 192.168.10.72 |
| Web Server 1 | | Web Server 2 |
+—————–+ +—————–+

+—————–+
| 192.168.10.65 |
| Database IP |
+——–+——–+
|

Make sure to turn off the httpd server in both Load balancers

Make sure to turn on the Firewall in all the load balancers and web servers

Apache Configuration

HAProxy will work as a transparent proxy – so the user’s IP address will be passed in the field “X-Forwarded-For” to the web servers. In order that the web servers will log the user’s IP address and not the IP addresses of the load balancers we have to modify the log format within the apache configuration file on both web servers.

vi /etc/httpd/conf/httpd.conf

Search the Lines that begin with “LogFormat”

[...]
LogFormat “%h %l %u %t \”%r\” %>s %b \”%{Referer}i\” \”%{User-Agent}i\”" combined
LogFormat “%h %l %u %t \”%r\” %>s %b” common
LogFormat “%{Referer}i -> %U” referer
LogFormat “%{User-agent}i” agent
[...]
… and replace “%h” with “%{X-Forwarded-For}i“. The content should look like this:
[...]
LogFormat “%{X-Forwarded-For}i %l %u %t \”%r\” %>s %b \”%{Referer}i\” \”%{User-Agent}i\”" combined
LogFormat “%{X-Forwarded-For}i %l %u %t \”%r\” %>s %b” common
LogFormat “%{Referer}i -> %U” referer
LogFormat “%{User-agent}i” agent
[...]

We’ll configure HAProxy to check the web servers’ health by continuously requesting the file “check.txt” from the web servers. To keep the logs small, we’ll customize the first vhost on each web server (HAProxy will use the web servers’ IP adresses to request the file – so the first vhost will answer the request) to ensure that the access to “check.txt” won’t be logged. In this example the vhosts are configured in

/etc/httpd/conf.d/vhosts.conf“.

Add the following line to the configuration of your first vhost …

SetEnvIf Request_URI “^/check\.txt$” dontlog

… and add the exception (env=!dontlog) to the line for the CustomLog. For example, the configuration for the first vhost could look like this:

NameVirtualHost 192.168.0.112:80
 
<VirtualHost 192.168.0.112:80>
ServerName health.example.com
ServerAdmin admin@example.com
DocumentRoot /var/www/haproxy
 
SetEnvIf Request_URI "^/check\.txt$" dontlog
LogLevel warn
ErrorLog /var/log/httpd/vhost_error.log
CustomLog /var/log/httpd/vhost_access.log combined env=!dontlog
</VirtualHost>

Now we have to create the file “check.txt” (this file can be empty) within the document root of the first vhost.

touch /var/www/haproxy/check.txt

Afterwards the configuration of the web servers is completed – restart the web servers.

/etc/init.d/httpd restart

Firewall Configuration In LB1 and LBb2

In order that HTTP & HTTPS connections can be forwarded to the web servers and the heartbeat daemons can communicate with each other you have to open the corresponding ports on both load balancers.

system-config-securitylevel-tui

Set HTTP & HTTPS as trusted service and insert the heartbeat-port (694 udp) into the section “Other Ports” as shown on the screenshot below. After that save the settings.

Needed Packages On Both Load Balancers

Install the needed packages via:

yum -y install haproxy heartbeat

3.3 HAProxy Configuration

cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg_orig
cat /dev/null > /etc/haproxy/haproxy.cfg
vi /etc/haproxy/haproxy.cfg

The content should look like this on both load balancers.

global
log 127.0.0.1   local0
log 127.0.0.1   local1 notice
#log loghost    local0 info
maxconn 4096
#debug
#quiet
user haproxy
group haproxy
 
defaults
log     global
mode    http
option  httplog
option  dontlognull
retries 3
redispatch
maxconn 2000
contimeout      5000
clitimeout      50000
srvtimeout      50000
 
listen webfarm 192.168.10.206:80
mode http
stats enable
stats auth someuser:somepassword
balance roundrobin
cookie JSESSIONID prefix
option httpclose
option forwardfor
option httpchk HEAD /check.txt HTTP/1.0
server webA 192.168.10.100:80 cookie A check
server webB 192.168.10.72:80 cookie B check

Note: If you want to know more about the available options to configure HAProxy, you should take a look at http://haproxy.1wt.eu/download/1.3/doc/haproxy-en.txt and http://haproxy.1wt.eu/download/1.2/doc/architecture.txt.

Heartbeat Configuration

On Both Load Balancers

Heartbeat will tell LB1 & LB2 that they should listen on the shared IP (192.168.10.206). First we have to allow HAProxy to bind to the shared IP.

vi /etc/sysctl.conf

Add the following lines to the file …

# Allow HAProxy shared IP
net.ipv4.ip_nonlocal_bind = 1

… and run:

sysctl -p

Now we have to create three configuration files for heartbeat.

vi /etc/ha.d/authkeys

The content should look like this – replace %auth_password% with a password of your choice. The heartbeat daemons on the both load balancers will use this password to authenticate against each other (so it should be a very secure password).

auth 3
3 md5 authpassword

Change the rights so that only root is allowed to access the file.

chmod 600 /etc/ha.d/authkeys

vi /etc/ha.d/haresources

The content should look like this (on both load balancers!) – the first word is the output of

uname -n

on load balancer 1.

lb1.example.com 192.168.10.206
 

On Load Balancer 1 (LB1)

vi /etc/ha.d/ha.cf

The content should look like this – the last two lines contain the output of uname -n from both load balancers!:

#
#       keepalive: how many seconds between heartbeats
#
keepalive 2
#
#       deadtime: seconds-to-declare-host-dead
#
deadtime 10
#
#       What UDP port to use for udp or ppp-udp communication?
#
udpport        694
bcast  eth0
mcast eth0 225.0.0.1 694 1 0
ucast eth0 192.168.10.54
#       What interfaces to heartbeat over?
udp     eth0
#
#       Facility to use for syslog()/logger (alternative to log/debugfile)
#
logfacility     local0
#
#       Tell what machines are in the cluster
#       node    nodename ...    -- must match uname -n
node    lb1.example.com
node    lb2.example.com
 

On Load Balancer 2 (LB2)

vi /etc/ha.d/ha.cf

The content should look like this – the last two lines contain the output of uname -n from both load balancers!:

#
#       keepalive: how many seconds between heartbeats
#
keepalive 2
#
#       deadtime: seconds-to-declare-host-dead
#
deadtime 10
#
#       What UDP port to use for udp or ppp-udp communication?
#
udpport        694
bcast  eth0
mcast eth0 225.0.0.1 694 1 0
ucast eth0 192.168.10.91
#       What interfaces to heartbeat over?
udp     eth0
#
#       Facility to use for syslog()/logger (alternative to log/debugfile)
#
logfacility     local0
#
#       Tell what machines are in the cluster
#       node    nodename ...    -- must match uname -n
node    lb1.example.com
node    lb2.example.com

Afterwards start heartbeat on both load balancers.

/etc/init.d/heartbeat start

Check Heartbeat On LB1

If all went well, the output of …

ip addr sh eth0

… should also contain the shared IP – it’s the active load balancer.

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:0c:29:02:ae:eb brd ff:ff:ff:ff:ff:ff
inet 192.168.10.54/24 brd 192.168.10.255 scope global eth0
inet 192.168.10.206/24 brd 192.168.10.255 scope global secondary eth0:0
inet6 fe80::20c:29ff:fe02:aeeb/64 scope link
valid_lft forever preferred_lft forever

Check Heartbeat On LB2

If all went well, the output of …

ip addr sh eth0

… should not contain the shared IP as long as load balancer 1 is up – it’s the passive load balancer.

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:0c:29:e6:66:18 brd ff:ff:ff:ff:ff:ff
inet 192.168.10.91/24 brd 192.168.0.255 scope global eth1
inet6 fe80::20c:29ff:fee6:6618/64 scope link
valid_lft forever preferred_lft forever

Now we can add HAProxy to autostart and start HAProxy on both load balancers.

chkconfig –level 3 haproxy on
/etc/init.d/haproxy start

Web Server

Shut down one of the both web servers and make a HTTP request to the shared IP 192.168.10.206 (or to any domain/hostname that is pointing to the shared IP) – you should get content from the remaining web server.

Load Balancer

Shut down the active load balancer (LB1) – the passive loadbalancer (LB2) should take over immediately. The output of …

ip addr sh eth0

… on the second load balancer (LB) should now also contain the shared ip.

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:0c:29:e6:66:18 brd ff:ff:ff:ff:ff:ff
inet 192.168.10.91/24 brd 192.168.10.255 scope global eth1
inet 192.168.10.206/24 brd 192.168.10.255 scope global secondary eth1:0
inet6 fe80::20c:29ff:fee6:6618/64 scope link
valid_lft forever preferred_lft forever

When the first load balancer (LB1) is up again, it will take over the active role again.

HAProxy Statistics

HAProxy provides a webinterface for statistics. You can access it via http://192.168.10.206/haproxy?stats within your preferred browser. Log in with the data you configured in the HAProxy configuration file (in this example you can log in with the username someuser and the password somepassword (both without the quotes). If you don’t want/need statistics, simply remove the lines that begin with “stats” within the HAProxy configuration file on both load balancers.

For more technical articles, news and forum discussion logon

Advertisements
Categories: Linux

Setting Up A High-Availability Load Balancer (Cluster) with heartbeat

April 18, 2008 Leave a comment

This document describes how to set up a two-node load balancer in an active/passive configuration with HAProxy and heartbeat on Fedora 8. The load balancer acts between the user and two (or more) Apache web servers that hold the same content. The load balancer passes the requests to the web servers and it also checks their health. If one of them is down, all requests will automatically be redirected to the remaining web server(s). In addition to that, the two load balancer nodes monitor each other using heartbeat. If the master fails, the slave becomes the master – users won’t notice any disruption of the service. HAProxy is session-aware – you can use it with any web application that makes use of sessions like forums, shopping carts, etc.

From the HAProxy web site: “HAProxy is a free, very fast and reliable solution offering high availability, load balancing, and proxying for TCP and HTTP-based applications. It is particularly suited for web sites crawling under very high loads while needing persistence or Layer7 processing. Supporting tens of thousands of connections is clearly realistic with todays hardware. Its mode of operation makes its integration into existing architectures very easy and riskless, while still offering the possibility not to expose fragile web servers to the Net.”

This howto is a practical guide without any warranty – it doesn’t cover the theoretical backgrounds. There are many ways to set up such a system – this is the way I chose.

1 Preparation

For this howto I set up four Fedora 8 systems (minimal installation without gui etc.) with the following configuration:

1.1 Load Balancer 1

Hostname: lb1.example.com
IP: 192.168.0.110
Shared IP: 192.168.0.120

1.2 Load Balancer 2

Hostname: lb2.example.com
IP: 192.168.0.111
Shared IP: 192.168.0.120

1.3 Web Server 1

Hostname: http1.example.com
IP: 192.168.0.112

1.4 Web Server 2

Hostname: http2.example.com
IP: 192.168.0.113

1.5 Overview

+—————–+
| 192.168.0.120 |
| Shared IP |
+——–+——–+
|
+———————-+
| |
+——–+——–+ +——–+——–+
| 192.168.0.110 | | 192.168.0.111 |
| Load Balancer 1 | | Load Balancer 2 |
+——–+——–+ +——–+——–+

+——–+——–+ +——–+——–+
| 192.168.0.112 | | 192.168.0.113 |
| Web Server 1 | | Web Server 2 |
+—————–+ +—————–+

2 HTTP1 & HTTP2

2.1 Firewall Configuration

In order that the webservers are accessible from outside you have to open the corresponding ports on both web servers.

system-config-firewall-tui

Click to enlarge

(JavaScript must be enabled in your browser to view the large image as an image overlay.)

Set HTTP & HTTPS as trusted service as shown on the screenshot below and save the settings.

Click to enlarge

(JavaScript must be enabled in your browser to view the large image as an image overlay.)

Click to enlarge

(JavaScript must be enabled in your browser to view the large image as an image overlay.)

2.2 Apache Configuration

HAProxy will work as a transparent proxy – so the user’s IP address will be passed in the field “X-Forwarded-For” to the web servers. In order that the web servers will log the user’s IP address and not the IP addresses of the load balancers we have to modify the log format within the apache configuration file on both web servers.

vi /etc/httpd/conf/httpd.conf

Search the Lines that begin with “LogFormat” …

[...]
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
[...]
... and replace "%h" with "%{X-Forwarded-For}i". The content should look like this:
[...]
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
[...]

We’ll configure HAProxy to check the web servers’ health by continuously requesting the file “check.txt” from the web servers. To keep the logs small, we’ll customize the first vhost on each web server (HAProxy will use the web servers’ IP adresses to request the file – so the first vhost will answer the request) to ensure that the access to “check.txt” won’t be logged. In this example the vhosts are configured in “/etc/httpd/conf.d/vhosts.conf“.

Add the following line to the configuration of your first vhost …

SetEnvIf Request_URI “^/check\.txt$” dontlog

… and add the exception (env=!dontlog) to the line for the CustomLog. For example, the configuration for the first vhost could look like this:

NameVirtualHost 192.168.0.112:80

<VirtualHost 192.168.0.112:80>
ServerName health.example.com
ServerAdmin admin@example.com
DocumentRoot /var/www/haproxy

SetEnvIf Request_URI "^/check\.txt$" dontlog
LogLevel warn
ErrorLog /var/log/httpd/vhost_error.log
CustomLog /var/log/httpd/vhost_access.log combined env=!dontlog
</VirtualHost>

Now we have to create the file “check.txt” (this file can be empty) within the document root of the first vhost.

touch /var/www/haproxy/check.txt

Afterwards the configuration of the web servers is completed – restart the web servers.

/etc/init.d/httpd restart

3 LB1 & LB2

3.1 Firewall Configuration

In order that HTTP & HTTPS connections can be forwarded to the web servers and the heartbeat daemons can communicate with each other you have to open the corresponding ports on both load balancers.

system-config-firewall-tui

Click to enlarge

(JavaScript must be enabled in your browser to view the large image as an image overlay.)

Set HTTP & HTTPS as trusted service and insert the heartbeat-port (694 udp) into the section “Other Ports” as shown on the screenshot below. After that save the settings.


<!–
document.write(‘</div>’);
//–>

Click to enlarge

(JavaScript must be enabled in your browser to view the large image as an image overlay.)

3.2 Needed Packages On Both Load Balancers

Install the needed packages via:

yum -y install haproxy heartbeat

3.3 HAProxy Configuration

cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg_orig
cat /dev/null > /etc/haproxy/haproxy.cfg
vi /etc/haproxy/haproxy.cfg

The content should look like this on both load balancers.

global
log 127.0.0.1   local0
log 127.0.0.1   local1 notice
#log loghost    local0 info
maxconn 4096
#debug
#quiet
user haproxy
group haproxy

defaults
log     global
mode    http
option  httplog
option  dontlognull
retries 3
redispatch
maxconn 2000
contimeout      5000
clitimeout      50000
srvtimeout      50000

listen webfarm 192.168.0.120:80
mode http
stats enable
stats auth someuser:somepassword
balance roundrobin
cookie JSESSIONID prefix
option httpclose
option forwardfor
option httpchk HEAD /check.txt HTTP/1.0
server webA 192.168.0.112:80 cookie A check
server webB 192.168.0.113:80 cookie B check

Note: If you want to know more about the available options to configure HAProxy, you should take a look at http://haproxy.1wt.eu/download/1.3/doc/haproxy-en.txt and http://haproxy.1wt.eu/download/1.2/doc/architecture.txt.

3.4 Heartbeat Configuration

3.4.1 On Both Load Balancers

Heartbeat will tell LB1 & LB2 that they should listen on the shared IP (192.168.0.120). First we have to allow HAProxy to bind to the shared IP.

vi /etc/sysctl.conf

Add the following lines to the file …

# Allow HAProxy shared IP
net.ipv4.ip_nonlocal_bind = 1

… and run:

sysctl -p

Now we have to create three configuration files for heartbeat.

vi /etc/ha.d/authkeys

The content should look like this – replace %auth_password% with a password of your choice. The heartbeat daemons on the both load balancers will use this password to authenticate against each other (so it should be a very secure password).

auth 3
3 md5 %authpassword%

Change the rights so that only root is allowed to access the file.

chmod 600 /etc/ha.d/authkeys

vi /etc/ha.d/haresources

The content should look like this (on both load balancers!) – the first word is the output of

uname -n

on load balancer 1.

lb1.example.com 192.168.0.120

3.4.2 On Load Balancer 1 (LB1)

vi /etc/ha.d/ha.cf

The content should look like this – the last two lines contain the output of “uname -n” from both load balancers!:

#
#       keepalive: how many seconds between heartbeats
#
keepalive 2
#
#       deadtime: seconds-to-declare-host-dead
#
deadtime 10
#
#       What UDP port to use for udp or ppp-udp communication?
#
udpport        694
bcast  eth0
mcast eth0 225.0.0.1 694 1 0
ucast eth0 192.168.0.110
#       What interfaces to heartbeat over?
udp     eth0
#
#       Facility to use for syslog()/logger (alternative to log/debugfile)
#
logfacility     local0
#
#       Tell what machines are in the cluster
#       node    nodename ...    -- must match uname -n
node    lb1.example.com
node    lb2.example.com

3.4.3 On Load Balancer 2 (LB2)

vi /etc/ha.d/ha.cf

The content should look like this – the last two lines contain the output of “uname -n” from both load balancers!:

#
#       keepalive: how many seconds between heartbeats
#
keepalive 2
#
#       deadtime: seconds-to-declare-host-dead
#
deadtime 10
#
#       What UDP port to use for udp or ppp-udp communication?
#
udpport        694
bcast  eth0
mcast eth0 225.0.0.1 694 1 0
ucast eth0 192.168.0.111
#       What interfaces to heartbeat over?
udp     eth0
#
#       Facility to use for syslog()/logger (alternative to log/debugfile)
#
logfacility     local0
#
#       Tell what machines are in the cluster
#       node    nodename ...    -- must match uname -n
node    lb1.example.com
node    lb2.example.com

Afterwards start heartbeat on both load balancers.

/etc/init.d/heartbeat start

3.4.4 Check Heartbeat On LB1

If all went well, the output of …

ip addr sh eth0

… should also contain the shared IP – it’s the active load balancer.

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:0c:29:02:ae:eb brd ff:ff:ff:ff:ff:ff
inet 192.168.0.110/24 brd 192.168.0.255 scope global eth0
inet 192.168.0.120/24 brd 192.168.0.255 scope global secondary eth0:0
inet6 fe80::20c:29ff:fe02:aeeb/64 scope link
valid_lft forever preferred_lft forever

3.4.5 Check Heartbeat On LB2

If all went well, the output of …

ip addr sh eth0

… should not contain the shared IP as long as load balancer 1 is up – it’s the passive load balancer.

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:0c:29:e6:66:18 brd ff:ff:ff:ff:ff:ff
inet 192.168.0.111/24 brd 192.168.0.255 scope global eth1
inet6 fe80::20c:29ff:fee6:6618/64 scope link
valid_lft forever preferred_lft forever

Now we can add HAProxy to autostart and start HAProxy on both load balancers.

chkconfig –level 3 haproxy on
/etc/init.d/haproxy start

4 Failover Test

4.1 Web Server

Shut down one of the both web servers and make a HTTP request to the shared IP 192.168.0.120 (or to any domain/hostname that is pointing to the shared IP) – you should get content from the remaining web server.

4.2 Load Balancer

Shut down the active load balancer (LB1) – the passive loadbalancer (LB2) should take over immediately. The output of …

ip addr sh eth0

… on the second load balancer (LB) should now also contain the shared ip.

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:0c:29:e6:66:18 brd ff:ff:ff:ff:ff:ff
inet 192.168.0.111/24 brd 192.168.0.255 scope global eth1
inet 192.168.0.120/24 brd 192.168.0.255 scope global secondary eth1:0
inet6 fe80::20c:29ff:fee6:6618/64 scope link
valid_lft forever preferred_lft forever

When the first load balancer (LB1) is up again, it will take over the active role again.

5 HAProxy Statistics

HAProxy provides a webinterface for statistics. You can access it via http://192.168.0.120/haproxy?stats within your preferred browser. Log in with the data you configured in the HAProxy configuration file (in this example you can log in with the username “someuser” and the password “somepassword” (both without the quotes). If you don’t want/need statistics, simply remove the lines that begin with “stats” within the HAProxy configuration file on both load balancers.

For more technical articles, news and forum discussion logon

Categories: Linux

HowTo: No-effort Backup Solution for Partitions and Hard Drives

March 6, 2008 3 comments

HowTo: Install SystemRescueCD on a Dedicated Hard Disk Partition.
OR
HowTo: No-effort Backup Solution for Partitions and Hard Drives

Downloading and burning SystemRescueCD provides a bootable Gentoo-based distro on a CD. The installed applications focus on restoring disabled Linux/windows distros on the hard drives, or retrieving data if things go terribly wrong. You can operate direct from the booted CD. It’s great. But SystemRescueCD also contains PartImage, the powerful free Linux alternative to Norton Ghost. So it’s a too-easy tool for backing up single or multiple partitions, or whole drives.

Here I recount HowTo install SystemRescueCD onto a dedicated partition. I include a script for creating backup images of your hard drive partitions. Once you go through this tutorial as a practical exercise, you’ll have the knowledge and confidence to customise all manner of backup solutions so very easily.

This tutorial is for the middle ground reader, too hard for new Linux users and too simple for Gurus. It’s drawn from material in the On Line Manual at the System Rescue CD Site.

Summary of the steps for installing SystemRescueCD on a dedicated hard disk partition:

  1. Prepare a separate SystemRescue partition
  2. Download the SystemRescueCD ISO file
  3. Extract bootable image files from the ISO to the boot partitiion
  4. Edit Suse’s GRUB configuration to facilitate booting the SystemRescue partition
  5. Prepare and place your scripts, if any
  6. Boot with Suse’s loader –> select item SystemRescueCd

Step 1: Prepare a separate SystemRescue partition: I leave it to you to make the partition. You need about 160Mb plus any extra storage you might need. I use 400Mb. Note that this partition becomes the root of the cdrom after SysRescueCD boots from it, so its filesystem becomes read-only. This means you will select writeable workspaces on other partitions.

Suppose for illustration that you have prepared partition hda13 for the installation. Now make a directory to mount hda13 into openSUSE, e.g. /SysRescCD. You can mount hda13 with this command:

mount /dev/hda13 /SysRescCD

BUT I use the more convenient permanent mount created by placing this line into /etc/fstab:

/dev/hda13    /SysRescCD    ext3    defaults    1 2

You can do that with Yast –> System –> Partitioner OR more simply by issuing this command in a console: kdesu kwrite /etc/fstab and then typing the line in.

Step 2: Download the SystemRescueCD ISO file: You can download the CD ISO for the SystemRescueCD by following this project download link. The ISO filename looks like this: systemrescuecd-x86-x.y.z.iso. Place it anywhere on your hard drives, at e.g. /path_to/systemrescuecd-x86-x.y.z.iso

Step 3: Extract bootable image files from the ISO and place them in boot partition: You can mount the ISO file for viewing the files on the CD. First create a folder to mount the ISO in, e.g. /iso. Then mount the ISO with this command in a root terminal:

mount -o loop -t iso9660  /path_to/systemrescuecd-x86-0.3.6.iso    /iso

You’ll find these three files of special interest on these paths inside the mount folder:

/iso/sysrcd.dat
/iso/isolinux/rescuecd
/iso/isolinux/rescuecd.igz

Create the folder sysrcd in the root of hda13, using the mount point /SysRescCD to place it at /SysRescCD/sysrcd. Then copy the three files into folder sysrcd. The name sysrcd is immutable.

The partition hda13 is now configured with the bootable Gentoo distro and all that remains is to point Suse’s bootloader at it.

Step 4: Edit Suse’s GRUB configuration to facilitate booting the SystemRescue partition: You can open the Grub configuration file in a text editor with commands like this one for Kwrite:

kdesu kwrite /boot/grub/menu.lst

Edit/add these lines at the bottom of the file, one blank line below the last entry:

title    SystemRescueCd
root    (hd0,12)
kernel    /sysrcd/rescuecd root=/dev/ram0 init=/linuxrc looptype=squashfs loop=/sysrcd/sysrcd.dat splash=silent nosound subdir=sysrcd cdroot=/dev/hda13 setkmap=us vga=0x31a
initrd    /sysrcd/rescuecd.igz
boot

Remember to adapt my (hd0,12) which is for my hda13, across to your situation. Also, note that the sequence beginning “kernel” and ending “0x31a” is all the same/one line. I’ve included three parameters at the end: cdroot=/dev/hda13 setkmap=us vga=0x31a. These set the distro upto have hda13 at the root of the cd (on /mnt/cdrom), to have the US keyboard and for a vga screen that suits me. If you wanted to boot into the Window Manager Desktop Environment to access GUI tools, you would use this line instead:

kernel    /sysrcd/rescuecd root=/dev/ram0 init=/linuxrc looptype=squashfs loop=/sysrcd/sysrcd.dat splash=silent nosound subdir=sysrcd cdroot=/dev/hda13 setkmap=us vga=0 dostartx

A list of boot options can be seen on this link at the SystemRescueCD site.

Step 5: Prepare and place your scripts, if any: Pre defined sites [like the floppy disk, the root of the CDROM, the root of the installation partition] are searched straight after booting for scripts which if found are executed. You can have one or many scripts. See the SystemRescueCD site for full details. I’ll deal with only one location here: the root of the installation partition. It’s really simple. Rust create a script called autorun and lodge it in the root of the installation partition, hda13. It will run just after the system boots to a console.

Step 6: Boot with Suse’s loader: Reboot and select item SystemRescueCd. The root of partition hda13 automounts at location /mnt/cdrom in the booted-up virtual filesystem. All files and scripts placed on the partition are thus available at /mnt/cdrom.

Backup Script: I constantly change the filesystems on my primary hard drive and it’s hard to prevent damage to them. So I back them up regularly. This takes a long time. I use a script called autorun in the root partition of hda13 and simply boot to SystemRescueCD on hda13 and walk away to let the job proceed. Here’s my scenario and script. You could easily modify the script for your scenario.

Scenario: I have a Suse root partition at hda5 and a /home partition at hda6. These have to be backed up when they’re not being used, i.e. from within another operating system. The Gentoo installation on the SystemRescueCD partition contains a script, “autorun”, which employs “partimage”, the Linux free version of “Ghost”. It is perfect for the task. I have prepared a folder called “partimage” on partition hdb2 on IDE2 drive. The script mounts hdb2 into Gentoo/SystemRescueCD’s filesystem, generates a date-coded folder on hdb2 and copies image files across from the Suse root and home partitions.

Script

#!/bin/sh
# mount the target directory
mkdir /mnt/hdb2
mount /dev/hdb2 /mnt/hdb2
# assign today’s date to xx, example 071130 on 30Nov2007
xx=`date +%y%m%d`
# make a directory in the folder “partimage” on hdb2 and name it for the date
mkdir /mnt/hdb2/partimage/$xx
cd /mnt/hdb2/partimage/$xx
# write start time to a logfile
zz=$xx’logfile’
echo ‘start at: ‘`date`>$zz
# make an image of suse102_root options: -z1=gzip -d=no description save=save_image -b=batch(not gui) -f3=quit when finished
partimage -z1 -d save -b -f3 /dev/hda5 /mnt/hdb2/partimage/$xx/hda5.partimg.gz
# make an image of /home options: -z1=gzip -d=no description save=save_image -b=batch(not gui) -f3=quit when finished
partimage -z1 -d save -b -f3 /dev/hda6 /mnt/hdb2/partimage/$xx/hda6.partimg.gz
# write contents of file autorun to a file in the target directory
cat /mnt/cdrom/autorun >>script.used
# write end time to the logfile
echo ‘end at: ‘`date`>>$zz
# write the contents of the backup directory into the logfile
ls -l>>$zz
reboot

These are the things to customise: Change hdb2 to match your target storage partition. Change hda5 to match your root partition. Change hda6 to match your home partition. Everything else should match your system.

This is the key line:

partimage -z1 -d save -b -f3 /dev/hda5 /mnt/hdb2/partimage/$xx/hda5.partimg.gz

If you have six partitions, duplicate this line six times, replacing hda5 with your correct partition designations and hdb2 with your target storage/backup partition.

That’s all there is folks, enjoy.

For more technical articles, news and forum discussion logon

Categories: Linux

How to: Compile Linux kernel 2.6

March 6, 2008 Leave a comment

Compiling custome kernel has its own advantages and disadvantages. However new Linux user / admin find it difficult to compile kernel. Compiling kernel needs to understand few things and then just type couple of commands. This step by step howto covers compiling Linux kernel version 2.6 under Debian GNU Linux. However, instructions remains same for any other distribution except for apt-get command.

Step # 1 Get Latest Linux kernel code

Visit http://kernel.org/ and download the latest source code. File name would be linux-x.y.z.tar.bz2, where x.y.z is actual version number. For example file inux-2.6.23.tar.bz2 represents 2.6.23 kernel version. Use wget command to download kernel source code:
$ cd /tmp
$ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-x.y.z.tar.bz2

Note: Replace x.y.z with actual version number.

Step # 2 Extract tar (.tar.bz3) file

Type the following command:
# tar -xjvf linux-2.6.23.tar.bz2 -C /usr/src
# cd /usr/src

Step # 3 Configure kernel

Before you configure kernel make sure you have development tools (gcc compilers and related tools) are installed on your system. If gcc compiler and tools are not installed then use apt-get command under Debian Linux to install development tools.
# apt-get install gcc

Now you can start kernel configuration by typing any one of the command:

  • $ make menuconfig – Text based color menus, radiolists & dialogs. This option also useful on remote server if you wanna compile kernel remotely.
  • $ make xconfig – X windows (Qt) based configuration tool, works best under KDE desktop
  • $ make gconfig – X windows (Gtk) based configuration tool, works best under Gnome Dekstop.

For example make menuconfig command launches following screen:
$ make menuconfig

You have to select different options as per your need. Each configuration option has HELP button associated with it so select help button to get help.

Step # 4 Compile kernel

Start compiling to create a compressed kernel image, enter:
$ make
Start compiling to kernel modules:
$ make modules

Install kernel modules (become a root user, use su command):
$ su -
# make modules_install

Step # 5 Install kernel

So far we have compiled kernel and installed kernel modules. It is time to install kernel itself.
# make install

It will install three files into /boot directory as well as modification to your kernel grub configuration file:

  • System.map-2.6.23
  • config-2.6.23
  • vmlinuz-2.6.23

Step # 6: Create an initrd image

Type the following command at a shell prompt:
# cd /boot
# mkinitrd -o initrd.img-2.6.23 2.6.23

initrd images contains device driver which needed to load rest of the operating system later on. Not all computer requires initrd, but it is safe to create one.

Step # 7 Modify Grub configuration file – /boot/grub/menu.lst

Open file using vi:
# vi /boot/grub/menu.lst

title           Debian GNU/Linux, kernel 2.6.23 Default
root            (hd0,0)
kernel          /boot/vmlinuz root=/dev/hdb1 ro
initrd          /boot/initrd.img-2.6.23
savedefault
boot

Remember to setup correct root=/dev/hdXX device. Save the file. If you think editing and writing all lines by hand is too much for you then try out update-grub command to update the lines for each kernel in /boot/grub/menu.lst file. Just type command:
# update-grub
Neat. Huh?

Step # 8 : Reboot computer and boot into your new kernel

Just issue reboot command:
# reboot
For more information see:

  • Our Exploring Linux kernel article and Compiling Linux Kernel module only.
  • Official README file has more information on kernel and software requirement to compile it. This file is kernel source directory tree.
  • Documentation/ directory has interesting kernel documentation for you in kernel source tree.

For more technical articles, news and forum discussion logon

Categories: Kernel

Multiboot a mixture of Linux and Microsoft distributions Using Lilo and/or Grub [tested for Suse, Mandriva, Debian, Fedora, MSDos & Windows XP]

March 6, 2008 1 comment

This Post has been moved to http://opensupportindia.com

For more technical articles, news and forum discussion logon

Categories: Linux

How to access Windows Fat32 partition in Suse 10.3?

March 6, 2008 1 comment

This post has been moved to http://opensupportindia.com

For more technical articles, news and forum discussion logon

Categories: Linux

Why can’t we create a folder by name CON?

February 21, 2008 34 comments

I’ve been asked this question many a times: Why can’t we create a folder by name CON? Although it seems a wonder or magic that we can’t create a folder by that name, in reality, it is not so. It has a definite reason, and in fact, a folder can be created using that reserved name.

Gone are the days when computers had only CUI OS, that is, Character User Interface Operating Systems, like MS-DOS. When I joined my first computer course nine years ago, Windows 95 was ruling. You could see Windows 98 here and there. We were in 8th standard, and working on a computer was like a dream coming true. Microsoft’s Paint Brush was the only known (for us) GUI software and was the greatest means of entertainment. The instructors taught us only MS-DOS commands and how to Shut Down the computer. Remembering such weird names as DIR, CD, MD, RD, CHKDSK, FDISK, VER, ATTRIB, REN, DEL etc. along with their syntax and usage was a great accomplishment. But I had a problem understanding this: DOS has a separate dedicated command for every action; literally every action, except… creating a file!

Yes, we used COPY CON filename to create a file with name filename. Anyone can say that it is a form of COPY command. So, why was creating a file different than all other commands? I didn’t understand it, till I found out how to print using DOS, almost four years later.

DOS uses different names for the attached devices, I learnt. PRN was one such name. TYPE filename would display the contents of a file and TYPE filename > PRN would print it instead of displaying. Curiosity brings many hidden matters out. PRN would surely mean Printer and will redirect the output to the printer instead of console. Console (monitor) is the implicit default output device, and it can be bypassed if needed. So, how to put it explicitly? There must be some means to do that. Yes, there is! TYPE filename > CON performs exactly same function as TYPE filename. These special names for the devices really mean something special for the operating system and those names can not be used as folder or file names: CON, PRN, NUL, COM1 to COM9, LPT1 to LPT9, which stand for CONsole, PRiNter, NULl, serial COMmmunication ports, Line PrinTer ports.

The time has changed and Operating System can also be fooled! But still, many people think that it is not possible to create a folder by name CON. Using the path of network drive, these special names can also be used as folder names! Here is how:

  1. Goto DOS
  2. Type MD \\.\C:\CON. The folder will be created. You can check it in Windows Explorer also, but you can’t access it
  3. To delete the folder, type RD \\.\C:\CON

In short, use the network path syntax instead of absolute path syntax.

Now on to the practical aspect of this. Why can’t we create it directly but using the network path syntax? The answer is simple. A computer can have only one default console, printer, null etc. So, if it is accessed from a network, theoretically, the console should belong to another node in the network. Since that node may not have a device which can be referred using the name CON, it will no longer be considered as a reserved name. Hence, the folder can be created.

The next time when someone asks the question why we can’t create a folder by name CON, say with confidence that it is not true…

For more technical articles, news and forum discussion logon

Categories: Windows