XenLvmCman

Setting up a cluster with at least three nodes with GFS2 volumes, LVM2 and various services. All nodes are running inside Xen-DomU instances.

Goals

• Have an easy way to create new nodes for the cluster by cloning existing DomU systems.
• Share one or more resizeable volumes between all nodes in the cluster and provide parallel access (R/W) to these volumes.
• Run Dovecot and Postfix on each node utilizing the shared volume.
• Provide some ways of controlling each service on the nodes.
• Manage configuration on all nodes equally with a centralized system.

Requirements

Xen

As of the time this document was written, the Debian package xen-linux-system-2.6.26-1-xen-686 provide the latest available Xen system. It depends on the binary Linux kernel and the Xen hypervisor. Once the Xen system is installed, the machine has to be rebooted and in Grub there should be a new entry for booting the Xen hypervisor. Select this entry and see if it works. Now we have a running Dom0 which will serve as the host system for all our future DomUs.

LVM

If the Dom0 is not already running one or more LVG VGs it would now be the time to create one. I will assume that there is a spare block device on /dev/sdx which will serve as a physical volume for our new LVM VG named domu: %CODE{"sh"}% #> pvcreate /dev/sdx #> vgcreate -s 32M domu /dev/sdx %ENDCODE%

Packages on cluster nodes

• cman
• clvm
• rgmanager
• gfs2-tools

Install them with all their dependencies: %CODE{"sh"}% #> aptitude install cman clvm rgmanager gfs2-tools %ENDCODE%

Configuration

Enable clustered LVM

To use LVM in a clustered environment it has to uses it's built-in cluster locking. This can be enabled inside /etc/lvm/lvm.conf inside the global section:

locking_type = 3

It seems that the Debian package of clvm does not create the Sys-V init symlinks automatically so I had to set them up by hand: %CODE{"sh"}% #> update-rc.d clvm defaults %ENDCODE%

Create cluster.conf

Here is a sample cluster.conf which is used to power my experimental cluster for dovecot and postfix:

<?xml version="1.0"?>
<cluster name="test1" config_version="11">
    <clusternodes>
        <clusternode name="cluster1" nodeid="1">
            <fence>
                <method name="xvm">
                    <device name="xvm" nodename="cluster1" domain="cluster1"/>
                </method>
            </fence>
        </clusternode>
        <clusternode name="cluster2" nodeid="2">
            <fence>
                <method name="xvm">
                    <device name="xvm" nodename="cluster2" domain="cluster2"/>
                </method>
            </fence>
        </clusternode>
        <clusternode name="cluster3" nodeid="3">
            <fence>
                <method name="xvm">
                    <device name="xvm" nodename="cluster3" domain="cluster3"/>
                </method>
            </fence>
        </clusternode>
    </clusternodes>
        <fencedevices>
          <fencedevice name="last_resort" agent="fence_manual"/>
          <fencedevice name="xvm" agent="fence_xvm"/>
        </fencedevices>
    <rm>
        <failoverdomains>
            <failoverdomain name="cluster1" restricted="1">
                <failoverdomainnode name="cluster1"/>
            </failoverdomain>
            <failoverdomain name="cluster2" restricted="1">
                <failoverdomainnode name="cluster2"/>
            </failoverdomain>
            <failoverdomain name="cluster3" restricted="1">
                <failoverdomainnode name="cluster3"/>
            </failoverdomain>
        </failoverdomains>
        <resources>
            <clusterfs device="/dev/mapper/cluster-mail" force_unmount="0" fstype="gfs2" mountpoint="/home" name="mail" options="acl"/>
            <script name="Dovecot" file="/etc/init.d/dovecot"/>
            <script name="Postfix" file="/etc/init.d/postfix"/>
            <script name="Spamassassin" file="/etc/init.d/spamassassin"/>
        </resources>
        <service name="cluster1-mail" autostart="1" domain="cluster1">
            <script ref="Spamassassin"/>
            <script ref="Dovecot"/>
            <script ref="Postfix"/>
            <clusterfs ref="mail"/>
        </service>
        <service name="cluster2-mail" autostart="1" domain="cluster2">
            <script ref="Spamassassin"/>
            <script ref="Dovecot"/>
            <script ref="Postfix"/>
            <clusterfs ref="mail"/>
        </service>
        <service name="cluster3-mail" autostart="1" domain="cluster3">
            <script ref="Spamassassin"/>
            <script ref="Dovecot"/>
            <script ref="Postfix"/>
            <clusterfs ref="mail"/>
        </service>
    </rm>
</cluster>