Angelo’s blog

Installing Pacemaker/Corosync

Install 2 or more CentOS machines and make sure they’re updated.

Add the EPEL repository:
rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-4.noarch.rpm

Add the pacemaker repository:
wget -O /etc/yum.repos.d/pacemaker.repo http://clusterlabs.org/rpm/epel-5/clusterlabs.repo

(This is all according to the guide at http://www.clusterlabs.org/wiki/Install#Installing_on_EPEL_Compatible_Distributions:_RHEL.2C_CentOS.2C_etc. Don’t use the RPMForge repo, as both the EPEL and RPMForge have different sets of packages, split up, etc.)

Then install the software by running:

yum install -y pacemaker corosync heartbeat

Now you have all the software installed. Generate the key:

corosync-keygen
chown root:root /etc/corosync/authkey
chmod 400 /etc/corosync/authkey

Copy the above key to all nodes, and make sure the permissions are correct by running the chown and chmod command.

Then run the following script (change the bind ip to the network address of the interface on which the nodes can communicate with each other like 10.123.10.0 and change the mcastaddress to something random, same on all hosts).

echo "totem {" > /etc/corosync/corosync.conf
echo " version: 2" >> /etc/corosync/corosync.conf
echo " token: 5000" >> /etc/corosync/corosync.conf
echo " token_retransmits_before_loss_const: 20" >> /etc/corosync/corosync.conf
echo " join: 1000" >> /etc/corosync/corosync.conf
echo " consensus: 7500" >> /etc/corosync/corosync.conf
echo " vsftype: none" >> /etc/corosync/corosync.conf
echo " max_messages: 20" >> /etc/corosync/corosync.conf
echo " secauth: off" >> /etc/corosync/corosync.conf
echo " threads: 0" >> /etc/corosync/corosync.conf
echo " clear_node_high_bit: yes" >> /etc/corosync/corosync.conf
echo " " >> /etc/corosync/corosync.conf
echo " interface {" >> /etc/corosync/corosync.conf
echo " ringnumber: 0" >> /etc/corosync/corosync.conf
echo " " >> /etc/corosync/corosync.conf
echo " # changethis!" >> /etc/corosync/corosync.conf
echo " bindnetaddr: 10.0.0.1" >> /etc/corosync/corosync.conf
echo " mcastaddr: 226.94.1.1" >> /etc/corosync/corosync.conf
echo " mcastport: 5405" >> /etc/corosync/corosync.conf
echo " }" >> /etc/corosync/corosync.conf
echo " }" >> /etc/corosync/corosync.conf
echo " " >> /etc/corosync/corosync.conf
echo " logging {" >> /etc/corosync/corosync.conf
echo " fileline: off" >> /etc/corosync/corosync.conf
echo " to_syslog: yes" >> /etc/corosync/corosync.conf
echo " to_stderr: no" >> /etc/corosync/corosync.conf
echo " syslog_facility: daemon" >> /etc/corosync/corosync.conf
echo " debug: on" >> /etc/corosync/corosync.conf
echo " timestamp: on" >> /etc/corosync/corosync.conf
echo " }" >> /etc/corosync/corosync.conf
echo " " >> /etc/corosync/corosync.conf
echo " amf {" >> /etc/corosync/corosync.conf
echo " mode: disabled" >> /etc/corosync/corosync.conf
echo " }" >> /etc/corosync/corosync.conf
echo "service {" > /etc/corosync/service.d/pcmk
echo " # Load the Pacemaker Cluster Resource Manager" >> /etc/corosync/service.d/pcmk
echo " name: pacemaker" >> /etc/corosync/service.d/pcmk
echo " ver: 0" >> /etc/corosync/service.d/pcmk
echo " }" >> /etc/corosync/service.d/pcmk

chkconfig corosync on
/etc/init.d/corosync start

Make sure you open port udp/5405 in the firewall. (I don’t know if everything works with SELinux right now)

Now you should run crm_mon on all nodes, and see all nodes are part of the cluster, and online. This means the cluster framework is succesfully running. Initially, it can take up to 3-5 minutes before the cluster stuff starts, because it will first look for any other pre-existing nodes on the network before deciding there isn’t any.

You can now add resources:

crm configure property stonith-enabled="false"
crm configure property no-quorum-policy=ignore
crm configure primitive failover-ip ocf:heartbeat:IPaddr params ip="10.123.254.213" op monitor interval="2s"

That’s it, now you a highly available ip address! You could of course add another ip address, and use DNS round robin to balance. Now you have an active-active high available cluster!

Pacemaker basics

Everything about pacemaker is managed using the ‘crm*’ commands. crm_mon will show the status. You can run the ‘crm’ command interactively, or just issue single commands. Example of a single commands:

crm configure show (show configuration)
crm node standby (put this node in standby and migrate resources)
crm node online (put this node online again)

Pacemaker can do a lot more, but for now we’ll just use it to mange virtuel ip’s.

IPv6

The above setup, although it will work fine, does not support IPv6. This is because the packager of the ‘resource-agents’ package decided to leave out the IPv6addr module because of dependencies. I compiled a version of the package with the module included (instructions below). If you need IPv6, follow these steps:

/etc/init.d/corosync stop
yum remove resource-agents -y
yum install libnet -y
rpm -i http://files.hongens.nl/RPM/resource-agents-1.0.3-2.6.x86_64.rpm
yum install corosync pacemaker -y
/etc/init.d/corosync start

Now you should be able to add an ipv6 address to the cluster stack:

crm configure primitive failover-ip3 ocf:heartbeat:IPv6addr params ipv6addr="2001:888:2085:2fe:250::213" op monitor interval="2s"

Compiling a custom RPM with IPv6addr built in

To compile, first add the EPEL and Clusterlabs repository, as described above. Then install the dependencies:

yum install autoconf automake gcc libnet-devel libtool libxml2-devel bzip2-devel glib2-devel libxslt-devel e2fsprogs-devel docbook-style-xsl cluster-glue-libs-devel cluster-glue-libs rpm-build python-devel

Install the source package, it will be added to /usr/src/redhat:

mkdir /usr/src/redhat
rpm -i http://www.clusterlabs.org/rpm/epel-5/src/resource-agents-1.0.3-2.6.el5.src.rpm
cd /usr/src/redhat
rpmbuild -bb SPECS/resource-agents.spec
Done! Now the rpm is in /usr/src/redhat/RPMS/x86_64