RedHat 6


In support of the Kuali project.

Setting up true fail over for the Kuali application servers.  Currently if a node went down, the user would need to re-authenticate.  The following procedure configures the system so it can lose a node and the users on that node will not lose their session.

My part on the system side was fairly straightforward:

yum install memcached
iptables -I INPUT -m state --state NEW -m tcp -p tcp --dport 11211 -j ACCEPT
service iptables save
chkconfig memcached on
service memcached start

With that configured the work to enable tomcat to leverage memcached can begin:

Parts of the following information was found at (

Download the most recent copy of the following jars (links provided) and install them to the tomcat_dir/lib directory:

For each jar, open tomcat_dir/conf/context.xml, and add the following lines inside the <Context> tag:

<Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
    requestUriIgnorePattern=".*.(ico|png|gif|jpg|css|js)$" />

If memcached is listening on a different port, change the value in memcachedNodes.  port 11211 is the default port for memcached.

Open tomcat_dir/conf/server.xml, look for the following lines:

<Server port="8005" ...>
    <Connector port="8080" protocol="HTTP/1.1" ...>
    <Connector port="8009" protocol="AJP/1.3" ...>

Change the ports, so the two installations listen to different ports. This is optional, but I would also disable the HTTP/1.1 connector by commenting out its <Connector> tag, as the setup documented here only requires the AJP connector to be enabled.

Finally, look for this line, also in tomcat_dir/conf/server.xml:

<Engine name="Catalina" defaultHost="localhost" ...>

Add the jvmRoute property, and assign it a value, that is different between the two installations. For example:

<Engine name="Catalina" defaultHost="localhost" jvmRoute="jvm1" ...>

And, for the second instance:

<Engine name="Catalina" defaultHost="localhost" jvmRoute="jvm2" ...>

That’s it for Tomcat configuration. This configuration uses memcached-session-manager’s default serialization strategy and enables sticky session support. For more configuration options, refer to the links in the references section.

In our apache load balancer we add the following definition:

ProxyPass /REFpath balancer://Cluster_Name
ProxyPassReverse /REFpath balancer://Cluster_Name

<Proxy balancer://Cluster_Name>
   BalancerMember ajp://HOSTNAME:8009/REFpath route=jvm1  timeout=600 min=10 max=100 ttl=60 retry=120 connectiontimeout=10
   BalancerMember ajp://HOSTNAME:8009/REFpath route=jvm2  timeout=600 min=10 max=100 ttl=60 retry=120 connectiontimeout=10
   BalancerMember ajp://HOSTNAME:8009/REFpath route=jvm3  timeout=600 min=10 max=100 ttl=60 retry=120 connectiontimeout=10
   BalancerMember ajp://HOSTNAME:8009/REFpath route=jvm4  timeout=600 min=10 max=100 ttl=60 retry=120 connectiontimeout=10
   ProxySet lbmethod=byrequests
   ProxySet stickysession=JSESSIONID|jsessionid
   ProxySet nofailover=On

Note that the BalancerMember lines point to the ports and jvmRoutes configured above.  This sets up a load balancer that dispatches web requests to multiple Tomcat installations. When one of the Tomcat instance gets shutdown, requests will be served by the other one that is still up. As a result, user does not experience downtime when one of the Tomcat instances is taken down for maintenance or application redeployment.

This step also sets up sticky session. What this means is that, if user begins session with instance 1, she would be served by instance 1 throughout the entire session, unless of course this instance goes down. This can be beneficial in a clustered environment, as application servers can use session data stored locally without contacting a remote memcached.

Increasing the size of a filesystem


fdisk -l
fdisk /dev/sdc

In fdisk

p  (print the partition table to make sure the disk is not in use)
n (new partition)
p (primary partition)
1 (give it a number 1-4, then set start and end sectors)
w (write table to disk and exit)

Now create a physical volume, add it to the VG, extend the LV and then the file system.

pvcreate /dev/sdc1
vgextend VG_NAME /dev/sdc1
lvextend -L+5G LV_PATH (i.e.: /dev/VG_NAME/LV_NAME)
resize2fs LV_PATH
(OR if using xfs: xfs_grow LV_PATH)


Other useful commands when working with disks include:

# lsblk
NAME                             MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sr0                               11:0    1  1024M  0 rom  
sda                                8:0    0 501.1M  0 disk 
└─sda1                             8:1    0   500M  0 part /boot
sdb                                8:16   0  29.5G  0 disk 
└─sdb1                             8:17   0  29.5G  0 part 
  ├─vg_name-lv_root (dm-0) 253:0    0  40.6G  0 lvm  /
  └─vg_name-lv_swap (dm-1) 253:1    0   3.7G  0 lvm  [SWAP]
sdc                                8:32   0    20G  0 disk

The lsblk will list all block devices.  Above it is an easy way to see disks, disk usage and LVM affiliations.  Of course if you just want the block device names this will work too:

ls /sys/block/* | grep block | grep sd


Extended ACLs

To remove permanently ACL from a file:

# setfacl -bn file.txt

To remove permanently ACL from an entire directory:

# setfacl -b --remove-all

To overwrite permissions, setting them to rw for files and rwx for dirs

$ find . ( -type f -exec setfacl -m g:mygroup:rw '{}' ';' ) 
      -o ( -type d -exec setfacl -m g:mygroup:rwx '{}' ';' )

To set mygroup ACL permissions based on existing group permissions

$ find . ( -perm -g+x -exec setfacl -m g:mygroup:rw '{}' ';' ) 
      -o ( -exec setfacl -m g:mygroup:rwx '{}' ';' )

You’ll probably want to check that the group mask provides effective permissions. If not you can do it the old school way and run this too:

$ find . -type d -exec chmod g+rwX '{}' ';'


X11 error on login to RedHat Servers

I noticed that since the last set of patches many of my redhat 6 systems are reporting an X11 forwarding error after login:

X11 forwarding request failed on channel 0

To correct this problem you need to install the following package

yum install xorg-x11-xauth

I have not had the time to investigate why this is suddenly a problem.  When I have time I’ll report back the why.