md5sum differences on identical systems

In troubleshooting a problem a colleague noticed that there were md5sum differences in the files of identical systems:

ServerOne # md5sum /lib64/
27a605fdeaf7c835493a098213c9eec1  /lib64/

ServerTwo # md5sum /lib64/
13e3eb598abd09279efc05e215e77ae2 /lib64/libc-2.12.s

Analyzing a hex dump of the .so files showed cyclical differences at matching locations.  This is where I began helping to look at the problem.  After walking through the above (these files were the focus of an application error) I suggested we do a yum reinstall.  This resulted in different md5sums from those listed above.  The cyclical and matching locations are the link locations in the hex dump, as expected there are differences in the hex locations between systems.

After a little research I found the reason the md5sum’s don’t match is yum install uses prelink.  ‘prelink’ is a program that modifies ELF shared libraries and ELF dynamically linked binaries used by yum, therefore there would be differences in the md5sum even on identical systems.

To see the actual md5sum of the package before prelink is run, use the following command:

# prelink -y /lib64/ | md5sum
14486c78922a8dc8376a715d4c78d85a  -
 I ran this on the two systems in question and they match.  The files are the same on each systems, therefore this was a red hearing and the search continues.