Author: Rifkin, Jonathan

Cacti, RRDs, and Disk Block Sizes

Abstract

Disk block size on Linux for the ext2/3/4 file system does not affect the amount of data written to disk.  Apparently, all of the allowed values for blocksize (1024, 2048 or the default of 4096 bytes) result in data of 4096 bytes being written to disk.

Introduction

Cacti is a network monitoring program which gathers statistics every minute or so, and writes them the disk.   When monitoring large systems, you may find that volume disk IO can be a problem.

Cacti writes the its data to RRD files (Round-Robin Databases).   Each RRD holds related values from a piece of data equipment.  These values may be stored for every minute, and then aggregated every 30 minutes and 24 hours, for example.  As a result, in an RRD of one megabyte, only a handful bytes are updated every minute (or so) interval.

Unfortunately, although only a few bytes are updated, disk IO happens in much larger blocks.  Thus diskIO can be 100 times or more of the theoretical minimum.

To help with this, I attempted to verify that a reducing the blocksize on a filesystem will reduce the IO.  This failed.  Modifying the blocksize had no effect on IO.

Method

Creating the Filesystem

A file system was created to test each blocksize, with a command similar to the following

mke2fs -T ext4 -b 4096 /dev/sdb1

where /dev/sdb1 is the file partition and 4096 is a blocksize (the others were 2048 and 1024).

Writing to the File

The ‘dd’ command was used to create an initial 1MB file filled with null chars.  This file was used to write to in the test.

dd if=/dev/zero of=test.fil bs=1024 count=1024

The actual test was done with this short Python program:

import sys
import mmap
import random

NSIZE = 0x100000    #  File size
fname = "test.fil"  #  File name

#  Open mmap file
f = open(fname,"r+b")
mm = mmap.mmap(f.fileno(), 0)

#  Initialize random number generator object
r = random.Random()

#  Get number of bytes to write into memory
n = int(sys.argv[1])           #  Read command line
for i in range(n):
    pos = r.randint(0,NSIZE-1) #  Choose memory location at random
    mm[pos] = 'X'              #  Place X character onto disk
mm.close()
f.close()

Measuring the Disk IO

The Python program was run with values for n of 0, 1, 2, 3, while watching the output from iostat to determine the number of KBytes written to disk.  Here’s an example of using iostat:

> iostat -p sdb 5
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sdb               1.00         0.00         1.60          0          8
sdb1              1.00         0.00         1.60          0          8
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.40    0.00    1.30    1.70    0.00   93.60
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sdb               1.20         0.00         2.40          0         12
sdb1              1.20         0.00         2.40          0         12
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           4.41    0.00    1.30    0.50    0.00   93.79
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sdb               1.40         0.00         3.20          0         16
sdb1              1.40         0.00         3.20          0         16

Here “-p sdb” tells iostat to only report on the device /dev/sdb, and the “5″ tells iostat to display every 5 seconds.

While “iostat” is running, the test Python program is run.  Iostat will show how many Kbytes are written to disk.

testwrite.py 1; sync   #  Wait five seconds
testwrite.py 2; sync   #  Wait five seconds
testwrite.py 3; sync

The lines above write 1, 2 and 3 bytes to disk.  By waiting 5 seconds between each command, iostat will show the amount of Kbytes written for each one — the iostat output shown above is from a test.  It shows that 8, 12 and 16 kilobytes were written to disk when 1, 2 and 3 bytes are written from the program.  There is a constant amount of overhead of 6 KB for each write, which means that each bytes written to by the test program resulted in 4KB written to disk.  This value persisted even when the filesystem blockwas was 1024, 2048 and 4096 (in each case, only the overhead grew, but not the amout of data written disk disk per byte written from the program).

Recursive Gzip Sometimes Helps

Everyone knows that gzip’ing a file will usually make it smaller, but gzip’ing it again will not. In other words, once you’ve compressed a file, it won’t compress further.   But here’s an exception.

If you make a file of identical bytes, it can be gzip’ed several times and still become smaller.  In the example below, we made a 2GB file which consisted of identical bytes, all equal 0.  The first compression reduced the size by a factor of 1024.  The second by a factor of 602.  Next, by a factor of 14.  The next compression, not shown, grew the file by about 20 bytes.  The total compression, over the three generations, was a factor of 9,023,041 !

> dd if=/dev/zero of=2gb.zero bs=1024576 count=2048
> gzip -c 2gb.zero       > 2gb.zero.gz
> gzip -c 2gb.zero.gz    > 2gb.zero.gz.gz
> gzip -c 2gb.zero.gz.gz > 2gb.zero.gz.gz.gz
> ls -lh 2gb.zero
2.0G  2gb.zero
2.0M  2gb.zero.gz
3.4K  2gb.zero.gz.gz
328B  2gb.zero.gz.gz.gz

mailservertest: A command line script to test mail servers

mailservertest can be used to make testing mail servers a bit easier. It is a short script written in Python. Here a few use cases.

You’ve just restarted a number of mail servers, and you want to verify that they are delivering mail

mailservertest  my.email@example.com  mail1.example.com mail2 mail3

This will send a test email to my.email@example.com through the four mail servers mail1, mail2, mail3 and mail4.  Note that the full domain name needs to be specified for the first mail server mail1.example.com; afterwards the ‘example.com’ will be appended to the remaining mail servers.  This will save you some typing.

When you run the command, you will see output like this:

mail1.example.com
   ... Connecting to server ...
   OK
   ... Sending the "helo" command ...
   OK
   ... Sending the email ...
   OK
mail2.example.com
   ... Connecting to server ...
   OK
   ... Sending the "helo" command ...
   OK
   ... Sending the email ...
   OK
mail3.example.com
   ... Connecting to server ...
   OK
   ... Sending the "helo" command ...
   OK
   ... Sending the email ...
   OK

The script verifies each step of the mail process for each server.  Thus, you can see where in the process a problem is occurring.  The -q option will turn off these messages, but error messages (if any) will still be shown.

The emails you receive will look like this

From:    my.email@example.com
Subject: MAILTEST: mail1 16:28:29 2013-12-08
To:      my.email@example.com

This is a test email

   Tag             : MAILTEST: 
   Date            : 16:28:29 2013-12-08
   All Recipients  : my.email@example.com
   All Servers     : mail1.example.com mail2.example.com mail3.example.com
   This Server     : mail3.example.com
----------------------------------------------------
Sent from the script 'mailservertest'
on the host 'Your-Host'

The Subject line was chosen to make it as useful as possible when viewed in an email client; it shows the server that sent the message and the time that the script was run.  The “tag” MAILTEST can be customized using the -t option.

You need to verify that your mail server rejects invalid addresses during the initial connection.

If your mail server should reject email for invalid addresses, you can verify that with this command (shown with it’s output).

mailservertest -q noone@example.com mail1.example.com
   ERROR:  {'noone@example.com': (550, '5.1.1 <noone@example.com>: Recipient address rejected: User unknown in virtual alias table')}

First note that the -q option has removed the status output shown in the previous example.  Only the error message is shown; and it verifies that the address is unknown.

You want to generate and inspect a bounce message

You may be able to generate an email bounce message by sending an email through one mail server functioning as an MTA (mail transfer agent) to a second mail server functioning as a MDA (mail delivery agent).  If the MTA passes your message to the MDA, and the recipient address is invalid, the MDA will return a bounce message to the return address.  Since in this case the recipient address and the return address must be different, we will need to use the -f option to mailservertest.  The -f option sets the return address.  By default, the recipient and return address are the same; this worked fine for the previous example, but not now.

mailservertest -f my-email@example.com  broken-email@other.com  mail.example.com

Here, broken-email@other.com is an invalid email address.  We first send the email to our MTA mail.example.com.  It will send the email to whatever mail server handles other.com.  The email bounce will be sent to our email address my-email@example.com.  Below is an example of what is returned.

Delivery has failed to these recipients or groups:

broken-email@other.com
The e-mail address you entered couldn't be found. Please check the recipient's e-mail address and try to resend the message. If the problem continues, please contact your helpdesk.

Diagnostic information for administrators:

Generating server: CASHUBE.grove.ad.other.com

broken-email@other.com
#550 5.1.1 RESOLVER.ADR.RecipNotFound; not found ##rfc822;broken-email@other.com

Original message headers:

Received: from mail.example.com (192.168.25.234) by
 CashubE.grove.ad.other.com (192.168.30.167) with Microsoft SMTP Server id
 14.2.347.0; Sun, 8 Dec 2013 17:25:49 -0500
Received: from localhost (host.example.com [192.168.80.33]) by
 mail.example.com (Postfix) with SMTP id 2F1007F44  for
 <broken-email@other.com>; Sun,  8 Dec 2013 17:25:49 -0500 (EST)
From: "my-email@example.com" <my-email@example.com>
Subject: MAILTEST: mail 17:25:49 2013-12-08
To: <broken-email@other.com>
Date: Sun, 8 Dec 2013 17:25:49 -0500
MIME-Version: 1.0
Content-Type: text/plain
Message-ID: <e91f8c64-57e5-47db-a2bc-9857ce0c04f3@CASHUBE.grove.ad.other.com>
Return-Path: my-email@example.com

Reporting-MTA: dns;CASHUBE.grove.ad.other.com
Received-From-MTA: dns;mail.example.com
Arrival-Date: Sun, 8 Dec 2013 22:25:49 +0000

Original-Recipient: rfc822;broken-email@other.com
Final-Recipient: rfc822;broken-email@other.com
Action: failed
Status: 5.1.1
Diagnostic-Code: smtp;550 5.1.1 RESOLVER.ADR.RecipNotFound; not found

ForwardedMessage.eml
Subject:
MAILTEST: mail 17:25:49 2013-12-08
From:
"my-email@example.com" <my-email@example.com>
Date:
12/08/2013 05:25 PM
To:
<broken-email@other.com>

This is a test email

   Tag             : MAILTEST:
   Date            : 17:25:49 2013-12-08
   All Recipients  : broken-email@other.com
   All Servers     : mail.example.com
   This Server     : mail.example.com

----------------------------------------------------
Sent from the script 'mailservertest'
on the host 'host'

mailservertest: A command line script to test mail servers

mailservertest can be used to make testing mail servers a bit easier. It is a short script written in Python. Here a few use cases.

You’ve just restarted a number of mail servers, and you want to verify that they are delivering mail

mailservertest  my.email@example.com  mail1.example.com mail2 mail3

This will send a test email to my.email@example.com through the four mail servers mail1, mail2, mail3 and mail4.  Note that the full domain name needs to be specified for the first mail server mail1.example.com; afterwards the ‘example.com’ will be appended to the remaining mail servers.  This will save you some typing.

When you run the command, you will see output like this:

mail1.example.com
   ... Connecting to server ...
   OK
   ... Sending the "helo" command ...
   OK
   ... Sending the email ...
   OK
mail2.example.com
   ... Connecting to server ...
   OK
   ... Sending the "helo" command ...
   OK
   ... Sending the email ...
   OK
mail3.example.com
   ... Connecting to server ...
   OK
   ... Sending the "helo" command ...
   OK
   ... Sending the email ...
   OK

The script verifies each step of the mail process for each server.  Thus, you can see where in the process a problem is occurring.  The -q option will turn off these messages, but error messages (if any) will still be shown.

The emails you receive will look like this

From:    my.email@example.com
Subject: MAILTEST: mail1 16:28:29 2013-12-08
To:      my.email@example.com

This is a test email

   Tag             : MAILTEST: 
   Date            : 16:28:29 2013-12-08
   All Recipients  : my.email@example.com
   All Servers     : mail1.example.com mail2.example.com mail3.example.com
   This Server     : mail3.example.com
----------------------------------------------------
Sent from the script 'mailservertest'
on the host 'Your-Host'

The Subject line was chosen to make it as useful as possible when viewed in an email client; it shows the server that sent the message and the time that the script was run.  The “tag” MAILTEST can be customized using the -t option.

You need to verify that your mail server rejects invalid addresses during the initial connection.

If your mail server should reject email for invalid addresses, you can verify that with this command (shown with it’s output).

mailservertest -q noone@example.com mail1.example.com
   ERROR:  {'noone@example.com': (550, '5.1.1 <noone@example.com>: Recipient address rejected: User unknown in virtual alias table')}

First note that the -q option has removed the status output shown in the previous example.  Only the error message is shown; and it verifies that the address is unknown.

You want to generate and inspect a bounce message

You may be able to generate an email bounce message by sending an email through one mail server functioning as an MTA (mail transfer agent) to a second mail server functioning as a MDA (mail delivery agent).  If the MTA passes your message to the MDA, and the recipient address is invalid, the MDA will return a bounce message to the return address.  Since in this case the recipient address and the return address must be different, we will need to use the -f option to mailservertest.  The -f option sets the return address.  By default, the recipient and return address are the same; this worked fine for the previous example, but not now.

mailservertest -f my-email@example.com  broken-email@other.com  mail.example.com

Here, broken-email@other.com is an invalid email address.  We first send the email to our MTA mail.example.com.  It will send the email to whatever mail server handles other.com.  The email bounce will be sent to our email address my-email@example.com.  Below is an example of what is returned.

Delivery has failed to these recipients or groups:

broken-email@other.com
The e-mail address you entered couldn't be found. Please check the recipient's e-mail address and try to resend the message. If the problem continues, please contact your helpdesk.

Diagnostic information for administrators:

Generating server: CASHUBE.grove.ad.other.com

broken-email@other.com
#550 5.1.1 RESOLVER.ADR.RecipNotFound; not found ##rfc822;broken-email@other.com

Original message headers:

Received: from mail.example.com (192.168.25.234) by
 CashubE.grove.ad.other.com (192.168.30.167) with Microsoft SMTP Server id
 14.2.347.0; Sun, 8 Dec 2013 17:25:49 -0500
Received: from localhost (host.example.com [192.168.80.33]) by
 mail.example.com (Postfix) with SMTP id 2F1007F44  for
 <broken-email@other.com>; Sun,  8 Dec 2013 17:25:49 -0500 (EST)
From: "my-email@example.com" <my-email@example.com>
Subject: MAILTEST: mail 17:25:49 2013-12-08
To: <broken-email@other.com>
Date: Sun, 8 Dec 2013 17:25:49 -0500
MIME-Version: 1.0
Content-Type: text/plain
Message-ID: <e91f8c64-57e5-47db-a2bc-9857ce0c04f3@CASHUBE.grove.ad.other.com>
Return-Path: my-email@example.com

Reporting-MTA: dns;CASHUBE.grove.ad.other.com
Received-From-MTA: dns;mail.example.com
Arrival-Date: Sun, 8 Dec 2013 22:25:49 +0000

Original-Recipient: rfc822;broken-email@other.com
Final-Recipient: rfc822;broken-email@other.com
Action: failed
Status: 5.1.1
Diagnostic-Code: smtp;550 5.1.1 RESOLVER.ADR.RecipNotFound; not found

ForwardedMessage.eml
Subject:
MAILTEST: mail 17:25:49 2013-12-08
From:
"my-email@example.com" <my-email@example.com>
Date:
12/08/2013 05:25 PM
To:
<broken-email@other.com>

This is a test email

   Tag             : MAILTEST:
   Date            : 17:25:49 2013-12-08
   All Recipients  : broken-email@other.com
   All Servers     : mail.example.com
   This Server     : mail.example.com

----------------------------------------------------
Sent from the script 'mailservertest'
on the host 'host'

Configuring Apache to Redirect All Traffic to One Local URL

GOAL

You want to redirect all requests to your web server to a single page on the server.

PROBLEM

Note that redirection to an entirely different server is trivial with RedirectMatch. You can redirect all traffic to http://my-server.org to http://other-server.org by adding this rule to the Apache config files for http://my-server.org

#  Redirecting to another server - WRONG!  Not our goal.
RedirectMatch ^.*$ http://other-server.org/

However, we want to redirect all traffic to one local URL, you might try this

#  Redirecting to this server - WRONG!  Creates infinite redirection.
RedirectMatch ^.*$  http://my-server.org/new.html

This will causes an infinite redirection loop, because every redirection to http://my-server.org/new.html will trigger the RedirectMatch rule.

SOLUTION

Use a negative lookahead in the regular expression. The following configuration will work

#  Redirecting to this server - CORRECT!
RedirectMatch ^(?!/new.html)$  http://my-server.org/new.html

The negative lookahead requires that the requested URL not match /new.html. This prevents the infinite redirection.

Setting up MySQL over TLS

MySQL supports session encryption using TLS. Here’s how to configure your server and client to use it.

On The Server

To start, you will need a server SSL certificate file and a key file, and a file containing the certificate that signed your cert. In the MySQL configuration file /etc/my.cnf or /etc/mysql/my.cnf, add these three lines to both the [mysqld] and [mysqld_safe] sections,

ssl-ca=SIGNING-CERT-FILE
ssl-cert=CERT-FILE
ssl-key=KEY-FILE

Restart your server so this new configuration will take effect.

Using MySQL Command-Line Client

You have two choices here.

Edit the my.cnf file

Add the following line under the [client] section

ssl-ca=SIGNING-CERT-FILE

and all subsequent network traffic when using the mysql command-line client will encrypted. The SIGNING-CERT-FILEis the same as above.

Use command-line option
You don’t need to edit the my.cnf file if you run the client like this

mysql --ssl-ca=SIGNING-CERT-FILE ...

Using Perl as a Client

First, you will need to configure my.cnf as above in the section “Using MySQL Command-Line Client”. Below is an example of how to call Perl’s DBI packag using DBI->connect with the mysql_ssl option

$handle = DBI->connect(
   "dbi:mysql:DB_NAME:DB_HOST:mysql_ssl=1", 
   DB_USER, 
   DB_PASSWORD
);

Replace DB_NAME, DB_HOST, DB_USER, DB_PASSWORD with the database name, host, user and user’s password. Warning: If the database does not support SSL, the connection will still succeed, but it will be plain text.

Using Python as a Client

You will need a copy of cert for the signing authority of the MySQL server’s cert, as in previous examples. The difference here is that Python will read the signing authority cert directly, and not via the MySQL my.cnf file. We use the Python’s MySQLdb module to connect to MySQL. Here’s is an example

import MySQLdb
dbh = MySQLdb.connect(
   host=DB_HOST,
   user=DB_USER,
   passwd=DB_PASS,
   db=DB_NAME,
   ssl={"ca":"SIGNING-CERT-FILE"}
)

Warning: If the database does not support SSL, the connection will still succeed, but it will be plain text.

Verifying SSL Transport

The only way to verify that your connection is using SSL is to sniff the traffic on the server or client using tcpdump, like this

tcpdump -nn -s2048 -X host CLIENT-OR-SERVER

where CLIENT-OR-SERVER is the IP address of the MySQL client if you are listening on the server, and vice versa.