I came across this today and learned something new so thought I would share it here.
After killing 2 processes that had hung I noticed the following in the ps output:
root 373 2 0 Jun11 ? 00:00:00 [kdmflush] root 375 2 0 Jun11 ? 00:00:00 [kdmflush] root 863 2 0 Jun11 ? 00:00:00 [kdmflush] root 867 2 0 Jun11 ? 00:00:00 [kdmflush] root 1132 2 0 Jun11 ? 00:01:03 [flush-253:0] root 1133 2 0 Jun11 ? 00:00:43 [flush-253:2]
Now kdmflush I am use to seeing, but flush-253: was something I had never noticed so I decided to dig. I started with man flush but that seemed to lead no where since I am not running sendmail or any mail server. I turned to google (not to proud to admit it) and searched “linux process flush”. Turns out ‘flush-# is kernel garbage collection that flushes unused memory allocations to disk so the RAM can be reused. So ‘flush’ is trying to write out dirty pages from virtual memory, most likely associated with the processes I just killed.
I discovered these commands that shed more light on what is actually happening:
grep 253 /proc/self/mountinfo 20 1 253:0 / / rw,relatime - ext4 /dev/mapper/vg_kfs10-lv_root rw,seclabel,barrier=1,data=ordered 25 20 253:3 / /home rw,relatime - ext4 /dev/mapper/vg_kfs10-lv_home rw,seclabel,barrier=1,data=ordered 26 20 253:2 / /var rw,relatime - ext4 /dev/mapper/vg_kfs10-LogVol03 rw,seclabel,barrier=1,data=ordered
Remember my listings were for flush-253:0 and flush-253:2 so I now know what partitions are being worked with. Another interesting command to use is the following, which shows the activity of writing out dirty pages:
watch grep -A 1 dirty /proc/vmstat nr_dirty 2 nr_writeback 0
If these numbers are significantly higher you might be having a bigger problem on your system. Though from what I have read this sometimes indicates sync’ing. If this becomes a problem on your server you can set system parameters in /etc/sysctl.conf to head this off by adding the following lines:
vm.dirty_background_ratio = 50 vm.dirty_ratio = 80
Then (as root) execute:
# sysctl -p
The “vm.dirty_background_ratio” tells at what ratio should the linux kernel start the background task of writing out dirty pages. The above increases this setting from the default 10% to 50%. The “vm.dirty_ratio” tells at what ratio all IO writes become synchronous, meaning that we cannot do IO calls without waiting for the underlying device to complete them (which is something you never want to happen).
I did not add these to the sysctl.conf file but thought it worth documenting.