recurring mounts (like snapshot backups) cause failures from diskstats/iostats

[mushuweasel]

We have a daily snapshot backup procedure that causes munin-update to throw UNKNOWNs on both diskstats and iostat_ios. When the snapshot is mounted, the next fetch on diskstats and iostat_ios logs in munin-node.log:

2012/08/05-07:10:06 [28786] Error output from diskstats:
2012/08/05-07:10:06 [28786]     Use of uninitialized value in subtraction (-) at /etc/munin/plugins/diskstats line 157.
2012/08/05-07:10:06 [28786]     Use of uninitialized value in subtraction (-) at /etc/munin/plugins/diskstats line 158.
2012/08/05-07:10:06 [28786]     Use of uninitialized value in subtraction (-) at /etc/munin/plugins/diskstats line 160.
2012/08/05-07:10:06 [28786]     Use of uninitialized value in subtraction (-) at /etc/munin/plugins/diskstats line 161.
2012/08/05-07:10:06 [28786]     Use of uninitialized value in subtraction (-) at /etc/munin/plugins/diskstats line 163.
2012/08/05-07:10:06 [28786]     Use of uninitialized value in subtraction (-) at /etc/munin/plugins/diskstats line 164.
2012/08/05-07:10:06 [28786]     Use of uninitialized value in subtraction (-) at /etc/munin/plugins/diskstats line 166.
2012/08/05-07:10:06 [28786]     Use of uninitialized value in subtraction (-) at /etc/munin/plugins/diskstats line 157.
2012/08/05-07:10:06 [28786]     Use of uninitialized value in subtraction (-) at /etc/munin/plugins/diskstats line 158.
2012/08/05-07:10:06 [28786]     Use of uninitialized value in subtraction (-) at /etc/munin/plugins/diskstats line 160.
2012/08/05-07:10:06 [28786]     Use of uninitialized value in subtraction (-) at /etc/munin/plugins/diskstats line 161.
2012/08/05-07:10:06 [28786]     Use of uninitialized value in subtraction (-) at /etc/munin/plugins/diskstats line 163.
2012/08/05-07:10:06 [28786]     Use of uninitialized value in subtraction (-) at /etc/munin/plugins/diskstats line 164.
2012/08/05-07:10:06 [28786]     Use of uninitialized value in subtraction (-) at /etc/munin/plugins/diskstats line 166.
2012/08/05-07:10:06 [28786]     Use of uninitialized value in subtraction (-) at /etc/munin/plugins/diskstats line 157.
2012/08/05-07:10:06 [28786]     Use of uninitialized value in subtraction (-) at /etc/munin/plugins/diskstats line 158.
2012/08/05-07:10:06 [28786]     Use of uninitialized value in subtraction (-) at /etc/munin/plugins/diskstats line 160.
2012/08/05-07:10:06 [28786]     Use of uninitialized value in subtraction (-) at /etc/munin/plugins/diskstats line 161.
2012/08/05-07:10:06 [28786]     Use of uninitialized value in subtraction (-) at /etc/munin/plugins/diskstats line 163.
2012/08/05-07:10:06 [28786]     Use of uninitialized value in subtraction (-) at /etc/munin/plugins/diskstats line 164.
2012/08/05-07:10:06 [28786]     Use of uninitialized value in subtraction (-) at /etc/munin/plugins/diskstats line 166.
2012/08/05-07:10:08 [28786] Error output from iostat_ios:
2012/08/05-07:10:08 [28786]     Use of uninitialized value in subtraction (-) at /etc/munin/plugins/iostat_ios line 194.
2012/08/05-07:10:08 [28786]     Use of uninitialized value in subtraction (-) at /etc/munin/plugins/iostat_ios line 195.
2012/08/05-07:10:08 [28786]     Use of uninitialized value in subtraction (-) at /etc/munin/plugins/iostat_ios line 197.
2012/08/05-07:10:08 [28786]     Use of uninitialized value in subtraction (-) at /etc/munin/plugins/iostat_ios line 198.
2012/08/05-07:10:08 [28786]     Use of uninitialized value in subtraction (-) at /etc/munin/plugins/iostat_ios line 194.
2012/08/05-07:10:08 [28786]     Use of uninitialized value in subtraction (-) at /etc/munin/plugins/iostat_ios line 195.
2012/08/05-07:10:08 [28786]     Use of uninitialized value in subtraction (-) at /etc/munin/plugins/iostat_ios line 197.
2012/08/05-07:10:08 [28786]     Use of uninitialized value in subtraction (-) at /etc/munin/plugins/iostat_ios line 198.
2012/08/05-07:10:08 [28786]     Use of uninitialized value in subtraction (-) at /etc/munin/plugins/iostat_ios line 194.
2012/08/05-07:10:08 [28786]     Use of uninitialized value in subtraction (-) at /etc/munin/plugins/iostat_ios line 195.
2012/08/05-07:10:08 [28786]     Use of uninitialized value in subtraction (-) at /etc/munin/plugins/iostat_ios line 197.
2012/08/05-07:10:08 [28786]     Use of uninitialized value in subtraction (-) at /etc/munin/plugins/iostat_ios line 198.

and results in an UNKNOWN alert. Same results from v1.4.6 and most recent commit d110606b4a8979d81aa47531bd140cfd85e80a92

[kenyon]

Will need to know what your devices look like in order to debug this. Can you post the output of the following when your snapshot is mounted:

ls -l /sys/block/*/stat /dev/mapper
cat /proc/diskstats
cat /proc/devices

[mushuweasel]

Output of requested data for pre-backup and during-backup states

[mushuweasel]

Attached output. Note, each time backup runs, the snapshot is lvcreate'd with the same name, mount'ed for the duration of generating a tarball, then umount'ed and lvremove'd.