Bug in munin-cgi-graph and "diskstats" with Linux MD devices

[robertns]

Hello,

I recently installed munin-2.0.0 (although this bug seems to have existed for a while in the beta and rc releases).

munin-cgi-graph seems to fail when trying to view "diskstats" graphs (ie. diskstats_iops, diskstats_latency, diskstats_utilization etc..) when the target is a Linux metadisk device with a name of the form "dm-X" (it works ok for devices with more conventional device names such as "sda").

The error I see on my browser is this:

Could not draw graph "/tmp/munin-cgi-graph/Athena/chaos/diskstats_latency/dm-0-day.png"
Status: 200
Content-Type: image/png
X-Munin-Pid: 3829
X-Munin-Request: 0/0
Content-Length: 
Last-Modified: Thu, 01 Jan 1970 00:00:00 GMT
Expires: Thu, 01 Jan 1970 00:05:00 GMT

<h1>Software error:</h1>
<pre>[FATAL] Could not open image file &quot;/tmp/munin-cgi-graph/Athena/chaos/diskstats_latency/dm-0-day.png&quot; for reading: No such file or directory
</pre>
<p>
For help, please send mail to the webmaster (<a href="mailto:root@localhost">root@localhost</a>), giving this error message 
and the time and date of the error.

</p>

.. and here is logfile fragment showing the problem:

2012/06/07 13:13:35 Opened log file
2012/06/07 13:13:35 Request path is /Athena/chaos/diskstats_latency/dm-0-day.png
2012/06/07 13:13:35 asked for (Athena/chaos, diskstats_latency, dm-0, day)
2012/06/07 13:13:35 Starting munin-graph
2012/06/07 13:13:35 [INFO] Looking into drawing /tmp/munin-cgi-graph/Athena/chaos/diskstats_latency/dm-0-day.png
2012/06/07 13:13:35 [RRD ERROR] Unable to graph /tmp/munin-cgi-graph/Athena/chaos/diskstats_latency/dm-0-day.png : rrdc_flush (/var/munin-2.0/Athena/chaos-diskstats_latency-dm_0-avgwrwait-g.rrd) failed with status -1.
2012/06/07 13:13:35 [RRD ERROR] rrdtool 'graph' '/tmp/munin-cgi-graph/Athena/chaos/diskstats_latency/dm-0-day.png' \
	'--title' \
	'Disk latency for /dev/mapper/sysvg-root - by day' \
	'--start' \
	'-2000m' \
	'--base' \
	'1000' \
	'--vertical-label' \
	'seconds' \
	'--slope-mode' \
	'--height' \
	'175' \
	'--width' \
	'450' \
	'--imgformat' \
	'PNG' \
	'--font' \
	'DEFAULT:0:DejaVuSans,DejaVu Sans,DejaVu LGC Sans,Bitstream Vera Sans' \
	'--font' \
	'LEGEND:7:DejaVuSansMono,DejaVu Sans Mono,DejaVu LGC Sans Mono,Bitstream Vera Sans Mono,monospace' \
	'--color' \
	'BACK#F0F0F0' \
	'--color' \
	'FRAME#F0F0F0' \
	'--color' \
	'CANVAS#FFFFFF' \
	'--color' \
	'FONT#666666' \
	'--color' \
	'AXIS#CFD6F8' \
	'--color' \
	'ARROW#CFD6F8' \
	'--border' \
	'0' \
	'-W' \
	'Munin 2.0.0' \
	'DEF:aavgwrwait=/var/munin-2.0/Athena/chaos-diskstats_latency-dm_0-avgwrwait-g.rrd:42:MAX' \
	'DEF:iavgwrwait=/var/munin-2.0/Athena/chaos-diskstats_latency-dm_0-avgwrwait-g.rrd:42:MIN' \
	'DEF:gavgwrwait=/var/munin-2.0/Athena/chaos-diskstats_latency-dm_0-avgwrwait-g.rrd:42:AVERAGE' \
	'DEF:aavgrdwait=/var/munin-2.0/Athena/chaos-diskstats_latency-dm_0-avgrdwait-g.rrd:42:MAX' \
	'DEF:iavgrdwait=/var/munin-2.0/Athena/chaos-diskstats_latency-dm_0-avgrdwait-g.rrd:42:MIN' \
	'DEF:gavgrdwait=/var/munin-2.0/Athena/chaos-diskstats_latency-dm_0-avgrdwait-g.rrd:42:AVERAGE' \
	'DEF:aavgwait=/var/munin-2.0/Athena/chaos-diskstats_latency-dm_0-avgwait-g.rrd:42:MAX' \
	'DEF:iavgwait=/var/munin-2.0/Athena/chaos-diskstats_latency-dm_0-avgwait-g.rrd:42:MIN' \
	'DEF:gavgwait=/var/munin-2.0/Athena/chaos-diskstats_latency-dm_0-avgwait-g.rrd:42:AVERAGE' \
	'DEF:asvctm=/var/munin-2.0/Athena/chaos-diskstats_latency-dm_0-svctm-g.rrd:42:MAX' \
	'DEF:isvctm=/var/munin-2.0/Athena/chaos-diskstats_latency-dm_0-svctm-g.rrd:42:MIN' \
	'DEF:gsvctm=/var/munin-2.0/Athena/chaos-diskstats_latency-dm_0-svctm-g.rrd:42:AVERAGE' \
	'CDEF:csvctm=gsvctm,POP,UNKN' \
	'COMMENT:                          ' \
	'COMMENT: Cur\:' \
	'COMMENT:Min\:' \
	'COMMENT:Avg\:' \
	'COMMENT:Max\:  \j' \
	'LINE1:gsvctm#00CC00:Average device IO time     ' \
	'GPRINT:csvctm:LAST:%6.2lf%s' \
	'GPRINT:isvctm:MIN:%6.2lf%s' \
	'GPRINT:gsvctm:AVERAGE:%6.2lf%s' \
	'GPRINT:asvctm:MAX:%6.2lf%s\j' \
	'CDEF:cavgwait=gavgwait,POP,UNKN' \
	'LINE1:gavgwait#0066B3:Average IO Wait time       ' \
	'GPRINT:cavgwait:LAST:%6.2lf%s' \
	'GPRINT:iavgwait:MIN:%6.2lf%s' \
	'GPRINT:gavgwait:AVERAGE:%6.2lf%s' \
	'GPRINT:aavgwait:MAX:%6.2lf%s\j' \
	'CDEF:cavgrdwait=gavgrdwait,POP,UNKN' \
	'LINE1:gavgrdwait#FF8000:Average Read IO Wait time  ' \
	'GPRINT:cavgrdwait:LAST:%6.2lf%s' \
	'GPRINT:iavgrdwait:MIN:%6.2lf%s' \
	'GPRINT:gavgrdwait:AVERAGE:%6.2lf%s' \
	'GPRINT:aavgrdwait:MAX:%6.2lf%s\j' \
	'CDEF:cavgwrwait=gavgwrwait,POP,UNKN' \
	'LINE1:gavgwrwait#FFCC00:Average Write IO Wait time ' \
	'GPRINT:cavgwrwait:LAST:%6.2lf%s' \
	'GPRINT:iavgwrwait:MIN:%6.2lf%s' \
	'GPRINT:gavgwrwait:AVERAGE:%6.2lf%s' \
	'GPRINT:aavgwrwait:MAX:%6.2lf%s\j' \
	'COMMENT:Last update\: Thu Jan  1 10\:00\:00 1970\r'
2012/06/07 13:13:35 [INFO] Graphed service Athena;chaos:diskstats_latency.dm_0 (0.01 sec for 1 graphs)
2012/06/07 13:13:35 Munin-graph finished (0.01 sec)
2012/06/07 13:13:35 [WARNING] Could not draw graph "/tmp/munin-cgi-graph/Athena/chaos/diskstats_latency/dm-0-day.png": /tmp/munin-cgi-graph/Athena/chaos/diskstats_latency/dm-0-day.png
2012/06/07 13:13:35 [PERL WARNING] Use of uninitialized value $epoch in gmtime at /usr/local/munin-2.0/www/cgi/munin-cgi-graph line 289.
2012/06/07 13:13:35 [PERL WARNING] Use of uninitialized value $mtime_epoch in modulus (%) at /usr/local/munin-2.0/www/cgi/munin-cgi-graph line 276.
2012/06/07 13:13:35 [PERL WARNING] Use of uninitialized value $mtime_epoch in subtraction (-) at /usr/local/munin-2.0/www/cgi/munin-cgi-graph line 276.
2012/06/07 13:13:35 [PERL WARNING] Use of uninitialized value in concatenation (.) or string at /usr/local/munin-2.0/www/cgi/munin-cgi-graph line 243.
2012/06/07 13:13:35 [FATAL] Could not open image file "/tmp/munin-cgi-graph/Athena/chaos/diskstats_latency/dm-0-day.png" for reading: No such file or directory

I've traced this problem through munin-cgi-graph. The problem
appears to be that the grapher is looking for an RRD with a name
of (for example)

chaos-diskstats_iops-dm_0-avgwrrqsz-g.rrd

.. but the actual RRD file is called

chaos-diskstats_iops-dm-0-avgwrrqsz-g.rrd

Note that the file has a hyphen between the "dm" and the "0", but the
grapher is trying to use an underscore instead.

"munin-cgi-graph" does not seem to be altering anything, so I assume that the buggy code is somewhere in GraphOld?.pm, but delving into that is a little beyond my knowledge of the system at this point.

Regards,

Robert.

[kenyon]

This one looks similar to #1218.

[robertns]

Yes it has that appearance, but it's actually a separate bug. I logged #1218 and that fixed an issue with graphing in general, but this one then crops up affecting the "diskstats" plugin specifically.

[robertns]

As a workaround to fix this problem, I applied the following patch to GraphOld?.pm:

--- GraphOld?.pm.orig 2012-07-17 15:37:27.000000000 +1000
+++ GraphOld?.pm 2012-07-17 15:37:46.000000000 +1000
@@ -987,6 +987,7 @@

# Getting name of rrd file
$filename = munin_get_rrd_filename($field, $path);

+ $filename =~ s/-dm_/-dm-/ if ($filename =~ /-dm_[0-9]+-(rdio|wrio|avg|svc|rdbytes|wrbytes)/);

if (! $filename) {

ERROR "[ERROR] filename is empty for " . munin_dumpconfig_as_str($field) . ", $path";
# Ignore this field

This is just a workaround pending (hopefully) a better fix.