fetch error in warning

[lluisga]

Hello,

In my /etc/munin/munin.conf file I want to include a warning to a certain field that I monitor and whose graph works ok. I end up with the following lines in the host tree of the munin.conf file:

[server.example.com]

address server.example.com
use_node_name yes
mysql_slave-heartbeat.warning 60

The problem is that when I check the /var/log/munin/munin-limits.log I get the following lines:

[WARNING] Could not fetch data from /var/lib/munin/example.com-server.example.com-mysql_slave-heartbeat-g.rrd(AVERAGE): opening '/var/lib/munin/example.com-server.example.com-mysql_slave-heartbeat-g.rrd': No such file or directory

The real file it should fetch has the followin path:

/var/lib/munin/example.com/server.example.com-mysql_slave-heartbeat-g.rrd

So it differs from the file it is trying to fetch by a change of a / for a -, but I can't figure how to implement this 'little' change so that the warning works.

Thanks in advance for your help.

[lluisga]

While I don't come up with another solution I've solved it with a symlink.

[bcg]

I bumped to the same problem. Symlink solved it to me too, but there should be a real fix!

Munin version 1.4.5 on RHEL5, following symlinks were needed:

lrwxrwxrwx 1 root  root     41 Nov 16 12:02 example.com-example.com-df_inode-_dev_md0-g.rrd -> example.com/example.com-df_inode-_dev_md0-g.rrd
lrwxrwxrwx 1 root  root     41 Nov 16 12:02 example.com-example.com-df_inode-_dev_md1-g.rrd -> example.com/example.com-df_inode-_dev_md1-g.rrd
lrwxrwxrwx 1 root  root     41 Nov 16 12:02 example.com-example.com-df_inode-_dev_shm-g.rrd -> example.com/example.com-df_inode-_dev_shm-g.rrd
lrwxrwxrwx 1 root  root     40 Nov 16 14:20 example.com-example.com-if_err_eth0-rcvd-c.rrd -> example.com/example.com-if_err_eth0-rcvd-c.rrd
lrwxrwxrwx 1 root  root     41 Nov 16 14:20 example.com-example.com-if_err_eth0-trans-c.rrd -> example.com/example.com-if_err_eth0-trans-c.rrd
lrwxrwxrwx 1 root  root     40 Nov 16 14:20 example.com-example.com-if_err_eth1-rcvd-c.rrd -> example.com/example.com-if_err_eth1-rcvd-c.rrd
lrwxrwxrwx 1 root  root     41 Nov 16 14:20 example.com-example.com-if_err_eth1-trans-c.rrd -> example.com/example.com-if_err_eth1-trans-c.rrd
lrwxrwxrwx 1 root  root     33 Nov 16 11:53 example.com-example.com-load-g.rrd -> example.com/example.com-load-load-g.rrd
lrwxrwxrwx 1 root  root     39 Nov 16 14:19 example.com-example.com-open_files-used-g.rrd -> example.com/example.com-open_files-used-g.rrd

[stephanhughson]

This is driving me mad. Where is it getting the incorrect path?!
I moved from 1.2 to 1.4 and have had this for some graphs only ever since.

[bcg]

I bumped to the same problem. Symlink solved it to me too, but there should be a real fix!

I don't know why but I don't need those symlinks anymore. Munin is still same version but I deleted all datafiles some time ago.

[jo]

I'm not sure how this can occur, and have been unable to reproduce myself. What I did find (and fixed in r4118) while looking at this, was issues with plugins with dashes in their name.

If anybody still gets this issue, please run munin-limits with --debug, and supply the log for the full run, as well as the munin.conf.

[jo]

Aha, I figured it out!

This bug can happen when field-level options are spesified at plugin level (i.e. one level above where they should be). E.g., if a "warning" or "critical" is put directly under a plugin, instead of under one of the fields in a plugin.

I'll get to fixing this (the fix will probably be a more sane error message.)

[Mickael]

Hi,

After installing Munin server and nodes, I got error messages in the /var/log/munin/munin-limits.log

2011/09/19 10:09:09 [WARNING] Could not fetch data from /var/lib/munin/defaultdomain-u-server-4-guest-5.defaultdomain-load-g.rrd(AVERAGE): opening '/var/lib/munin/defaultdomain-u-server-4-guest-5.defaultdomain-load-g.rrd': No such file or directory
2011/09/19 10:09:09 [WARNING] Could not fetch data from /var/lib/munin/defaultdomain-u-server-4-guest-5.defaultdomain-df_inode-_var_lock-g.rrd(AVERAGE): opening '/var/lib/munin/defaultdomain-u-server-4-guest-5.defaultdomain-df_inode-_var_lock-g.rrd': No such file or directory
2011/09/19 10:09:09 [WARNING] Could not fetch data from /var/lib/munin/defaultdomain-u-server-4-guest-5.defaultdomain-df_inode-_dev-g.rrd(AVERAGE): opening '/var/lib/munin/defaultdomain-u-server-4-guest-5.defaultdomain-df_inode-_dev-g.rrd': No such file or directory
2011/09/19 10:09:09 [WARNING] Could not fetch data from /var/lib/munin/defaultdomain-u-server-4-guest-5.defaultdomain-df_inode-_dev_shm-g.rrd(AVERAGE): opening '/var/lib/munin/defaultdomain-u-server-4-guest-5.defaultdomain-df_inode-_dev_shm-g.rrd': No such file or directory
2011/09/19 10:09:09 [WARNING] Could not fetch data from /var/lib/munin/defaultdomain-u-server-4-guest-5.defaultdomain-df_inode-_dev_sda1-g.rrd(AVERAGE): opening '/var/lib/munin/defaultdomain-u-server-4-guest-5.defaultdomain-df_inode-_dev_sda1-g.rrd': No such file or directory
2011/09/19 10:09:09 [WARNING] Could not fetch data from /var/lib/munin/defaultdomain-u-server-4-guest-5.defaultdomain-df_inode-_var_run-g.rrd(AVERAGE): opening '/var/lib/munin/defaultdomain-u-server-4-guest-5.defaultdomain-df_inode-_var_run-g.rrd': No such file or directory
2011/09/19 10:09:09 [WARNING] Could not fetch data from /var/lib/munin/defaultdomain-u-server-4-guest-5.defaultdomain-open_files-used-g.rrd(AVERAGE): opening '/var/lib/munin/defaultdomain-u-server-4-guest-5.defaultdomain-open_files-used-g.rrd': No such file or directory
2011/09/19 10:09:09 [WARNING] Could not fetch data from /var/lib/munin/defaultdomain-u-server-4-guest-5.defaultdomain-memory-free-g.rrd(AVERAGE): opening '/var/lib/munin/defaultdomain-u-server-4-guest-5.defaultdomain-memory-free-g.rrd': No such file or directory
2011/09/19 10:09:09 [WARNING] Could not fetch data from /var/lib/munin/defaultdomain-u-server-4-guest-5.defaultdomain-memory-swap-g.rrd(AVERAGE): opening '/var/lib/munin/defaultdomain-u-server-4-guest-5.defaultdomain-memory-swap-g.rrd': No such file or directory
2011/09/19 10:09:09 [WARNING] Could not fetch data from /var/lib/munin/defaultdomain-u-server-4-guest-5.defaultdomain-munin_stats-graph-g.rrd(AVERAGE): opening '/var/lib/munin/defaultdomain-u-server-4-guest-5.defaultdomain-munin_stats-graph-g.rrd': No such file or directory
2011/09/19 10:09:09 [WARNING] Could not fetch data from /var/lib/munin/defaultdomain-u-server-4-guest-5.defaultdomain-munin_stats-update-g.rrd(AVERAGE): opening '/var/lib/munin/defaultdomain-u-server-4-guest-5.defaultdomain-munin_stats-update-g.rrd': No such file or directory

I have searched for long time before found how to fix the problem. I first try to do symbolic links as you guys suggest it. It was working fine but was getting crazy when you have several servers to fix. I had even written a script to create those symbolic links. Although my script was creating 98% of the proper symbolic links, some errors remaining: bad file name + bad directory.

Finally, passing all the borring details, sorry it's a little bit long, to fix the problem you need to take cake of how you specify your limits(warning/critical) in your /etc/munin/munin.conf.

For instance:

[u-server-4-guest-5.defaultdomain]
        address 192.168.1.225
# Memory
        # Warning if higher than 1B
        memory.swap.warning 1
        # Critical if higher than 100MB
        memory.swap.critical 100000000
        # Warning if lower than 100MB
        memory.free.warning  100000000:
        # Critical if lower than 10MB
        memory.free.critical 10000000:
# Average load
        load.warning 4                        <-----
        load.critical 8                       <-----
#Network
        # Warning if higher than 20Mbits 
        if_eth0.up.warning 20000000
        # Critical if higher than 500Mbits
        if_eth0.up.critical 500000000
        # Need to setup type to COUNTER, default is GAUGE
        if_eth0.up.type COUNTER
        contacts mickael, nagios

The two lines(marked by arrows) will generate errors in the munin-limits.log. Noted that the error generated in the log file does not look connected to the service that generated it(excerted maybe the first line). Why ? I figure out that the proper config is:

        load.load.warning 4                        
        load.load.critical 8 

The format is

{service}.{internal name}.{warning/critical} value

You can know the name of the service, internal name, even the type of the field(GAUGE, DERIVE, COUNTER, etc..) if you use the user interface.

For instance for Load Average:

Overview :: defaultdomain :: u-server-4-guest-5.defaultdomain :: load 
...
Graphs
...
The load average of the machine describes how many processes are in the run-queue (scheduled to run "immediately").

Field 	Internal name 	Type 	Warn 	Crit 	Info
load 	load 	        gauge 	4  	8  	5 minute load average

In this example :

• the service name is given by the first line :u-server-4-guest-5.defaultdomain :: load --> service name is load

• the internal name is display in the table: load as well

• as the type is gauge, you don't need to setup the type manually.

So the proper config for setup a warning for Average Load is load.load.warning value

Hoping that will be helpful.

Cheers ;)

[Mickael]

Sorry I forgot my config:

Ubuntu server : Ubuntu 11.04 (GNU/Linux 2.6.38-10-virtual x86_64)
Munin version 1.4.5