Accessing Munin nodes through SSH tunnels

When there is a need to reach one or more hosts behind a non-routing *nix system, you may use SSH tunneling. In this configuration, SSH is used for its convenient tunneling, not for encryption. An alternative to using SSH is bouncing through (x)inetd.

Considering the illustration, an SSH connection is established between the Munin server and the in-between system (referred to as 'bouncer'). This connection will tunnel a number of forwarded ports between the local Munin server and the different Munin nodes. The connection will be encrypted between the Munin server and the bouncer, but unencrypted from the bouncer to the Munin nodes.

Create the SSH connection with the following command:

ssh -L 5001:Server-A:4949 -L 5002:Server-B:4949 -L 5003:Server-C:4949 user@bouncer

Please read the FAQ entry for useful SSH options'''

Then, munin.conf must be configured accordingly:

[localhost]                    # mark localhost with comments
  address 127.0.0.1            # this hides for munin-node check        
  port 4949                    #
  use_node_name yes            #

# use real hostnames in []

[Server-A]
  address 127.0.0.1
  port 5001
  use_node_name yes

[Server-B]
  address 127.0.0.1
  port 5002
  use_node_name yes

[Server-C]
  address 127.0.0.1
  port 5003
  use_node_name yes

If you have many tunnels to a bunch of servers in one or more DMZs you may want a mechanism, that takes care that the tunnel is up and running. Even if a tunnel is running yet, the connection can drop some day (a short-term network problem, a reboot of the munin-node etc). In this case, this machnism must restart the tunnel soon. Best way to implement this is to write a script, that parse the munin.conf to check if every tunnel is alive or not. The most easy way is an additional section in the /etc/init./munin if the munin-server also run a munin-node.

'check')
        # Start or restart ssh tunnel to munin clients in a DMZ
        # File with the munin configuration
        MUNINCONF=/etc/opt/munin/munin.conf
        # IP-Address for localhost (local tunnel endpoint)
        IP=127.0.0.1
        # tempoary list
        MUNINLIST=/tmp/munin.list

        cat $MUNINCONF  | grep -v "#" | grep "address $IP" --before-context=1 --after-context=1 > $MUNINLIST
        # set an 'end of file' marker
        echo "--" >> $MUNINLIST

        COUNTER=0
        cat $MUNINLIST | while read line ; do
                if [ "$line" != "--" ] ; then
                        if [ "$COUNTER" = "0" ] ; then
                                CLIENTNAME=`echo $line | tr -d '[] '
                        fi
                        if [ "`echo $line | grep port`" ] ; then
                                PORT=`echo $line | awk '{ print $2 }'`
                        fi
                        #echo "looking for $CLIENTNAME : $PORT"
                        COUNTER=`echo "$COUNTER + 1" | bc`
                else
                        echo "done with this triple, check for tunnel $CLIENTNAME port $PORT"
                        if [ "`ps -eaf | grep ssh | grep nNgf | grep $PORT | grep -v grep`" ] ; then
                                echo "tunnel to $CLIENTNAME port $PORT is up"
                        else
                                echo "SSH tunnel ${CLIENTNAME} NOT alive ... Restarting ..."
                                /usr/bin/ssh -nNgf -L $PORT:localhost:4949 root@$CLIENTNAME 
                                logger -p daemon.notice "SSH tunnel ${CLIENTNAME} NOT alive ... Restarting ...
"
                                sleep 1
                        fi
                        COUNTER=0
                        echo
                fi
        done
        ;;

If this script is called with the switch 'check' every tunnel is checkt and restarted if necessary :

.
.
.
done with this triple, check for tunnel Server-A port 5001
tunnel to Server-A port 5012 is up

done with this triple, check for Server-B port 5002
SSH tunnel Server-B NOT alive ... Restarting ...

done with this triple, check for tunnel Server-C port 5003
tunnel to Server-C port 5022 is up
.
.

Please take care, that the munin.conf follow the rules given in the example above: first the name (in brackets), then the address (the IP must match the value for the environment variable "IP-Address" in the check script. In the next line the port that should be used. The last thing you need is a cronjob, that runs every hour or more often. This cronjob just calls '/etc/init.d/munin check'.