View source for Esecuzione di un comando in caso di allarme Nagios (EventHandler)

==Esecuzione di un comando su server remoto==

Se si monitora un serivizio, è possibile far eseguire un comando quando il servizio entra per la prima volta in CRITICAL.

In questo caso facciamo ripartire il servizio NTP quando non si verifica che c'è una deriva dell'orario di sistema

===Configurazione server Nagios su macchina monitorante===

* Attivare l'event handlers nel servizio da monitorare, e definire il comando nagios da eseguire:

 sudoedit /etc/nagios3/conf.d/file.cfg

 #...
 event_handler_enabled           1
 event_handler                   restart-service!NTP
 #...

* Definire il comando nagios event handler

 sudoedit /etc/nagios3/conf.d/eventhandlers.cfg

<pre>
define command{
    command_name    restart-service
    command_line    /etc/nagios3/conf.d/eventhandlers/restart-service.sh           "$SERVICESTATE$" "$SERVICESTATETYPE$" "$SERVICEATTEMPT$" "$HOSTADDRESS$" "$ARG1$" "$SERVICEDESC$"
}
</pre>

* Creare lo script effettivo:

 sudo mkdir -p /etc/nagios3/conf.d/eventhandlers
 sudoedit /etc/nagios3/conf.d/eventhandlers/restart-service.sh

<pre>
#!/bin/sh                                                                                            
#
# Event handler script for restarting the nrpe server on the local machine
# Taken from the Nagios documentation and
# http://www.techadre.com/sites/techadre.com/files/event_handler_script_0.txt
# Adapted by L.C. Karssen
# Time-stamp: <2010-09-14 15:24:33 (root)>
#
# Note: This script will only restart the nrpe server if the service is
#       retried 3 times (in a "soft" state) or if the web service somehow
#       manages to fall into a "hard" error state.
#
 
date=`date`

#/etc/nagios3/conf.d/restart-service.sh
# define command{
#    command_name    restart-service
#    command_line    /etc/nagios3/conf.d/eventhandlers/ restart-service.sh           
#                       "$SERVICESTATE$" "$SERVICESTATETYPE$" "$SERVICEATTEMPT$" "$HOSTADDRESS$" 
#                       "$ARG1$" "$SERVICEDESC$"
#    }
# To check restart
# sudo -u nagios /etc/nagios3/conf.d/eventhandlers/restart-service.sh CRITICAL SOFT 1 hostname NTP TIME
#
# max_check_attempts 3
#

echo "$date - Eventhandler run #1=${1} #2=${2} #3=${3} #4=${4} #5=${5} #6=${6}"  >> /var/log/nagios3/eventhandlers.log
#Fri May 13 17:29:15 CEST 2011 - Eventhandler run #1=CRITICAL #2=HARD #3=3 #4=192.168.0.9 #5=cups #6=CUPS

# What state is the NRPE service in?
case "$1" in
OK)
        # The service just came back up, so don't do anything...
        ;;
WARNING)
        # We don't really care about warning states, since the service is probably still running...
        ;;
UNKNOWN)
        # We don't know what might be causing an unknown error, so don't do anything...
        ;;
CRITICAL)
        # Aha!  The BLAH service appears to have a problem - perhaps we should restart the server...
 
        # Is this a "soft" or a "hard" state?
        case "$2" in
 
        # Were in a soft state, meaning that Nagios is in the middle of retrying the
        # check before it turns into a "hard" state and contacts get notified
        SOFT)
                # What check attempt are we on?  We don't want to restart the web server on the first
                # check, because it may just be a fluke!
                case "$3" in
                # Wait until the check has been tried 3 times before restarting the web server.
                # If the check fails on the 4th time (after we restart the web server), the state
                # type will turn to "hard" and contacts will be notified of the problem.
                # Hopefully this will restart the web server successfully, so the 4th check will
                # result in a "soft" recovery.  If that happens no one gets notified because we
                # fixed the problem!
                1|2)
                        echo -n "Restarting service $6\n"
                        # Call NRPE to restart the service on the remote machine
                        /usr/lib/nagios/plugins/check_nrpe -H $4 -c restart-service -a $5
                        echo "$date - restart $6 (restart-service ${5}) - SOFT"  >> /var/log/nagios3/eventhandlers.log
                        ;;
                esac
                ;;
 
        # The service somehow managed to turn into a hard error without getting fixed.
        # It should have been restarted by the code above, but for some reason it didn't.
        # Let's give it one last try, shall we?
        # Note: Contacts have already been notified of a problem with the service at this
        # point (unless you disabled notifications for this service)
        HARD)
                case "$3" in
 
                3)
                        echo -n "Restarting $6 service...\n"
                        # Call the init script to restart the NRPE server
                        /usr/lib/nagios/plugins/check_nrpe -H $4 -c restart-service -a $5
                        echo "$date - restart $6 (restart-service $5) - HARD"  >> /var/log/nagios3/eventhandlers.log
                        ;;
                esac
                ;;
        esac
        ;;
esac
</pre>

 sudo chmod +x /etc/nagios3/conf.d/eventhandlers/restart-service.sh

* Creare il file di log
 sudo -u nagios touch /var/log/nagios3/eventhandlers.log

* Provare lo script
 sudo -u nagios /etc/nagios3/conf.d/eventhandlers/restart-service.sh CRITICAL SOFT 1 hostname NTP TIME

* Riavviare nagios
 
 sudo -u nagios nagios3 -v /etc/nagios3/nagios.cfg  && sudo invoke-rc.d nagios3 restart

===Configurazione agente nrpe su macchina monitorata===
====MAcchina linux====
* Definire il comando da eseguire
 sudoedit /etc/nagios/nrpe_local.cfg

 command[restart-service]=/usr/bin/sudo /usr/sbin/invoke-rc.d '$ARG1$' restart

* Abilitarne l'esecuzione allo user nagios, senza password:
 sudo visudo

 Cmnd_Alias NAGIOS_EH = /usr/sbin/invoke-rc.d

 nagios      ALL=NOPASSWD: NAGIOS_EH

* Abilitare il debug di nrpe, se si vuole debuggare
 sudoedit /etc/nagios/nrpe.cfg

 debug=1

* Riavviarlo:
 sudo invoke-rc.d nagios-nrpe-server restart

===Test===

* Fermare il servizio, o inserire un dummy check command

* Verificare nel log che l'event handler sia eseguito da nagios:
 sudo tail -f /var/log/nagios3/eventhandlers.log

<pre>
Fri May 13 17:37:08 CEST 2011 - Eventhandler run #1=CRITICAL #2=SOFT #3=1 #4=192.168.0.9 #5=cups #6=CUPS
Fri May 13 17:38:08 CEST 2011 - Eventhandler run #1=CRITICAL #2=SOFT #3=2 #4=192.168.0.9 #5=cups #6=CUPS
Fri May 13 17:38:08 CEST 2011 - restart CUPS (restart-service cups) - SOFT
Fri May 13 17:39:08 CEST 2011 - Eventhandler run #1=CRITICAL #2=HARD #3=3 #4=192.168.0.9 #5=cups #6=CUPS
Fri May 13 17:39:08 CEST 2011 - restart CUPS (restart-service cups) - HARD
Fri May 13 17:49:15 CEST 2011 - Eventhandler run #1=OK #2=HARD #3=3 #4=192.168.0.9 #5=cups #6=CUPS
</pre>

*Verificare che il comando sia eseguito da nrpe:
 sudo tail -f /var/log/syslog | grep nrpe

<pre>
May 13 17:39:08 galserver nrpe[4307]: Connection from 192.168.0.8 port 5803
May 13 17:39:08 galserver nrpe[4307]: Handling the connection...
May 13 17:39:08 galserver nrpe[4307]: Host is asking for command 'restart-service' to be run...
May 13 17:39:08 galserver nrpe[4307]: Running command: /usr/bin/sudo /usr/sbin/invoke-rc.d 'cups' restart
May 13 17:39:13 galserver nrpe[4307]: Command completed with return code 0 and output: Restarting Common Unix Printing  System: cupsd.
May 13 17:39:13 galserver nrpe[4307]: Return Code: 0, Output: Restarting Common Unix Printing System: cupsd.
May 13 17:39:13 galserver nrpe[4307]: Connection from 192.168.0.8 closed.
</pre>

* Disabilatare il debug nrpe

==Riferimenti==
*[http://blog.gnucom.cc/2009/event-handlers-in-nagios-using-nrpe/ Event handlers in nagios using NRPE.]
*[https://ssl-28158.1blu.de/wiked/doku.php?id=nagios_remote-event-handler_example nagios_remote-event-handler_example [gradweil.de]]