Este articulo describe como hacer una instalación de munin para una demanda de 30.000+ datos de multiples servidores.
Conocimiento de Munin.
La arquitectura se compone en un recolector y n servidores clientes.
Haga una instalación estandar siguiento el munin howto
Munin usualmente genera las graficas con su cron de actualización. Esto con un gran número de datos es insostenible, entonces es mejor generar las graficas SOLO bajo demanda.
Los pasos son
OJO con los permisos .. todo lo que toque el cgi debe ser de apache (www-data) . .. como /var/log/munin/munin-cgi* o /var/lib/munin/cgi-tmp
http://munin-monitoring.org/wiki/MuninConfigurationMasterCGI
Cuando se tiene un nodo que tiene demasiados datos no es posible recogerlos con una visita porque llega al timeout.
La solucion esta en que el nodo obtenga sus datos autonomamente y esto son recogidos por el recolector por SSH usando intercambio de llaves y asi evitando los timeouts generado por plugins lentos.
Se crea un servicio paralelo a munin-node llamado munin-async, se crea un usuario y intercambian llaves y los datos se traen de un spool usando ssh
El munin-async hace consultas al munin-node y las almacena en el spool. Si el puerto es diferente al 4949 toca decirselo.
- http://munin.readthedocs.org/en/latest/node/async.html –
- http://blog.hbis.fr/2013/07/25/munin-async_mode/ –
- http://www.matija.si/system-administration/2012/07/15/installing-munin-async/ –
El formato de la llave, requirio ciertos afinamientos. La que me funciona es del tipo
.ssh/authorized_keys
no-port-forwarding,no-agent-forwarding,no-X11-forwarding,command=“/var/lib/munin-async –spoolfetch –hostname localhost:4949” ssh-rsa AAAAB3NzaC1ysfadfafadfadfc2EAAA munin@kvdstat.com
Cuando se tienen muchos datos, el numero de escrituras a disco aumenta radicalmente ya que cada valor debe ser almacenado en un archivo RRD y si son 10.000 cada 5 minutos. !!!
RRDCache es un demononio que recibe todas las actualizaciones para RRDs y las almacena y va hacendo escrituras mas grandes y menos frecuentes lo que reduce el IO del disco y aumenta considerablemente el desempeño.
http://lzone.de/blog/HowTo-Munin-and-rrdcached-on-Ubuntu-12.04
http://munin-monitoring.org/wiki/rrdcached
Suponiendo que de la misma maquina tengo que recoger muchos datos. He descubierto por prueba y error que el limite viene siendo como 6 o 7 mil datos. Una vez se supera esta barrera, el nodo no puede enviar los datos sin tener timeout. No importa si los datos los genera un solo plugin o 7000 plugins .. el resultado es el mismo: timeout.
La solucion esta en correr varias instancias de munin-node (y munin-async en caso de estarlo usando). Como hacerlo:
PREFIX = $(DESTDIR)/opt/munin
a
PREFIX = $(DESTDIR)/opt/munin3 o /usr/local/munin3 .. como prefiera
Con la instalacion yo tuve algunos inconvenientes con los directorios que me toco alterarlos. Haciendo un “grep -R munin3 /opt/munin3/*” en una de mis instancias tengo los sitios a ajustar
etc/munin-node.conf:log_file /opt/munin3/log/munin-node.log
etc/munin-node.conf:pid_file /opt/munin3/var/run/munin-node.pid
lib/munin-async:require '/usr/local/munin3/share/perl5/Munin/Common/Defaults.pm';
lib/munin-async:my $SPOOLDIR = “/opt/munin3/var/munin-async”;
lib/munin-asyncd:require '/usr/local/munin3/share/perl5/Munin/Common/Defaults.pm';
lib/munin-asyncd:my $SPOOLDIR = “/opt/munin3/var/munin-async”;
sbin/munin-node:require '/usr/local/munin3/share/perl5/Munin/Common/Defaults.pm';
sbin/munin-run:require '/usr/local/munin3/share/perl5/Munin/Common/Defaults.pm';
share/perl5/Munin/Common/Defaults.pm:our $MUNIN_PREFIX = q{/opt/munin3};
share/perl5/Munin/Common/Defaults.pm:our $MUNIN_CONFDIR = q{/opt/munin3/etc};
share/perl5/Munin/Common/Defaults.pm:our $MUNIN_BINDIR = q{/opt/munin3/bin};
share/perl5/Munin/Common/Defaults.pm:our $MUNIN_SBINDIR = q{/opt/munin3/sbin};
share/perl5/Munin/Common/Defaults.pm:our $MUNIN_DOCDIR = q{/opt/munin3/doc};
share/perl5/Munin/Common/Defaults.pm:our $MUNIN_LIBDIR = q{/opt/munin3/lib};
share/perl5/Munin/Common/Defaults.pm:our $MUNIN_DBDIR = q{/opt/munin3/var/munin};
share/perl5/Munin/Common/Defaults.pm:our $MUNIN_PLUGSTATE = q{/opt/munin3/var/munin-node/plugin-state};
share/perl5/Munin/Common/Defaults.pm:our $MUNIN_SPOOLDIR = q{/opt/munin3/var/munin-async};
share/perl5/Munin/Common/Defaults.pm:our $MUNIN_MANDIR = q{/opt/munin3/man};
share/perl5/Munin/Common/Defaults.pm:our $MUNIN_LOGDIR = q{/opt/munin3/log};
share/perl5/Munin/Common/Defaults.pm:our $MUNIN_STATEDIR = q{/opt/munin3/var/run};
var/munin-async/.ssh/authorized_keys:no-port-forwarding,no-agent-forwarding,no-X11-forwarding,command=“/usr/local/munin3/lib/munin-async –spoolfetch –hostname localhost:4951” ssh-rsa AAAAB3NzaC1yc2EAAA munin@kvdstat.com
OJO con este ultimo que tambien incluye el puerto en el que estoy sirviendo al igual que la libreria de munin-async
lib/munin-asyncd:my $host = “localhost:4951”;
Este fue mi ultimo descubrimiento. Tenia que hacer 2000 graficas de un solo valor tomado desde una base de datos. La primera opcion obvia fue el miplugin_* y 2000 links a ese plugin del tipo miplugin_1967 para cada dato. Y este plugin simplemnte miraba una base de datos .. y pues hacia 2000 consultas.
Multigraph me permite hacer un solo plugin, con una sola consulta a mi BD y en poder construir una grafica con todos los datos (el cual omito porque son demasiados datos para una grafica), al mismo tiempo que tambien genera las graficas para cada uno de los valores. En esencia un plugin de multigraph (miplugin_multi) retorna lo mismo que corriendo los 2000 plugines en secuencia, claro con un plugin 0000 que es que tiene todos los valores. Por ejemplo .. si corro
munin-run miplugin_multi config
multigraph kvd_dispon
graph_category KVD-Dispon
graph_title Disponibilidad
graph_vlabel Cumplimiento
multigraph kvd_dispon.49822
graph_category KVD-Dispon-Nodo
graph_title Disponibilidad 49822
graph_vlabel Cumplimiento
49822-percent.label Porcentaje (20H)
multigraph kvd_dispon.49823
graph_category KVD-Dispon
graph_title Disponibilidad Kiosko 49823
graph_vlabel Cumplimiento
49823-percent.label Porcentaje (20H)
multigraph kvd_dispon.50000
graph_category KVD-Dispon
………………
Y los valores
munin-run miplugin_multi
multigraph kvd_dispon
multigraph kvd_dispon.49822
49822-percent.value 7.00
multigraph kvd_dispon.49823
49823-percent.value 31.00
multigraph kvd_dispon.50000
50000-percent.value 80.00
Observe que estoy omitiendo el consolidador, ni le doy label, ni le doy valores. Esto funciona perfecto para nosotros, porque me genera el dato de 2000 fuentes en una sola consulta que toma como 2 segundos en ejecutarse.
Referencias
http://munin-monitoring.org/wiki/MultigraphSampleOutput
https://munin.readthedocs.org/en/latest/plugin/multigraphing.html#plugin-multigraphing
-
-
FIN
Advertencia
Este documento es privado y es de u so exclusivo de sus autores y de SKINA TECH. Cualquier uso sin una autorización escrita es contra la ley de derechos de autor y de propiedad intelectual, y será motivo de una acción legal.
ANEXOS
001-default configuracion apache para munin-cgi
<VirtualHost *:80> # ServerName munin.example.org # ServerAlias munin
<IfModule !mod_rewrite.c>
# required because we serve out of the cgi directory and URLs are relative
Alias /munin-cgi/munin-cgi-html/static /var/cache/munin/www/static
RedirectMatch ^/$ /munin-cgi/munin-cgi-html/
</IfModule>
<IfModule mod_rewrite.c>
# Rewrite rules to serve traffic from the root instead of /munin-cgi
RewriteEngine On
# Static files
RewriteRule ^/favicon.ico /var/cache/munin/www/static/favicon.ico [L]
RewriteRule ^/static/(.*) /var/cache/munin/www/static/$1 [L]
# HTML
RewriteRule ^(/.*\.html)?$ /munin-cgi/munin-cgi-html/$1 [PT]
# Images
RewriteRule ^/munin-cgi/munin-cgi-graph/(.*) /$1
RewriteCond %{REQUEST_URI} !^/static
RewriteRule ^/(.*.png)$ /munin-cgi/munin-cgi-graph/$1 [L,PT]
</IfModule>
# Ensure we can run (fast)cgi scripts
ScriptAlias /munin-cgi/munin-cgi-graph /usr/lib/munin/cgi/munin-cgi-graph
<Location /munin-cgi/munin-cgi-graph>
Options +ExecCGI
<IfModule mod_fcgid.c>
SetHandler fcgid-script
</IfModule>
<IfModule mod_fastcgi.c>
SetHandler fastcgi-script
</IfModule>
<IfModule !mod_fastcgi.c>
<IfModule !mod_fcgid.c>
SetHandler cgi-script
</IfModule>
</IfModule>
Allow from all
# Replace above with "Require all granted" if running apache 2.4
</Location>
ScriptAlias /munin-cgi/munin-cgi-html /usr/lib/munin/cgi/munin-cgi-html
<Location /munin-cgi/munin-cgi-html>
Options +ExecCGI
<IfModule mod_fcgid.c>
SetHandler fcgid-script
</IfModule>
<IfModule mod_fastcgi.c>
SetHandler fastcgi-script
</IfModule>
<IfModule !mod_fastcgi.c>
<IfModule !mod_fcgid.c>
SetHandler cgi-script
</IfModule>
</IfModule>
Allow from all
# Replace above with "Require all granted" if running apache 2.4
</Location>
<IfModule !mod_rewrite.c>
<Location /munin-cgi/munin-cgi-html/static>
# this needs to be at the end to override the above sethandler directives
Options -ExecCGI
SetHandler None
</Location>
</IfModule>
</VirtualHost>
munin-node3 script de inicio en Debian
#! /bin/bash
### BEGIN INIT INFO # Provides: munin-node3 # Required-Start: $network $named $local_fs $remote_fs # Required-Stop: $network $named $local_fs $remote_fs # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: Start/stop Munin-Node 3 # Description: Start/stop Munin-Node 3 ### END INIT INFO
PATH=/sbin:/bin:/usr/sbin:/usr/bin DAEMON=/opt/munin3/sbin/munin-node PIDFILE=/opt/munin3/var/run/munin-node.pid CONFFILE=/opt/munin3/etc/munin-node.conf DAEMON_ARGS=“–config $CONFFILE”
. /lib/lsb/init-functions [ -r /etc/default/munin-node ] && . /etc/default/munin-node
if [ ! -x $DAEMON ]; then
log_failure_msg "Munin-Node3 appears to be uninstalled."
exit 5
elif [ ! -e $CONFFILE ]; then # log_failure_msg “Munin-Node3 appears to be unconfigured.”
exit 6
fi
# Figure out if the pid file is in a non-standard location while read line; do
line=${line%%\#*} # get rid of comments
set -f
line=$(echo $line) # get rid of extraneous blanks
set +f
if [ "$line" != "${line#pid_file }" ]; then
PIDFILE=${line#pid_file }
fi
done < $CONFFILE
verify_superuser() {
action=$1
[ $EUID -eq 0 ] && return
log_failure_msg "Superuser privileges required for the" \
"\"$action\" action."
exit 4
}
start() {
log_daemon_msg "Starting Munin-Node3"
mkdir -p /opt/munin3/var/run /opt/munin3/log
chown munin:root /opt/munin3/var/run
chown munin:www-data /opt/munin3/log
chmod 0755 /opt/munin3/var/run
chmod 0755 /opt/munin3/log
if pidofproc -p $PIDFILE $DAEMON >/dev/null; then
log_progress_msg "started beforehand"
log_end_msg 0
exit 0
fi
start_daemon -p $PIDFILE $DAEMON $DAEMON_ARGS
# /usr/sbin/munin-node3 –config /etc/munin3/munin-node.conf
ret=$?
# start_daemon() isn't thorough enough, ensure the daemon has been
# started manually
attempts=0
until pidofproc -p $PIDFILE $DAEMON >/dev/null; do
attempts=$(( $attempts + 1 ))
sleep 0.05
[ $attempts -lt 20 ] && continue
log_end_msg 1
return 1
done
[ $ret -eq 0 ] && log_progress_msg "done"
log_end_msg $ret
return $ret
}
stop() {
log_daemon_msg "Stopping Munin-Node3"
# killproc() doesn't try hard enough if the pid file is missing,
# so create it is gone and the daemon is still running
if [ ! -r $PIDFILE ]; then
pid=$(pidofproc -p $PIDFILE $DAEMON)
if [ -z "$pid" ]; then
log_progress_msg "stopped beforehand"
log_end_msg 0
return 0
fi
echo $pid 2>/dev/null > $PIDFILE
if [ $? -ne 0 ]; then
log_end_msg 1
return 1
fi
fi
killproc -p $PIDFILE /opt/munin3/sbin/munin-node
ret=$?
# killproc() isn't thorough enough, ensure the daemon has been
# stopped manually
attempts=0
until ! pidofproc -p $PIDFILE $DAEMON >/dev/null; do
attempts=$(( $attempts + 1 ))
sleep 0.05
[ $attempts -lt 20 ] && continue
log_end_msg 1
return 1
done
[ $ret -eq 0 ] && log_progress_msg "done"
log_end_msg $ret
return $ret
}
if [ “$#” -ne 1 ]; then
log_failure_msg "Usage: /etc/init.d/munin-node3" \
"{start|stop|restart|force-reload|try-restart}"
exit 2
fi
case “$1” in
start)
verify_superuser $1
start
exit $?
;;
stop)
verify_superuser $1
stop
exit $?
;;
restart|force-reload)
verify_superuser $1
stop || exit $?
start
exit $?
;;
try-restart)
verify_superuser $1
pidofproc -p $PIDFILE $DAEMON >/dev/null
if [ $? -eq 0 ]; then
stop || exit $?
start
exit $?
fi
log_success_msg "Munin-Node3 was stopped beforehand and thus not" \
"restarted."
exit 0
;;
reload)
log_failure_msg "The \"reload\" action is not implemented."
exit 3
;;
status)
pid=$(pidofproc -p $PIDFILE $DAEMON)
ret=$?
pid=${pid% } # pidofproc() supplies a trailing space, strip it
if [ $ret -eq 0 ]; then
log_success_msg "Munin-Node3 is running (PID: $pid)"
exit 0
# the LSB specifies that I in this case (daemon dead + pid file exists)
# should return 1, however lsb-base returned 2 in this case up to and
# including version 3.1-10 (cf. #381684). Since that bug is present
# in Sarge, Ubuntu Dapper, and (at the time of writing) Ubuntu Etch,
# and taking into account that later versions of pidofproc() do not
# under any circumstance return 2, I'll keep understanding invalid
# return code for the time being, even though the LSB specifies it is
# to be used for the situation where the "program is dead and /var/lock
# lock file exists".
elif [ $ret -eq 1 ] || [ $ret -eq 2 ]; then
log_failure_msg "Munin-Node3 is dead, although $PIDFILE exists."
exit 1
elif [ $ret -eq 3 ]; then
log_warning_msg "Munin-Node3 is not running."
exit 3
fi
log_warning_msg "Munin-Node3 status unknown."
exit 4
;;
*)
log_failure_msg "Usage: /etc/init.d/munin-node3" \
"{start|stop|restart|force-reload|try-restart}"
exit 2
;;
esac
log_failure_msg “Unexpected failure, please file a bug.” exit 1
munin-async3 script de inicio en Debian
#! /bin/sh ### BEGIN INIT INFO # Provides: munin-async3 # Required-Start: $network $named $local_fs $remote_fs munin-node # Required-Stop: $network $named $local_fs $remote_fs munin-node # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: Munin asynchronous server 3 # Description: Asynchronous munin node 3 ### END INIT INFO
# Author: Jorne Kandziora jorne@quarantainenet.nl # # Do NOT “set -e”
# PATH should only include /usr/* if it runs after the mountnfs.sh script PATH=/sbin:/usr/sbin:/bin:/usr/bin DESC=“Munin asynchronous server 3” NAME=munin-asyncd DAEMON=/opt/munin3/lib/$NAME DAEMON_ARGS=“” DAEMON_USER=“munin-async3” PIDFILE=/opt/munin3/var/run/$NAME.pid SCRIPTNAME=/etc/init.d/$NAME
# Exit if the package is not installed [ -x “$DAEMON” ] || exit 0
# Read configuration variable file if it is present [ -r /etc/default/$NAME ] && . /etc/default/$NAME
# Load the VERBOSE setting and other rcS variables . /lib/init/vars.sh
# Define LSB log_* functions. # Depend on lsb-base (>= 3.0-6) to ensure that this file is present. . /lib/lsb/init-functions
# # Function that starts the daemon/service # do_start() {
# Return
# 0 if daemon has been started
# 1 if daemon was already running
# 2 if daemon could not be started
start-stop-daemon --start --background --make-pidfile --pidfile $PIDFILE --chuid $DAEMON_USER --exec $DAEMON --test > /dev/null \
|| return 1
start-stop-daemon --start --background --make-pidfile --pidfile $PIDFILE --chuid $DAEMON_USER --exec $DAEMON -- \
$DAEMON_ARGS \
|| return 2
}
# # Function that stops the daemon/service # do_stop() {
# killproc() doesn't try hard enough if the pid file is missing,
# so create it is gone and the daemon is still running
if [ ! -r $PIDFILE ]; then
pid=$(pidofproc -p $PIDFILE $DAEMON)
if [ -z "$pid" ]; then
[ "$VERBOSE" != no ] && log_progress_msg "stopped beforehand"
log_end_msg 0
return 0
fi
echo $pid 2>/dev/null > $PIDFILE
if [ $? -ne 0 ]; then
log_end_msg 1
return 1
fi
fi
killproc -p $PIDFILE $DAEMON
ret=$?
# killproc() isn't thorough enough, ensure the daemon has been
# stopped manually
attempts=0
until ! pidofproc -p $PIDFILE $DAEMON >/dev/null; do
attempts=$(( $attempts + 1 ))
sleep 0.05
[ $attempts -lt 20 ] && continue
log_end_msg 1
return 1
done
[ $ret -eq 0 ] && [ "$VERBOSE" != no ] && log_progress_msg "done"
log_end_msg $ret
return $ret
}
# # Function that sends a SIGHUP to the daemon/service # do_reload() {
#
# If the daemon can reload its configuration without
# restarting (for example, when it is sent a SIGHUP),
# then implement that here.
#
start-stop-daemon --stop --signal 1 --background --make-pidfile --quiet --pidfile $PIDFILE --exec $DAEMON
return 0
}
case “$1” in
start)
[ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME"
do_start
case "$?" in
0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
esac
;;
stop)
[ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME"
do_stop
case "$?" in
0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
esac
;;
#reload|force-reload)
#
# If do_reload() is not implemented then leave this commented out
# and leave 'force-reload' as an alias for 'restart'.
#
#log_daemon_msg "Reloading $DESC" "$NAME"
#do_reload
#log_end_msg $?
#;;
restart|force-reload)
#
# If the "reload" option is implemented then remove the
# 'force-reload' alias
#
log_daemon_msg "Restarting $DESC" "$NAME"
do_stop
case "$?" in
0|1)
do_start
case "$?" in
0) log_end_msg 0 ;;
1) log_end_msg 1 ;; # Old process is still running
*) log_end_msg 1 ;; # Failed to start
esac
;;
*)
# Failed to stop
log_end_msg 1
;;
esac
;;
status)
pid=$(pidofproc -p $PIDFILE $DAEMON)
ret=$?
pid=${pid% } # pidofproc() supplies a trailing space, strip it
if [ $ret -eq 0 ]; then
log_success_msg "Munin-Async3 is running (PID: $pid)"
exit 0
# the LSB specifies that I in this case (daemon dead + pid file exists)
# should return 1, however lsb-base returned 2 in this case up to and
# including version 3.1-10 (cf. #381684). Since that bug is present
# in Sarge, Ubuntu Dapper, and (at the time of writing) Ubuntu Etch,
# and taking into account that later versions of pidofproc() do not
# under any circumstance return 2, I'll keep understanding invalid
# return code for the time being, even though the LSB specifies it is
# to be used for the situation where the "program is dead and /var/lock
# lock file exists".
elif [ $ret -eq 1 ] || [ $ret -eq 2 ]; then
log_failure_msg "Munin-Async3 is dead, although $PIDFILE exists."
exit 1
elif [ $ret -eq 3 ]; then
log_warning_msg "Munin-Async3 is not running."
exit 3
fi
log_warning_msg "Munin-Async3 status unknown."
exit 4
;;
*)
#echo "Usage: $SCRIPTNAME {start|stop|restart|reload|force-reload}" >&2
echo "Usage: $SCRIPTNAME {start|stop|restart|force-reload}" >&2
exit 3
;;
esac
kvd_dispon_multi
#!/bin/bash # # #
rrd_dir=“/var/lib/munin/kvd-estado”
conexion=“mysql -f -B –password=password –user=admin”
comando=“SELECT nombre FROM spt_inventario ORDER BY nombre ASC; ” kioskos=`$conexion -e “$comando” timer | grep -v nombre `
case $1 in
config) # primero el consolidador echo "multigraph kvd_dispon
graph_category KVD-Dispon graph_title Disponibilidad Kioskos graph_args –units-exponent 0 graph_vlabel Cumplimiento“ # for i in $kioskos ; do # echo “$i-Subida.label $i Carga (Upload)” # done
for i in $kioskos ; do
echo "multigraph kvd_dispon.$i
graph_category KVD-Dispon graph_title Disponibilidad Kiosko $i graph_args –units-exponent 0 graph_vlabel Cumplimiento $i-percent.label Porcentaje (20H) $i-percent.draw AREA $i-horas.label Horas $i-horas.draw AREA” done
exit 0
;;
esac
echo “multigraph kvd_dispon” for idkiosk in $kioskos ; do
echo "multigraph kvd_dispon.$idkiosk"
archivo="$rrd_dir/kvd-estado-kvd_estado-$idkiosk-`echo $idkiosk | sed 's/^./_/g'`-g.rrd"
valor=$(rrdtool fetch $archivo AVERAGE \
-s `date --date="last monday" +"%s" ` |\
grep ":" | grep -v '\-nan' | awk '{if($2 > 0) print $0}' | wc -l )
echo -n "$idkiosk-percent.value "
echo $valor/12/20*100 | bc -l | awk '{printf "%.2f\n",$1}'
echo -n "$idkiosk-horas.value "
echo $valor/12 | bc -l | awk '{printf "%.3f\n",$1}'
done