Custom log file

[Marco Ermini]

This is the patch to apply to allows Monit to create a custom log file
to be used by other monitoring applications, like HP OpenView,
logfilter, or Key Performance Indicators software. It should apply on
the CVS version of Monit. It should work, but testers are very
welcomed.

Just a brief intriduction. Don't be misleaded by the fact that it is
called "customlog" - this is just the first name I could recall :-)

This has in fact nothing to do with the Monit's log. This is a tool
written with the purpose to be parsed by external program. Did you
ever ask yourself, why Apache has two log files? (in fact, in recent
versions it has more than these, it added a forensic log for forensic
analisys). It does not means that Apache reports clean accesses in the
"access" log and errors in the "error" log; instead, the Apache
"error" log file reports Apache systems' events (starts and stops,
SIGHUP signals, system problems etc.), while the access log reports
"normal" events (not just http 200 codes, but also http 3xx, 4xx and
5xx.). While you can (and you should) personalize "access" log, you
can't personalize the "error" (the "no listening sockets available,
shutting down" message will always remain the same: it has NO SENSE to
customize it, it MUST be human-reviewed). They have different
purposes: the error log is for Unix system administrators, while the
access log is for application administrator, Key Performance
Indicators, customer experience access analisys etc. I can assure, in
large environment and server farms are reviewed by totally different
people.

This is more or less the same with my "custom log" patch. While the
Monit log should report Monit system issues and be reviewed by Unix
system administrators (and, maybe, you should not be able to customize
it too much), my "custom log" (you could call it "event log" or
"application log" or something similar, if you don't like "custom
log") is for integration issues with Key Performance Indicators,
monitoring stations, HP OpenView (the default format is modelled with
the latest purposes in mind) and similar. It is an "applicative" log
which should NOT report things like "SIGHUP recieved - reloading" and
similar... Front Level support should NOT care about Monit reloading
(I don't want to give them another issue, but to _alleviate_ their
jobs), but in Apache and Oracle not responding.

You could also use this code to customize the actual Monit log, but in
my opinion it has a very limited sense. In an enterprise environment,
you don't want to confuse Monit's own problem with monitoring messages
in the same log. You want Unix system administrators do a different
job that KPI people.

---

This is a very brief summary of what it does.

1) It does not affect the actual log file. Code is in place to
customize the actual log too, and it should be quite simply to apply
it. But actually it is a separate file. This is because it logs
_specific_ things (i.e. it does not care about Monit specific errors,
i.e. missing log file, cannot open the log file, assert violations,
etc). Look at it as the difference between Apache's error log and
access log: the Monit's own log is the Apache error log, the customlog
is the access log. The purpose is to have a log which you can parse
cleanly with KPI programs, HP OpenView etc. so no "trash" must go in
it.

2) if you specify the customlog file, you activate it.

set customlogfile /var/log/custommonit.log

from now then every service has its own customlog (if you don't
specify it, the service will carry default values).


3) to specify a format:

set customlog format "{%d/%m/%Y, %T}t, %a, %s, %m, %y(%Y)"
(this is useful for HP OpenView).

The default in the patch is: "{%d/%m/%Y, %T}t, %a, %s, %m, %d"
(but it should be the previous...)

Possible values are:

- %% = the % character
- %a = alarm id
- %d = event description (as from Event_get_description())
- %g = service group name (if any, or "no group" instead)
- %m = custom message specified in the service (see it later)
- %M = event message (i.e. the message which usually appears in the
normal Monit log)
- %n = service name (from the monitrc)
- %s = severity (see later)
- %S = status ("failed" or "passed")
- %t = standard Monit time format
- %y = service type (don't remeber now, please try yourself :-P)
- %Y = type of check ("Device", "Directory", "File", "Process", "Host")

If a character is nor recognized it is simply printed.

A special handling could be done for the date. If you specify the the
"{}t" token, everything which you put between the {} if formatted like
in the strftime() function (exactly the same as it works in Apache,
the same syntax).

The "severity" is a special things which I needed, but may be useful
to others too. You can choose a level of severity for your failing
service between normal, warning, minor, major, critical, fatal. When a
service fails, in the customlog file is printed the severity you
choose; when the service becomes responsive again, the "Normal"
severity is forcefully printed; if you defined a timeout, when the
service timeouts the "Fatal" severity is forcefully printed.

Remember that the customlog follows the same rule of the "normal" log
in which when the service becomes responsive again, no more "Normal"
events are printed (in fact the "Normal" should be used to clear the
event with HP OpenView).

the syntax to specify a customlogger is:

customlogger [string] [ALARMID alarmnumber] [SEVERITY severitystring]

every parameter is optional. Defaults are: "Monit Alarm", "auto", "Warning".

An example of usage:

check host localweb with address 127.0.0.1
   start "/Library/Apache2/bin/apachectl -k start"
   stop  "/Library/Apache2/bin/apachectl -k stop"
   if failed port 80 proto http request "/" then restart
   if 5 restart within 5 cycles then timeout
   customlogger "webserver at port 80" alarmid auto severity major

The alarmid may be "auto" (or zero, is the same as auto) which means
that you let Monit auto-assign an alarm id number. In any case if you
assign the same number to two or more services, it will silently
re-assign a new unique alarm id number.

This should be all... happy hacking. Hope it finds its way in the
Monit project. It is now correctly working for me.


Bye!
-- 
Marco Ermini
http://www.markoer.org
Dubium sapientiae initium. (Descartes)
address@hidden # mount -t life -o ro /dev/dna /genetic/research
<< This message is for the designated recipient only and may contain
privileged or confidential information. If you have received it in
error, please notify the sender immediately and delete the original.
Any other use of the email by you is prohibited. >>

monit.patch.tar.gz
Description: GNU Zip compressed data