monit-general
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: receiving memory corruption when trying to run monit


From: Martin Pala
Subject: Re: receiving memory corruption when trying to run monit
Date: Thu, 11 Jan 2007 17:21:18 +0100
User-agent: Thunderbird 1.5.0.9 (X11/20061222)

... problem fixed in cvs.

The problem is in byacc banner vs. autoconf config.h. Autoconf sets the appropriate symbols for large files support in config.h, which is included as first thing in the parser (p.y) prologue section.

byacc however prepends its banner to the prologue, where it includes the stdlib.h. Because of that, the _FILE_OFFSET_BITS wasn't set appropriately and off_t defaulted to 4 bytes during byacc generated parser source (y.tab.c) compilation. The rest of the code was compiled with off_t set to 8 bytes ... this lead to the crash.

This issue didn't occured when:

- either monit was compiled with 64-bit support

- or recommended bison was used as parser generator instead of byacc

- or largefiles support was disabled (LFS support was added in monit 4.8.2, so earlier versions shouldn't be affected)


Martin


Adam Greene wrote:
interesting!  good to know; I'll give it a shot.

thanks martin!!
Adam


On Jan 10, 2007, at 3:14 PM, Martin Pala wrote:

I've replicated the problem. It is caused when compiled with byacc (your case) parser instead of bison (my case). I have never tested monit compiled with byacc - maybe it is not compatible with it. I will look on it in more detail tomorrow.

You can fix it by installing bison ('yum install bison') and recompile monit (make clean && configure && make).

Thanks,
Martin



Martin Pala wrote:
Thanks :)
I see that your monit is stripped ... can you compile it again, but don't use 'make install' this time but copy monit to /usr/local/bin/monit by hand this time. It will allow to resolve the symbols in the corefile to the functions in monit source (replace these '??' in the gdb output).
I will try to reproduce in the meantime ...
Cheers,
Martin
Adam Greene wrote:
hi martin,
I've added the config and gdb info in attachments.

I'm running the monit that I installed; but thanks for double checking!!
-bash-3.00# which monit
/usr/local/bin/monit
-bash-3.00# monit
Segmentation fault (core dumped)
-bash-3.00# /usr/local/bin/monit
Segmentation fault (core dumped)
-bash-3.00# /usr/local/bin/monit -vIc ./monitrc
monit: Debug: Adding credentials for user 'monit'.
*** glibc detected *** malloc(): memory corruption: 0x0993f4a8 ***
Aborted (core dumped)

looking at the bits of info below, and comparing it to yours, I notice:
* I'm not running x86_64
* the uname shows an athlon, but the file shows it to be an Intel 80386 executable (not sure if that is a huge deal?) * I'm running Linux 2.2.5, not Linux 2.4.0 like you are...but we are both CentOS release 4.4 (Final) ?

thanks martin!
Adam

-bash-3.00# uname -a
Linux agreene.railsmachina.com 2.6.9-42.0.2.EL.xs148xenU #1 SMP Sun Oct 15 08:31:21 EDT 2006 i686 athlon i386 GNU/Linux -bash-3.00# file /usr/local/bin/monit /usr/local/bin/monit: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.5, dynamically linked (uses shared libs), stripped

-bash-3.00# cat /etc/redhat-release
CentOS release 4.4 (Final)

-bash-3.00# ldd /usr/local/bin/monit
        libpthread.so.0 => /lib/i686/libpthread.so.0 (0x00bf4000)
        libcrypt.so.1 => /lib/libcrypt.so.1 (0x00ba6000)
        libresolv.so.2 => /lib/libresolv.so.2 (0x0045c000)
        libnsl.so.1 => /lib/libnsl.so.1 (0x006cf000)
        libssl.so.4 => /lib/libssl.so.4 (0x001d7000)
        libcrypto.so.4 => /lib/libcrypto.so.4 (0x0020b000)
        libc.so.6 => /lib/i686/libc.so.6 (0x002f4000)
        /lib/ld-linux.so.2 (0x007a0000)
libgssapi_krb5.so.2 => /usr/lib/libgssapi_krb5.so.2 (0x0052d000)
        libkrb5.so.3 => /usr/lib/libkrb5.so.3 (0x00d26000)
        libcom_err.so.2 => /lib/libcom_err.so.2 (0x00638000)
        libk5crypto.so.3 => /usr/lib/libk5crypto.so.3 (0x00fca000)
        libdl.so.2 => /lib/libdl.so.2 (0x00708000)
        libz.so.1 => /usr/lib/libz.so.1 (0x00111000)





On Jan 10, 2007, at 9:01 AM, Martin Pala wrote:

Hi,

can you try to get the core?

1.) ulimit -c unlimited
2.) monit -vI

In the case that monit will crash, it should provide us more infortions.

3.) backtrace (for example using gdb):

gdb /path/to/monit/binary /path/to/monit/core
> info threads
> backtrace



Which configuration did you use for compilation?

I'm not able to replicate the problem anymore - are you sure you are calling the fixed monit binary? (since you have not specified the path, it may run maybe different monit binary which is in the PATH)

My environment:

address@hidden monit.d]# uname -a
Linux sh01ug01.oskarmobil.cz 2.6.9-42.0.3.EL #1 Fri Oct 6 06:02:37 CDT 2006 x86_64 x86_64 x86_64 GNU/Linux

address@hidden monit.d]# cat /etc/redhat-release
CentOS release 4.4 (Final)

configure output (abbreviated):
--8<--
monit has been configured with the following options:
                Architecture: LINUX
                 SSL support: enabled
       SSL include directory: /usr/include
       SSL library directory: /usr/lib64
         resource monitoring: enabled
               resource code: sysdep_LINUX.c
         large files support: enabled
Compiler flags: -g -O2 -Wall -D _REENTRANT -I/usr/include Linker flags: -lpthread -lcrypt -lresolv -lnsl -L/usr/lib64 -lssl -lcrypto
           pid file location: /var/run
--8<--

address@hidden monit.d]# file /root/cvs/monit/monit
/root/cvs/monit/monit: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), for GNU/Linux 2.4.0, dynamically linked (uses shared libs), not stripped

address@hidden monit.d]# ldd /root/cvs/monit/monit
libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x0000003b38f00000)
        libcrypt.so.1 => /lib64/libcrypt.so.1 (0x0000003b3af00000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x0000003b39300000)
        libnsl.so.1 => /lib64/libnsl.so.1 (0x0000003b3ad00000)
        libssl.so.4 => /lib64/libssl.so.4 (0x00000031e3600000)
        libcrypto.so.4 => /lib64/libcrypto.so.4 (0x00000031e3300000)
        libc.so.6 => /lib64/tls/libc.so.6 (0x0000003b38000000)
        /lib64/ld-linux-x86-64.so.2 (0x0000003b37e00000)
libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 (0x0000003b3e600000)
        libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x0000003b3d200000)
        libcom_err.so.2 => /lib64/libcom_err.so.2 (0x0000003b3d000000)
libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x0000003b3d400000)
        libdl.so.2 => /lib64/libdl.so.2 (0x0000003b38300000)
        libz.so.1 => /usr/lib64/libz.so.1 (0x0000003b39100000)


address@hidden monit.d]# pwd
/etc/monit.d


address@hidden monit.d]# cat ./monitrc
# Monit control file
# ==================
set daemon     120
#set logfile    syslog
set logfile syslog facility log_daemon
set mailserver 127.0.0.1
set alert      xxx
set httpd port 8135 allow monit:monit

check process cron with pidfile /var/run/crond.pid
    start program = "/etc/init.d/crond start"
    start program = "/etc/init.d/crond start"
    stop  program = "/etc/init.d/crond stop"
    group system
    depends cron_init, cron_bin

check file cron_init with path /etc/init.d/crond
    group system

check file cron_bin with path /usr/sbin/crond
    group system



address@hidden monit.d]# /root/cvs/monit/monit -vIc ./a
monit: Cannot stat the control file './a' -- No such file or directory
address@hidden monit.d]# /root/cvs/monit/monit -vIc ./monitrc
monit: Debug: Adding credentials for user 'monit'.
Runtime constants:
 Control file       = ./monitrc
 Log file           = syslog
 Pid file           = /var/run/monit.pid
 Debug              = True
 Log                = True
 Use syslog         = True
 Is Daemon          = True
 Use process engine = True
 Poll time          = 120 seconds
 Mail server(s)     = 127.0.0.1
 Mail from          = (not defined)
 Mail subject       = (not defined)
 Mail message       = (not defined)
 Start monit httpd  = True
 httpd bind address = Any/All
 httpd portnumber   = 8135
 httpd signature    = True
 Use ssl encryption = False
 httpd auth. style  = Basic Authentication
 Alert mail to      = xxx
   Alert on         = All events

The service list contains the following entries:

Process Name          = cron
 Group                = system
 Pid file             = /var/run/crond.pid
 Monitoring mode      = active
 Start program        = '/etc/init.d/crond start' timeout 1 cycle(s)
 Stop program         = '/etc/init.d/crond stop' timeout 1 cycle(s)
 Depends on Service   = cron_bin
 Depends on Service   = cron_init
Pid = if changed 1 times within 1 cycle(s) then alert Ppid = if changed 1 times within 1 cycle(s) then alert

File Name             = cron_init
 Group                = system
 Path                 = /etc/init.d/crond
 Monitoring mode      = active

File Name             = cron_bin
 Group                = system
 Path                 = /usr/sbin/crond
 Monitoring mode      = active

System Name           = xxx
 Monitoring mode      = active

-------------------------------------------------------------------------------
monit: pidfile '/var/run/monit.pid' does not exist
Starting monit daemon with http interface at [*:8135]
Starting monit HTTP server at [*:8135]
monit HTTP server started
Monit started
Monit instance changed notification is sent to address@hidden
'cron_init' file existence check passed
'cron_init' is regular file
'cron_bin' file existence check passed
'cron_bin' is regular file
'cron' zombie check passed [status_flag=0000]
Shutting down monit HTTP server
monit HTTP server stopped
monit daemon with pid [21982] killed
Monit stopped
Monit instance changed notification is sent to address@hidden





Adam Greene wrote:
hey martin,
I feel a bit silly... I can't figure out how to send an email back to the mailing list and have it show up in the correct spot (ie, as a response to your email). I had the digest option turned on; I've turned that off so in the future I should be able to respond to individual emails with ease.
I tried the cvs version (recompiled, etc), but no luck.
-bash-3.00# pwd
/etc/monit.d
-bash-3.00# ls
includes  monitrc
-bash-3.00# monit
Segmentation fault
the config file with the cron check is pasted below. If I take out all checks, it works w/o a hitch, but it doesn't do anything ;)
How can I help debug this with you guys?
Thanks!
Adam
# Monit control file
# ==================
set daemon     120
#set logfile    syslog
set logfile syslog facility log_daemon
set mailserver 127.0.0.1
set alert      xxx
set httpd port 8135 allow monit:monit
check process cron with pidfile /var/run/crond.pid
    start program = "/etc/init.d/crond start"
    start program = "/etc/init.d/crond start"
    stop  program = "/etc/init.d/crond stop"
    group system
    depends cron_init, cron_bin
check file cron_init with path /etc/init.d/crond
    group system
check file cron_bin with path /usr/sbin/crond
    group system







reply via email to

[Prev in Thread] Current Thread [Next in Thread]