Re: watchmalloc (Re: Corrupted messages...)

[Dan Nelson]

In the last episode (Jun 10), Chuck Yerkes said:
> Happened again last night (sendmail's commercial MTA), so that points
> to the milter.
> 
> Quoting Dan Nelson (address@hidden):
> > Can you also try running spamass-milter with the environment variable
> > LD_PRELOAD=watchmalloc.so.1 ?  You don't even have to set MALLOC_DEBUG;
> > simply loading the shared library will cause all malloc and free calls
> > to fill their memory blocks with a constant value (0xdeadbeef for
> > malloc and 0xbaddcafe for free).  If something is really reusing free'd
> > memory, your header values should be replaced with repeated sets of
> > "????????????" (possibly byte-swapped :)
> 
> Oh my, this IS slow...  (watchmalloc and MALLOC_DEBUG).

Yes, the docs say MALLOC_DEBUG can cause from a 10 to 1000 times
slowdown :)  Since it actually protects free'd memory at a hardware
level, though, once the bug triggers itself you should get a coredump
that points right at the problem (make sure you've got your resource
limits set to allow cores to be created)
 
> 1) I'm really glad I changed the syslog facility to LOCAL0
> 2) I'm happy to have an extra partition there for that log.
> 3) many timeouts (even at 4minutes). 

In another message:
> Is there a -d level you'd like to see?
> (realized I was using "-d 3" and THAT was filling the
> logs).

Most of the logging flags deal with milter-spamc interactions, and I
don't think that's where the problem is.  None of the flags are very
useful in a high-throughput setting, either, since you may potentially
have multiple threads running at the same time, and there's no
message-id tag in the logs.  I think I can add the queue ID to each log
line to help that.

-- 
        Dan Nelson
        address@hidden