Re: [bug-gawk] Memory leak

[Stephane Delsert]

Hi Andrew,

I added your lines in the code and reran the tests with 1MM and 2MM. I've 
always a memory leak : 1.5 Gb for 84MM of records processed . 

I think that you are right about the relationship of the issue and the user 
defined function mechanism. that this issue is related to the user defined 
function mechanism. I didn't know the valgrind tool  but what can I see looking 
at the two reports is that the problem comes from the allocation of 
Op_func_call in the assoc_list function of the array.c file. 

We have a bcalloc call with a size. Looking at the code we have automatically a 
call to malloc if the size is grower than 1.  
The call of the bcfree function doesn't free the memory but set a pool free 
list probably to avoid system calls. The leak of memory comes from this.
I don't know the architecture of this program and the management of its data 
structure so It's difficult for me to find a quick solution but I will try.

Best Regards,

Stéphane. 


-----Original Message-----
From: Andrew J. Schorr [mailto:address@hidden 
Sent: jeudi 30 mars 2017 18:48
To: Stephane Delsert <address@hidden>
Cc: address@hidden; address@hidden
Subject: Re: [bug-gawk] Memory leak

Hi Stéphane,

I'm surprised, since that's the only leak I can see in the valgrind output.
If the debug messages about the NODE and BUCKET counts still show very small 
numbers, then I just don't see what else could be leaking. Do you have any 
valgrind output that shows this leak?

There are basically 2 possibilities for why the memory footprint would grow:

1. If you're building a huge array in memory, then that consumes lots of NODEs 
and BUCKETs, and it takes a lot of space. But you say that you're not storing a 
large array, and the debug messages show that very few NODEs and BUCKETs are 
being allocated, unless your 60MM record dataset is much different than your 
smaller 1MM and 2MM datasets. Is that one building a huge in-memory array?

2. If you're not building a large array in memory, then there would have to be 
a leak of the type I fixed in that patch. I don't see any other leaks in the 
valgrind output you sent.

Regards,
Andy

P.S. I am attaching a more elegant version of the patch, but it should give the 
exact same results. Arnold -- what do you think of this issue? It seems like a 
memory leak to me.

On Thu, Mar 30, 2017 at 04:26:20PM +0000, Stephane Delsert wrote:
> Thanks Andrew,
> 
> 
> 
> I did the modification but  that seems that doesn't work :  I reach 
> 1gb of memory  for almost 60 MM of records and that still growth
> 
> 
> 
> Best regards,
> 
> 
> 
> Stéphane.
> 
> 
> 
> 
> 
> -----Original Message-----
> From: Andrew J. Schorr [mailto:address@hidden
> Sent: jeudi 30 mars 2017 17:20
> To: Stephane Delsert <address@hidden>
> Cc: address@hidden; address@hidden
> Subject: Re: [bug-gawk] Memory leak
> 
> 
> 
> Hi,
> 
> 
> 
> > ==19925== 11,301,792 bytes in 117,727 blocks are still reachable in 
> > loss record 124 of 124
> 
> > ==19925==    at 0x4A0717A: malloc (vg_replace_malloc.c:298)
> 
> > ==19925==    by 0x480FDE: emalloc_real (awk.h:1820)
> 
> > ==19925==    by 0x482599: bcalloc (symbol.c:722)
> 
> > ==19925==    by 0x40BBDA: assoc_list (array.c:1329)
> 
> > ==19925==    by 0x4477C9: r_interpret (interpret.h:900)
> 
> > ==19925==    by 0x4605F8: main (main.c:518)
> 
> 
> 
> This is what worries me. It does seem to correspond to a leak related to 
> using a user-defined sorting function. Looking at the code, I don't see why 
> it's not getting freed, unless this test is bailing out early before we free 
> the allocated instruction below:
> 
> 
> 
>         if (list == NULL || ! cmp_func || (assoc_kind & (AASC|ADESC)) 
> != 0)
> 
>                                 return list;    /* empty list or unsorted, or 
> list already sorted */
> 
> 
> 
> We know that cmp_func is not NULL, but perhaps list is NULL or the assoc_kind 
> flags were modified by symbol->alist. If this test fails, then we return 
> before we pop the instruction and free it, so it's still on the stack and 
> reachable...
> 
> 
> 
> Does the attached, not well-tested, patch fix your memory leak?
> 
> 
> 
> Regards,
> 
> Andy
> **********************************************************************
> ***** The information contained in this communication is confidential, 
> is intended only for the use of the recipient named above, and may be 
> legally privileged.
> 
> If the reader of this message is not the intended recipient, you are 
> hereby notified that any dissemination, distribution or copying of 
> this communication is strictly prohibited.
> 
> If you have received this communication in error, please resend this 
> communication to the sender and delete the original message or any 
> copy of it from your computer system.
> 
> Thank You.
> **********************************************************************
> ******

-- 
Andrew Schorr                      e-mail: address@hidden
Telemetry Investments, L.L.C.      phone:  917-305-1748
545 Fifth Ave, Suite 1108          fax:    212-425-5550
New York, NY 10017-3630

REPORT_2MM_V4.txt
Description: REPORT_2MM_V4.txt

REPORT_1MM_V4.txt
Description: REPORT_1MM_V4.txt