Re: GC Warning: Repeated allocation of very large block

[Damien Mattei]

i think the warning is not a false true because the program at reached in
10 days almost 3Gb :

 PID UTIL.     PR  NI    VIRT    RES    SHR S  %CPU  %MEM    TEMPS+ COM.


 332467 mattei    20   0 3262260   2,8g   3744 R 215,0  18,4  32277:54
guile

stopping it has released 3 Gb of memory, the problem can belong from the
hash table which is declared (not defined) with one identifier at top-level
but created in a function used many times in a loop so there was many
hash-table creates and the haskeys-values where not deleted when leaving
the function Quine-Mc-Cluskey that uses the hash-table because there is no
documentation and that in SRFI 69 i forget to do that, a simple function in
SRFI like this one could be useful:

;; delete all the elements of a hash table but not the hash table itself
(will be done by garbage collector)

(define (hash-table-clear! ht) (map (lambda (key) (hash-table-delete! ht
key))
  (hash-table-keys ht)))

i have modified and ran the program again, and will see if the message from
garbage collector happens again (in a week of computation...) i will also
check the whole program that seems to expand symbolic expressions a lot
when i have time.

Regards,
Damien

On Mon, Sep 19, 2022 at 7:26 PM Damien Mattei <damien.mattei@gmail.com>
wrote:

> thank you, i think my algorithm is right but can be improved, i have to
> work on it,activate tracing to see where is the big expression list and if
> it is possible to correct that. There is a main loop in code and the
> warning only appear when data is bigger so i do not think of any memory
> leak (only have alittle doubt on an hastable which is global variable and
> reused:
> (when debug-mode (display-nl "Quine-Mc-Cluskey:"))
>  {minterms-ht <- (make-hash-table)}  ;; need to be cleared at each run
> (init-hash-table-with-set-and-value minterms-ht minterms #f)
> this :
> ;; the hash table for minterms, better to be a top-level definition,it's
> nightmare otherwise...
> (declare minterms-ht)
> {minterms-ht <- (make-hash-table)} ;; Scheme+  syntax
> expression create a new hash table with pointer minterms-ht
> this is done at each loop in the for each Cn
> and i should clear it before but there is no instruction to do that in the
> SRFI 69:
> https://srfi.schemers.org/srfi-69/srfi-69.html
> ???
> I will also check the still running memory used by program.
> Damien
>
> On Mon, Sep 19, 2022 at 1:44 PM Olivier Dion <olivier.dion@polymtl.ca>
> wrote:
>
>> On Mon, 19 Sep 2022, Damien Mattei <damien.mattei@gmail.com> wrote:
>> > is this message appearing when a single scheme variable reach a given
>> > size?
>>
>> This message is from the bdwgc and not from Guile itself.  From their
>> documentation:
>> --8<---------------cut here---------------start------------->8---
>> The garbage collector generates warning messages of the form:
>>
>>
>>     Repeated allocation of very large block ...
>>     May lead to memory leak and poor performance
>>
>>
>> when it needs to allocate a block at a location that it knows to be
>> referenced
>> by a false pointer. These false pointers can be either permanent (e.g.
>> a static integer variable that never changes) or temporary. In the latter
>> case, the warning is largely spurious, and the block will eventually
>> be reclaimed normally. In the former case, the program will still run
>> correctly, but the block will never be reclaimed. Unless the block is
>> intended
>> to be permanent, the warning indicates a memory leak.
>>
>>   1. Ignore these warnings while you are using GC_DEBUG. Some of the
>> routines
>>   mentioned below don't have debugging equivalents. (Alternatively, write
>> the
>>   missing routines and send them to me.)
>>   2. Replace allocator calls that request large blocks with calls to
>>   `GC_malloc_ignore_off_page` or `GC_malloc_atomic_ignore_off_page`. You
>> may
>>   want to set a breakpoint in `GC_default_warn_proc` to help you identify
>> such
>>   calls. Make sure that a pointer to somewhere near the beginning of the
>>   resulting block is maintained in a (preferably volatile) variable as
>> long
>>   as the block is needed.
>>   3. If the large blocks are allocated with realloc, we suggest instead
>>   allocating them with something like the following. Note that the realloc
>>   size increment should be fairly large (e.g. a factor of 3/2) for this to
>>   exhibit reasonable performance. But we all know we should do that
>> anyway.
>>
>>
>>         void * big_realloc(void *p, size_t new_size) {
>>             size_t old_size = GC_size(p);
>>             void * result;
>>             if (new_size <= 10000) return(GC_realloc(p, new_size));
>>             if (new_size <= old_size) return(p);
>>             result = GC_malloc_ignore_off_page(new_size);
>>             if (result == 0) return(0);
>>             memcpy(result,p,old_size);
>>             GC_free(p);
>>             return(result);
>>         }
>>
>>
>>   4. In the unlikely case that even relatively small object (<20 KB)
>>   allocations are triggering these warnings, then your address space
>> contains
>>   lots of "bogus pointers", i.e. values that appear to be pointers but
>> aren't.
>>   Usually this can be solved by using `GC_malloc_atomic` or the routines
>>   in `gc_typed.h` to allocate large pointer-free regions of bitmaps, etc.
>>   Sometimes the problem can be solved with trivial changes of encoding
>>   in certain values. It is possible, to identify the source of the bogus
>>   pointers by building the collector with `-DPRINT_BLACK_LIST`, which will
>>   cause it to print the "bogus pointers", along with their location.
>>   5. If you get only a fixed number of these warnings, you are probably
>> only
>>   introducing a bounded leak by ignoring them. If the data structures
>> being
>>   allocated are intended to be permanent, then it is also safe to ignore
>> them.
>>   The warnings can be turned off by calling `GC_set_warn_proc` with
>>   a procedure that ignores these warnings (e.g. by doing absolutely
>>   nothing).
>> --8<---------------cut here---------------end--------------->8---
>>
>> It's my understanding that either A) there's a leak or B) its a false
>> positive.
>>
>> > i wanted to debug and trace because i know this algorithm expand a lot
>> > expressions (perheaps too much) sometimes (but it is an NP-problem and
>> > exponential, so perheaps stop the program is the only solution and run
>> it
>> > on more little data)
>> > i do not think there could be a memory leak in a recursive scheme
>> > program,
>>
>> Gotta be careful with a conservative GC I guess.  Your memory will get
>> reclaim at some point, but it's not guaranteed when since any value on
>> the C stack -- and global variables -- could reference your allocation
>> by accident.  So if by accident there's a static constant value in C
>> that has the value of an allocation by the GC, it will never reclaim it.
>> Since it's the Scheme stack and I suppose it's tail call optimized, that
>> should not impact the GC.
>>
>> The most important thing would be to check the memory usage of the
>> program with a tool like `htop'.  If there's leak, you will see that the
>> memory usage percentage keep increasing.
>>
>> --
>> Olivier Dion
>> oldiob.dev
>>
>