Re: GUILE_MAX_HEAP_SIZE

[Han-Wen Nienhuys]

Ludovic Courtès escreveu:
> Hi Han-Wen,
> 
> Han-Wen Nienhuys <address@hidden> writes:
> 
>> Ludovic Courtès escreveu:
> 
>>> is kind of hard to review in a glimpse.  Does it just randomly "clean
>>> things up" (whatever that means---it does not follow the GCS, for
>> GCS?
> 
> "GNU Coding Standards", the thing we're supposed to adhere to when
> writing code for the GNU Project:

I've pretty religiously followed those for LilyPond although google's 
coding style now has me remove spaces before ( in function calls. 

The existing code was not following GCS by any standard (which was my
fault 7 years ago or so.)

Can you be more specific about this?

>>> instance), or does it fix anything?  It's hard to tell.  Can you
>>> reproduce the heap usage graphs referred earlier in this thread?  Do
>> No, the memory usage is more stable now. 
> 
> Can you show what the graph looks like, for comparison purposes?

See below - note that the old .scm file was pretty much broken, as it 
was using gc-live-object-stats which is only accurate just after the
mark phase.

> Which assertion is it that failed?  Was that due to an old
> `libguile-i18n.so' being loaded?

yes - the old version had a scm_cells_allocated++ inlined, which was
included in the .so file. Also, the 
old version was handling odd cases in scm_i_sweep_card differently. 

>> If you think you need to roll back this change, please revoke my 
>> commit privilege and sort things out yourself.
> 
> I tried and failed, and so did Kevin
> (http://thread.gmane.org/gmane.lisp.guile.devel/6699/focus=6832).  AIUI,
> both Kevin and I tried to identify the root of the problem (the "bug")
> in a way that would allow us to fix the offending code as conservatively
> as possible.

being conservative is a good basic attitude, but I just saw general brokenness
all around.

> Conversely, the size and scope of your patch leaves me the impression
> that you rewrote parts of the GC, without actually pinpointing what
> was/is wrong with the code.  I'd have been much more confident with a
> one-liner along with an explanation and sample program to determine
> whether the problem is there.

The problem is that the previous state of the GC was very much confused
with contradicting definitions within the code of what was being kept as 
statistics.  This is problematic since the allocation strategy (should 
I allocate more memory?) was based on these confused statistics.

This change simplifies and corrects this status.  It does 
not introduce more clever behavior; it just trims any excess intelligence
and confusion from the code.

The thread you quote remarks something odd
about live-cell-heap, which is not that surprising.
The gc-live-object-statistics show the state of the last GC round rather
than the current state. - so it tends to jump around at unpredictable times

Attached is a plot of alive/total, where you can see that it fluctuates between
0.6 and 0.95 - consistent with the 40% yield percentage. Note that summing 
segments
is misleading: the segments are memory addresses, while the rest uses cells for 
measuring memory allocations. I've included the .scm  (slightly revised because 
I
trimmed the gc-stats output as well).

>> The garbage collector isn't that complicated after all.
> 
> Then the people, including me, who spent large amounts of time trying in
> vain to fix the code must have been dumb.

Hopefully, with this change, things will become less confused. 

It's still somewhat kludgy - especially the way that gc-stats is 
constructed is asking for trouble.  We should really have a scm_t_gc_stats
struct and use nice OO patterns for dealing with that.

-- 
 Han-Wen Nienhuys - address@hidden - http://www.xs4all.nl/~hanwen

(use-modules (srfi srfi-1)
             (srfi srfi-14))

(define (total-cell-heap)
  (assoc-ref (gc-stats) 'cell-heap-size))

(define (get-yields)
  (list 0 0))
  

(setvbuf (current-output-port) _IOLBF)
(format #t "### plot '-' using 1:($2/$3) with lines title \"total/alive\"~%")


# plot 'out' using 1:($2/$3) with lines title "alive/total" 

(let loop ((iteration 0))
  (if (> iteration 1000)
      #t
      (begin
        ; 100k cells, 
        (let ((lst (list-tabulate 1000
                                  (lambda (i)
                                    (char-set #\.)  ;; double-cell
                                    (make-list 100)))))

          (if (and (= (modulo iteration 1000) 0)
                   (> iteration 0))
              ;; sporadic heap-intensive job
              ; 1M cells.
              (make-list 1000000)))

        (if (= 0 (modulo iteration 10))
            (let* ((total (total-cell-heap))
                   (alive (assoc-ref (gc-stats)  'cells-allocated)))
              (format #t "~a ~a ~a~%" iteration
                      alive total )))

        (loop (1+ iteration)))))