bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Noam Postavsky]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Michael Heerdegen <michael_heerdegen@web.de>
>> Cc: bug-gnu-emacs@gnu.org,  30626@debbugs.gnu.org
>> Date: Tue, 27 Feb 2018 13:08:59 +0100
>> 
>> #+begin_src emacs-lisp
>> (seq-doseq (_ (stream-range 1 1000000)) nil)
>> #+end_src
>> 
>> Note that this is executed as a loop due how to streams are implemented,
>> although the definition of `seq-doseq' looks recursive.

Doesn't look recursive to me, it expands to a call to seq-do, which uses
a simple loop.

>> But it seems that gc has a problem with the large number of conses
>> created when processing that.
>
> What can we do instead in such cases?  Stack-overflow protection
> cannot work in GC, so you are shooting yourself in the foot by
> creating such large recursive structures.  By the time we get to GC,
> where the problem will happen, it's too late, because the memory was
> already allocated.
>
> Does anyone has a reasonable idea for avoiding the crash in such
> programs?

I don't have a quick answer for the general case, but I think it's a bug
in stream.el that it's creating such large structures in the first
place.  As far as I understand it, the point of streams is to handle
long lists by encoding them as

    (FIRST-VALUE . FUNCTION-TO-PRODUCE-REST-OF-LIST)
 
so as to avoid allocating large amounts of memory.  Is there an easy way
to find out what the large structures are, and where they are coming
from?