bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Michael Heerdegen]

Noam Postavsky <npostavs@gmail.com> writes:

> Aha, but the following also crashes, whether compiled or not:
>
> ;; -*- lexical-binding: t -*-
>
> (require 'stream)
>
> (let* ((stream0 (stream-range 1 1000000))
>        (stream stream0))
>   (while (not (stream-empty-p stream))
>     (cl-callf stream-rest stream)))

I guess you should have some awe if you write something like
(stream-range 1 1000000) and keep in mind what happens with this
gigantic thing.  Though, could the object `stream0' references not
already be garbage-collected when the loop has been entered, since there
is no lexical reference to that variable there?

BTW, it gets even worse if you append streams, since the original
streams are not copied and magically unfolded when you generate elements
from the concatenation.

> Now, I can see that letting stream0 automagically get access to the
> unfolded result can be an optimization in some cases, although in this
> case it's a pessimization.  It could also affect the semantics if
> unfolding the stream has side effects, not sure if stream.el makes
> guarantees about that though.

AFAICT we make no such guarantees at all.  When generating stream
elements has side effects (which is not ideal, but it's not forbidden),
then you must know what you are doing.  In my experience, side effects
often directly correlate with element generation - e.g. for a stream of
search matches, a side effect is to set a variable to the position where
to continue searching.  These kind of side effects are not problematic.

For non-trivial side-effects, dunno, never wanted something like that.
But this problem also concerns other forms of delayed evaluation,
including thunks, and generally everything you can do with closures.


Michael.