Streams: add stream-delay and streams of directory files

[Michael Heerdegen]

Hello,

attached (at the end) is a patch for stream.el adding two things: (1) a
new macro `stream-delay' and (2) a function `stream-of-directory-files'
returning streams of directory files, optionally recursively.
Especially in the recursive case, a stream of directory files is better
to handle than a list: since it's delayed, it just encapsulates a
directory search to do.  No huge list needs to be accumulated; you can
stop when you have found what you have searched for, resume searching
later, etc. - just as with any stream.


Some notes about semantics and implementation:

(1) It turned out that the template of the implementation of `seq-copy'
for streams is useful in other scenarios, too.  I made the template a
macro `stream-delay', fixed the case of an empty stream that the old
`stream-copy' didn't handle correctly, and got that:

--8<---------------cut here---------------start------------->8---
(defmacro stream-delay (expr)
  "Return a new stream to be obtained by evaluating EXPR.
EXPR will be evaluated once when an element of the resulting
stream is requested for the first time, and must return a stream.
EXPR will be evaluated in the lexical environment present when
calling this function."
  (let ((stream (make-symbol "stream")))
    `(stream-make (let ((,stream ,expr))
                    (if (stream-empty-p ,stream)
                        nil
                      (cons (stream-first ,stream)
                            (stream-rest ,stream)))))))
--8<---------------cut here---------------end--------------->8---


So, now

--8<---------------cut here---------------start------------->8---
(cl-defmethod seq-copy ((stream stream))
  "Return a shallow copy of STREAM."
  (stream-delay stream))
--8<---------------cut here---------------end--------------->8---

As a function, it is different from `stream-delay': the argument is
evaluated.  OTOH `stream-delay' accepts an arbitrary expression that is
evaluated delayed (i.e. when an element is requested the firs ttime) and
must return a stream.  I need this for (2), see there.  When the EXPR is
evaluated later, this is done in the lexical environment present when
`stream-delay' was called (this happens automatically, because
`thunk-delay' creates a closure).


An example of what `stream-delay' does, and what not:

--8<---------------cut here---------------start------------->8---
ELISP> (let* ((a 0))
         (setq f (lambda (x) (setq a x)))
         (setq s (stream-delay (stream-range a 10))))
(--stream--..21..)

ELISP> s
(--stream--..21..)

ELISP> (funcall f 5)
5 (#o5, #x5, ?\C-e)

ELISP> (seq-into s 'list)
(5 6 7 8 9)

ELISP> (funcall f 1)
1 (#o1, #x1, ?\C-a)

ELISP> (seq-into s 'list)
(5 6 7 8 9)
--8<---------------cut here---------------end--------------->8---


(2) I chose the signature

 stream-of-directory-files (directory &optional full nosort recurse 
follow-links filter)

where FULL and NOSORT are directly passed to `directory-files', RECURSE
can be boolean or a predicate (of a dir name) to decide into which dirs
to recurse.  FILTER can be a predicate to limit the resulting stream to
files fulfilling it.  It replaces the MATCH argument of
`directory-files'.

A typical call to `stream-of-directory-files' looks like this:

(setq s (stream-of-directory-files
         "~/gnu-emacs" t nil
         (lambda (dir) (not (string= ".git" (file-name-nondirectory dir))))
         nil
         #'file-directory-p))

This sets `s' to a stream of files under "~/gnu-emacs" that will recurse
into all directories not named ".git".  The last (the FILTER) argument
specifies that only directories will be included.

The function call terminates immediately without looking at the file
system (as we expect for streams).  This is exactly the reason for why I
needed something like `stream-delay' - see the implementation.

For testing: to list the elements of s, one can do e.g.:

(seq-doseq (file s) (message "%s" file))


Finally the patch:

diff --git a/stream.el b/stream.el
index 567a9e3..4d61cf1 100644
--- a/stream.el
+++ b/stream.el
@@ -152,7 +152,7 @@ range is infinite."
        (eq (car stream) stream--identifier)))
 
 (defun stream-empty ()
-  "Return an empty stream."
+  "Return a new empty stream."
   (list stream--identifier (thunk-delay nil)))
 
 (defun stream-empty-p (stream)
@@ -317,10 +317,69 @@ kind of nonlocal exit."
        (cons (stream-first stream)
              (seq-filter pred (stream-rest stream)))))))
 
+(defmacro stream-delay (expr)
+  "Return a new stream to be obtained by evaluating EXPR.
+EXPR will be evaluated once when an element of the resulting
+stream is requested for the first time, and must return a stream.
+EXPR will be evaluated in the lexical environment present when
+calling this function."
+  (let ((stream (make-symbol "stream")))
+    `(stream-make (let ((,stream ,expr))
+                    (if (stream-empty-p ,stream)
+                        nil
+                      (cons (stream-first ,stream)
+                            (stream-rest ,stream)))))))
+
 (cl-defmethod seq-copy ((stream stream))
   "Return a shallow copy of STREAM."
-  (stream-cons (stream-first stream)
-               (stream-rest stream)))
+  (stream-delay stream))
+
+(defun stream-of-directory-files-1 (directory &optional nosort recurse 
follow-links)
+  "Helper for `stream-of-directory-files'."
+  (stream-delay
+   (if (file-accessible-directory-p directory)
+       (let (files dirs (reverse-fun (if nosort #'identity #'nreverse)))
+         (dolist (file (directory-files directory t nil nosort))
+           (let ((is-dir (file-directory-p file)))
+             (unless (and is-dir
+                          (member (file-name-nondirectory (directory-file-name 
file))
+                                  '("." "..")))
+               (push file files)
+               (when (and is-dir
+                          (or follow-links (not (file-symlink-p file)))
+                          (if (functionp recurse) (funcall recurse file) 
recurse))
+                 (push file dirs)))))
+         (apply #'stream-append
+                (stream (funcall reverse-fun files))
+                (mapcar
+                 (lambda (dir) (stream-of-directory-files-1 dir nosort recurse 
follow-links))
+                 (funcall reverse-fun dirs))))
+     (stream-empty))))
+
+(defun stream-of-directory-files (directory &optional full nosort recurse 
follow-links filter)
+  "Return a stream of names of files in DIRECTORY.
+Call `directory-files' to list file names in DIRECTORY and make
+the result a stream.  Don't include files named \".\" or \"..\".
+The arguments FULL and NOSORT are directly passed to
+`directory-files'.
+
+Third optional argument RECURSE non-nil means recurse on
+subdirectories.  If RECURSE is a function, it should be a
+predicate accepting one argument, an absolute file name of a
+directory, and return non-nil when the returned stream should
+recurse into that directory.  Any other non-nil value means
+recurse into every readable subdirectory.
+
+Even with recurse non-nil, don't descent into directories by
+following symlinks unless FOLLOW-LINKS is non-nil.
+
+If FILTER is non-nil, it should be a predicate accepting one
+argument, an absolute file name.  It is used to limit the
+resulting stream to the files fulfilling this predicate."
+  (let* ((stream (stream-of-directory-files-1 directory nosort recurse 
follow-links))
+         (filtered-stream (if filter (seq-filter filter stream) stream)))
+    (if full filtered-stream
+      (seq-map (lambda (file) (file-relative-name file directory)) 
filtered-stream))))
 
 (provide 'stream)
 ;;; stream.el ends here
-- 
2.7.0