Re: Checking if this is a Eshell bug in emacs 29: Should eshell redirect all output in esh script to stdout

[Milan Zimmermann]

Jim,

Thanks for the explanation.

Well, it looks like I keep running against issues that would likely be gone if https://debbugs.gnu.org/cgi/bugreport.cgi?bug=57635 is addressed. Unfortunately I do not have much confidence in my ability to help ,as it would take me a long time to even get to the point of my Elisp to start attempting. 

On the limited scope of this output issue: Should I create a bug out of this, or do you want to go ahead and report it? (or something else)

Thanks,

Milan

PS: I needed the stdout and stderr redirection to behave properly to be able to finish my Org-babel ob-eshell project I started a few days ago.  There is already an existing ob-eshell, but it does not work for me, so I started a re-do.  (I mostly want :results output to work). I see now that effort is on a backburner for now until the redirection works.

On Wed, Nov 23, 2022 at 3:30 PM Jim Porter <jporterbugs@gmail.com> wrote:

On 11/23/2022 10:56 AM, Milan Zimmermann wrote:
> I would like to check here before reporting an Eshell bug.
>
> A script named 'redirect-echo.esh' with two echo commands.
>
> Emacs 29: Sourcing it and redirecting output to a file, results in the
> first echo string ('hello') showing in eshell, only the second
> ('there')(presumably because it is last) showing in the output file:
[snip]
>   I believe this is a bug but I am not sure about Eshell's intended
> features, so I wanted to check first.

This is definitely a bug. I think the best way to fix it though would be
to first replace 'eshell-do-eval', since it's a bit limited in what Lisp
forms it can manage correctly.

(Warning: the following is based on my memory of looking at this a few
months ago, so I might have misremembered some details. If so, sorry
about that.)

Currently, Eshell manages output handles in a fairly tricky way: many
internal Eshell forms automatically try to close the handles, so to get
around this, other Eshell forms increment the handles' refcount so that
closing just decrements that count instead of actually closing it.
That's a fine strategy in general, but the increment-refcount
('eshell-protect') and decrement-refcount-and-close
('eshell-close-handles') are called very far from each other in the
code, so the logic gets pretty hard to follow, leading to bugs like this.

I think the best way to fix this would be to put the increment and
decrement code in the same spot, so that you'd do something like so:

   (unwind-protect
       (progn
         (eshell-protect)
         (do-stuff))
     (eshell-close-handles))

Unfortunately, if 'do-stuff' calls a deferred command[1], Eshell ends up
evaluating 'eshell-close-handles' immediately, instead of when
'do-stuff' actually finishes. That means that either a) we'd need to fix
this without using 'unwind-protect', which would be painful to get
right, or b) we'd need to make 'unwind-protect' work as expected. (b) is
effectively bug#57635[2], I think. (That bug suggests using
generator.el's internals to drive Eshell's iterative evaluation.)

Bug#57635 is a pretty big change too, but I think it would greatly help
with the overall reliability of Eshell.

[1] Basically, a command that should yield execution back to the rest of
Emacs while waiting on some background task (e.g. a subprocess).

[2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=57635