Re: bash source code block: problem after ssh commands

[Matt]

 ---- On Fri, 17 Nov 2023 10:20:28 +0100  Ihor Radchenko  wrote --- 

 > This has nothing to do with Emacs comint and this is also not a bug in
 > Emacs 

Ihor, there were two claims made in the original report.  I was referring to 
Claim 2.  That deals with M-x shell and therefore comint-mode.

Regarding Claim 1:

- Can anyone verify Claim 1?
- Is anyone else unable to verify Claim 1 (like me)?
- What versions are people using?
  + M-x org-version
  + M-x emacs-version

I'm running Org mode version 9.7-pre (release_9.6.10-903-g9183e3.dirty @ 
/home/ahab/.emacs.d/straight/build/org/) on GNU Emacs 29.1 (build 1, 
x86_64-pc-linux-gnu, GTK+ Version 3.24.37, cairo version 1.16.0).

* The original report has two claims:
** Claim 1.
The following block is expected to write a remote file called "foo_file" with 
contents "foo" as well as give "bar" as the result.

    #+begin_src bash :results output
    ssh cochard@fruc.u-strasbg.fr "echo foo>foo_file"
    echo "bar"
    #+end_src

The reported behavior is that "foo_file" is created with "foo" (with "foo" is 
not stated, but implied) and "bar" is *not* given as the result.

** Claim 2.
Copying and pasting the two lines from the first claim into a terminal like 
xfce4-terminal executes the ssh line as expected and outputs the result of the 
second line.  It was noted that this does not happen with M-x shell.

* Comments about the claims:
** Comment 1.
tl;dr I can't reproduce the claim that "bar" is *not* the result.  The result 
is "bar" for me.

The exact "expected behavior" for a shell block is a little fuzzy.  According 
to my analysis (given below), what Alain reports (remote file and no "bar") is 
the "expected" behavior.  What I see (no remote file and "bar") is actually 
"unexpected".

I used the following to test the claim:

    #+begin_src bash :results output
    ssh localhost "echo foo>foo_file"
    echo "bar"
    #+end_src

I am unable to reproduce the reported behavior (of "bar" not returning).  
Instead, I get an ssh-askpass permission denied error, foo_file is not created, 
and "bar" is given as the result.  I do not see anywhere in the thread that the 
original claim was reproduced.

The thread preceded something like follows.

Leo Butler suggested two work arounds:

- add the -f to the ssh command
- add a semi-colon and line continuation to the first line.

Russell Adams suggested another work around:

- add -n to the ssh command

Ihor identified that a non-session call does something like the following 
command:

    bash -c bash /tmp/temp-file-with-source-block-code.sh

where

    ----- /tmp/temp-file-with-source-block-code -----
    ssh localhost "echo foo>foo_file"
    echo "bar" | tee /tmp/bar.txt
    -------------------------------------------------

The second line (significantly different from the original report) pipes the 
echo result to stdout and to a file, bar.txt.  Writing to a file allows us to 
confirm if that line was executed.

Ihor suggested that

    bash -c bash /tmp/temp-file-with-source-block-code.sh

does not run the second line because an interactive password prompt is 
displayed by ssh.  The reasoning is that the prompt hangs the process while 
waiting for input and the second line never runs.  Indeed, running the command 
does not produce /tmp/bar.txt.

Ihor is correct about prompts messing with shell blocks (this is not the first 
time he's seen this).  However, the way it's stated does not demonstrate it. 
This is because Emacs does *not* make a call like

    bash -c bash /tmp/temp-file-with-source-block-code.sh

Alain responded by pointing out that

    bash -c bash /tmp/temp-file-with-source-block-code.sh

does not execute the first line.  This is true.  Consider calling

    bash -c bash /tmp/two-lines.sh

where

    ------ /tmp/two-lines.sh ------
    echo "first" > /tmp/first.txt
    echo "second" > /tmm/second.txt
    -------------------------------

Neither first.txt or second.txt are created.

Max Nikulin interjected with a helpful reminder that Bash scripting is a 
snakepit of footguns.  (What Max said is more than that and interesting.  I 
skip it here because it depends on the form of the call.)

Before trying to untangle what a given Bash command does, we need to be sure 
what command is actually called.  Unfortunately, there's not a clear Bash 
command corresponding to how Emacs makes the call.

What happens goes something like this:

1. The Lisp function process-file is called with PROGRAM "bash", INFILE a path 
to a temp file containing the source block code, and ARGS ("-c" "bash")
2. This information is passed to DEFUN ("call-process"), a Lisp object 
implemented in C
3. DEFUN ("call-process") forwards this information to the C function 
call_process
4. call_process calls emacs_spawn
5. emacs_spawn creates a subprocess

A lot of cleaning and setup happens which is dependent on the system (GNU, 
Window, Darwin, etc.).  There's a call to openp which looks for the executable. 
 An emacs_pipe is set up (I assume for writing to stdout and stderr).  I'm 
unable to give a definitive answer as to what precisely emacs_spawn calls.  Do 
any of the args "get quotes?"  I can't say.

Bruno Barbier commented that his understanding of process-file is that it gets 
commands from stdin.  Maybe that's what the call to emacs_pipe is doing?

He gives the example:

#+begin_quote
(process-file "bash" "/tmp/test.sh")

is more equivalent to:

cat /tmp/test.sh | bash
#+end_quote

He then proposes an experiment to close stdin.  To do this, he calls

    #+begin_src shell :results output
    exec 0>&-
    echo OK
    #+end_src

He claims that "exec 0<&-" closes stdin.  I believe there is a typo.  It's not 
clear if it has a negative effect, though.  According to the 
[[https://tldp.org/LDP/abs/html/io-redirection.html][Advanced Bash-Scripting 
Guide]],

#+begin_quote
Closing File Descriptors

n<&-

    Close input file descriptor n.
0<&-, <&-

    Close stdin.
n>&-

    Close output file descriptor n.
1>&-, >&-

    Close stdout.
#+end_quote

What Bruno writes corresponds to "closing output file descriptor 0".  I 
honestly don't know what the difference is between an "output file descriptor" 
and an "input file descriptor".  I had no luck finding this information in man 
bash or info bash.

Rerunning the experiment according to the 
[[https://tldp.org/LDP/abs/html/io-redirection.html][Advanced Bash-Scripting 
Guide]], the result is the same: "OK" is *not* printed.

    #+begin_src shell :results output
    exec 0<&-
    echo OK
    #+end_src

Doing the following echoes OK for either direction of the redirection:

  -- /tmp/exec-OKlt.sh ---
  exec 0<&-
  echo OK
  ----------------------

  #+begin_example
  bash /tmp/exec-OKlt.sh
  #+end_example

  -- /tmp/exec-OKgt.sh ---
  exec 0>&-
  echo OK
  ----------------------

  #+begin_example
  bash /tmp/exec-OKgt.sh
  #+end_example

The INFILE passed to process-file looks like,

    #+begin_src emacs-lisp
    (process-file
       "bash"
       "/tmp/two-lines.sh"
       '(t "/tmp/babel-mS0Yyg/ob-error-AoxNqH")
       nil
       "-c" "bash")
    #+end_src

So, the call Emacs makes is probably more close to:

    #+begin_example
    cat /tmp/two-lines.sh | bash -c bash
    #+end_example

What this exactly does is unclear to me.  It appears to pass the contents of 
/tmp/two-lines.sh to a subshell process.  That is, it seems to behave like 
"bash /tmp/two-lines.sh" is run in a subprocess.

Running this in xfce4-terminal, I get what I expect:

    #+begin_example
    cat /tmp/two-lines.sh | bash -c bash
    #+end_example

Each line is echoed to file so nothing is written to the console.  However, 
both files are created with the expected text.  Both lines executed.

If I update two-lines to output to std,

    ------ /tmp/two-lines-tee.sh ------
    echo "first"  | tee /tmp/first.txt
    echo "second" | tee /tmp/second.txt
    -----------------------------------

I see "first" and "second" echoed to the console:

    ahab@pequod /tmp$ cat two-lines-tee.sh | bash -c bash
    first
    second

Running the following, neither give an output to the console:

#+begin_example
ahab@pequod /tmp$ cat exec-OKlt.sh | bash -c bash
ahab@pequod /tmp$ cat exec-OKgt.sh | bash -c bash
#+end_example

This is what we see in Org.  I'll be honest, though, I don't really know what 
to expect with exec 0>&- and exec 0<&-.  When I call them in the terminal, it 
kills the terminal.

The surprising bit is that running this in xfce4-terminal

    ----- /tmp/temp-file-with-source-block-code -----
    ssh localhost "echo foo>foo_file"
    echo "bar" | tee /tmp/bar.txt
    -------------------------------------------------

    #+begin_example
    cat /tmp/temp-file-with-source-block-code.sh | bash -c bash
    #+end_example

does *not* echo bar (and does not create /tmp/bar.txt) yet it creates foo_file. 
 I get prompted for my password and then the second line doesn't execute.  
Nothing prints to the console and no bar.txt is created.

This is the behavior Alain reports happening in Org (that I am unable to 
reproduce).  That is, the *reported behavior is the expected behavior* 
(assuming my analysis is correct).  However, according to the behavior I see 
when I run the block (fails to create the remote file and echoes "bar"), Org 
does the "wrong thing".  I can't account for this.

Anyway, Ihor's main point stands: a prompt does not work with non-session shell 
blocks.  The following returns exit code 1 (which means fail):

    #+begin_src bash :results output
    read -p "What? "
    #+end_src

As far as I can tell, though, that's not what prevents "bar" from being 
returned.  As far as I can reproduce, calling

    #+begin_src bash :results output
    ssh localhost "echo foo>foo_file"
    echo "bar"
    #+end_src

*does* give "bar" for results even though it shouldn't.

** Comment 2.
The second claim has nothing to do with Org Babel.  I was able to confirm it 
and provide the steps to reproduce.  I think it would make sense to report it 
upstream and let them decide if it's expected behavior.  I'm still happy to do 
that, but I need to step away from the keyboard :)