[BUG] Prompt appears in async shell results

[Matt]

* [BUG] Prompt appears in async shell results
** Minimal reproducible example
#+begin_src emacs-lisp
(org-babel-do-load-languages 'org-babel-load-languages '((shell . t)))
#+end_src

#+begin_src sh :results output :session *test* :async t
cd /tmp
echo "hello world"
#+end_src

#+RESULTS:
: org_babel_sh_prompt> hello world

or

#+begin_src sh :results output :session *test* :async t
# comment
# comment
#+end_src

#+RESULTS:
: org_babel_sh_prompt> org_babel_sh_prompt>

or

#+begin_src sh :results output :session *test* :async t
# print message
echo \"hello world\"
#+end_src

#+RESULTS:
: org_babel_sh_prompt> "hello world"

Interestingly, this returns without the prompt:

,#+begin_src sh :results output :session *test* :async t
echo "hello"
echo "world"
#+end_src

#+RESULTS:
: hello
: world

** Test
Here's a test that checks one of the MREs:
#+begin_src emacs-lisp
(ert-deftest test-ob-shell/session-async-removes-prompt-from-results ()
  "Test that async evaluation removes prompt from results."
  (let* ((session-name 
"test-ob-shell/session-async-removes-prompt-from-results")
         (kill-buffer-query-functions nil)
         (start-time (current-time))
         (wait-time (time-add start-time 3))
         uuid-placeholder)
    (org-test-with-temp-text
        (concat "#+begin_src sh :session " session-name " :async t
# print message
echo \"hello world\"<point>
,#+end_src")
      (setq uuid-placeholder (org-trim (org-babel-execute-src-block)))
      (catch 'too-long
        (while (string-match uuid-placeholder (buffer-string))
          (progn
            (sleep-for 0.01)
            (when (time-less-p wait-time (current-time))
              (throw 'too-long (ert-fail "Took too long to get result from 
callback"))))))
    (search-forward "#+results")
    (beginning-of-line 2)
    (if (should (string= ": hello world\n" (buffer-substring-no-properties 
(point) (point-max))))
          (kill-buffer session-name)))))
#+end_src

** Thoughts
A quick fix is:

#+begin_src diff
modified   lisp/ob-shell.el
@@ -289,7 +289,7 @@ See `org-babel-comint-async-indicator'.")
 (defun ob-shell-async-chunk-callback (string)
   "Filter applied to results before insertion.
 See `org-babel-comint-async-chunk-callback'."
-  (replace-regexp-in-string comint-prompt-regexp "" string))
+  (replace-regexp-in-string org-babel-sh-prompt "" string))

 (defun org-babel-sh-evaluate (session body &optional params stdin cmdline)
   "Pass BODY to the Shell process in BUFFER.
#+end_src

I'm not sure this is the best way.

There are two ways I can think to look at it: how text is passed to the process 
and what's done with it afterward.

Regarding how text is passed to the process:

Blocks without :session and :async are sent via =process-file=.  Blocks with 
:session, including those with :async, are inserted into a process buffer and 
sent to the process with =comint-send-input=.  The details differ, but I think 
the general concept holds: a chunk of text is inserted into the process buffer 
and that chunk is then sent to the process.  This is in contrast to successive 
insert-send pairs.  AFAICT, there's no major difference in how text is passed 
to the process between :session only and :session with :async.

Regarding how process results are handled:

AFAIU, =process-file= interfaces directly with the process and no terminal 
emulation is involved.  So, there's no prompt to worry about.  Blocks with only 
:session go through =org-babel-comint-with-output= which filters out the prompt 
(https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tree/lisp/ob-comint.el#n110).
  Async block results go through =org-babel-comint-async-filter= which doesn't 
filter results 
(https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tree/lisp/ob-comint.el#n212).

It seems to me that we should extract the filter from 
=org-babel-comint-with-output= and use it in both 
=org-babel-comint-with-output= and =org-babel-comint-async-filter=.  

Thoughts?

--
Matt Trzcinski
Emacs Org contributor (ob-shell)
Learn more about Org mode at https://orgmode.org
Support Org development at https://liberapay.com/org-mode