Re: [Orgmode] [Babel] Executing sh code within sessions

[Eric Schulte]

Hi Seb,

Yes, I've noticed the shell session evaluation problems as well.  I
think one possible solution is for us to move from the current schema of
echoing a babel-specific marker to indicate that evaluation has
completed (which isn't working very well and is fairly ugly to boot) and
move towards something more like what Dan has done with R, where the
output is written to a file, and we check for that file to be created to
signal the end of session based evaluation.

This is certainly something that needs to be addressed, however I for
one won't have any time over the next week (I'm attending a conference).

Rest assured this issue is in our inbox. :)

Best Regards -- Eric

Sébastien Vauban <address@hidden> writes:

> Hi Eric,
>
> As requested, I'm here emphasizing the important part of this post:
>
> When executing sh code block within a session, it never terminates, echoing...
>
> : executing Sh code block...
>
> in the minibuffer.
>
> The =sva= shell buffer contains:
>
> #+begin_src sh
> echo In $(pwd):
>
> /address@hidden:...ples/Org-scraps 0$ In 
> /cygdrive/c/home/sva/Examples/Org-scraps:
> /address@hidden:...ples/Org-scraps 0$ echo In $(pwd):
> In /cygdrive/c/home/sva/Examples/Org-scraps:
> /address@hidden:...ples/Org-scraps 0$ ls *.org
> Agenda-Sorting-Strategy.org
> Clock-Report.org
> org-beamer-fpu-rules.org
> org-hist.org
> /address@hidden:...ples/Org-scraps 0$ echo 'org_babel_sh_eoe'
> org_babel_sh_eoe
> /address@hidden:...ples/Org-scraps 0$
> #+end_src
>
> Does that ring a bell to you?
>
> Details below, if needed.
>
> Best regards,
>   Seb
>
> Sébastien Vauban wrote:
>> #+TITLE:     Executing sh-code within a session
>> #+AUTHOR:    Seb Vauban
>> #+LANGUAGE:  en_US
>>
>> * Executing sh-code within a session
>>
>> ** Context
>>
>> - git pull done this morning (2010-11-04 Thu).
>>
>> - EmacsW32 + Cygwin bash as inferior shell
>>
>>   #+begin_src emacs-lisp
>>   (setq shell-file-name "bash")
>>   (setenv "SHELL" "bash")
>>   (setq shell-command-switch "-c")
>>   (setq w32-quote-process-args t)
>>   (setq explicit-shell-file-name "bash")
>>   #+end_src
>>
>> ** First time
>>
>> The first time I execute the following:
>>
>> #+begin_src sh :session sva
>> echo In $(pwd):
>> ls *.org
>> #+end_src
>>
>> I get the following error:
>>
>> #+begin_src emacs-lisp
>> Debugger entered--Lisp error: (error "Marker does not point anywhere")
>>   ansi-color-apply-on-region(#<marker in no buffer> #<marker at 17 in sva>)
>>   ansi-color-process-output("")
>>   run-hook-with-args(ansi-color-process-output "")
>>   comint-send-input(nil t)
>>   (lambda (line) (insert line) (comint-send-input nil t) (sleep-for 
>> 0.25))("echo In $(pwd):")
>>   mapc((lambda (line) (insert line) (comint-send-input nil t) (sleep-for 
>> 0.25)) ("echo In $(pwd):" "ls *.org" "echo 'org_babel_sh_eoe'"))
>>   (progn (goto-char (process-mark ...)) (let (... ...) (setq dangling-text 
>> ...) (delete-region start end)) (mapc (lambda ... ... ... ...) (append ... 
>> ...)) (while (progn ... ...) (accept-process-output ...)) (goto-char 
>> (process-mark ...)) (insert dangling-text))
>>   (unwind-protect (progn (goto-char ...) (let ... ... ...) (mapc ... ...) 
>> (while ... ...) (goto-char ...) (insert dangling-text)) (remove-hook (quote 
>> comint-output-filter-functions) (quote my-filt)))
>>   (progn (fset (quote my-filt) (function* ...)) (add-hook (quote 
>> comint-output-filter-functions) (quote my-filt)) (unwind-protect (progn ... 
>> ... ... ... ... ...) (remove-hook ... ...)))
>>   (unwind-protect (progn (fset ... ...) (add-hook ... ...) (unwind-protect 
>> ... ...)) (if --cl-letf-bound-- (fset ... --cl-letf-save--) (fmakunbound 
>> ...)))
>>   (let* ((--cl-letf-bound-- ...) (--cl-letf-save-- ...)) (unwind-protect 
>> (progn ... ... ...) (if --cl-letf-bound-- ... ...)))
>>   (letf ((... ...)) (add-hook (quote comint-output-filter-functions) (quote 
>> my-filt)) (unwind-protect (progn ... ... ... ... ... ...) (remove-hook ... 
>> ...)))
>>   (letf* ((... ...)) (add-hook (quote comint-output-filter-functions) (quote 
>> my-filt)) (unwind-protect (progn ... ... ... ... ... ...) (remove-hook ... 
>> ...)))
>>   (flet ((my-filt ... ...)) (add-hook (quote comint-output-filter-functions) 
>> (quote my-filt)) (unwind-protect (progn ... ... ... ... ... ...) 
>> (remove-hook ... ...)))
>>   (let ((string-buffer "") dangling-text raw) (flet (...) (add-hook ... ...) 
>> (unwind-protect ... ...)) (if (and t body ...) (setq raw ...)) (split-string 
>> string-buffer comint-prompt-regexp))
>>   (progn (unless (org-babel-comint-buffer-livep session) (error "buffer %s 
>> doesn't exist or has no process" session)) (set-buffer session) (let (... 
>> dangling-text raw) (flet ... ... ...) (if ... ...) (split-string 
>> string-buffer comint-prompt-regexp)))
>>   (unwind-protect (progn (unless ... ...) (set-buffer session) (let ... ... 
>> ... ...)) (set-match-data save-match-data-internal (quote evaporate)))
>>   (let ((save-match-data-internal ...)) (unwind-protect (progn ... ... ...) 
>> (set-match-data save-match-data-internal ...)))
>>   (save-match-data (unless (org-babel-comint-buffer-livep session) (error 
>> "buffer %s doesn't exist or has no process" session)) (set-buffer session) 
>> (let (... dangling-text raw) (flet ... ... ...) (if ... ...) (split-string 
>> string-buffer comint-prompt-regexp)))
>>   (save-excursion (save-match-data (unless ... ...) (set-buffer session) 
>> (let ... ... ... ...)))
>>   (org-babel-comint-in-buffer session (let (... dangling-text raw) (flet ... 
>> ... ...) (if ... ...) (split-string string-buffer comint-prompt-regexp)))
>>   (org-babel-comint-with-output (session org-babel-sh-eoe-output t body) 
>> (mapc (lambda ... ... ... ...) (append ... ...)))
>>   (butlast (org-babel-comint-with-output (session org-babel-sh-eoe-output t 
>> body) (mapc ... ...)) 2)
>>   (mapcar (function org-babel-trim) (butlast (org-babel-comint-with-output 
>> ... ...) 2))
>>   (mapconcat (function org-babel-sh-strip-weird-long-prompt) (mapcar 
>> (function org-babel-trim) (butlast ... 2)) "\n")
>>   (if (not session) (org-babel-eval org-babel-sh-command (org-babel-trim 
>> body)) (mapconcat (function org-babel-sh-strip-weird-long-prompt) (mapcar 
>> ... ...) "\n"))
>>   ((lambda (results) (when results ...)) (if (not session) (org-babel-eval 
>> org-babel-sh-command ...) (mapconcat ... ... "\n")))
>>   org-babel-sh-evaluate(#<buffer sva> "echo In $(pwd):\nls *.org\n" 
>> ("replace"))
>>   (org-babel-reassemble-table (org-babel-sh-evaluate session full-body 
>> result-params) (org-babel-pick-name (cdr ...) (cdr ...)) 
>> (org-babel-pick-name (cdr ...) (cdr ...)))
>>   (let* ((session ...) (result-params ...) (full-body ...)) 
>> (org-babel-reassemble-table (org-babel-sh-evaluate session full-body 
>> result-params) (org-babel-pick-name ... ...) (org-babel-pick-name ... ...)))
>>   org-babel-execute:sh("echo In $(pwd):\nls *.org\n" ((:colname-names) 
>> (:rowname-names) (:result-params "replace") (:result-type . value) 
>> (:comments . "") (:shebang . "") (:cache . "no") (:noweb . "no") (:tangle . 
>> "no") (:exports . "code") (:results . "replace") (:session . "sva") (:hlines 
>> . "no")))
>>   funcall(org-babel-execute:sh "echo In $(pwd):\nls *.org\n" 
>> ((:colname-names) (:rowname-names) (:result-params "replace") (:result-type 
>> . value) (:comments . "") (:shebang . "") (:cache . "no") (:noweb . "no") 
>> (:tangle . "no") (:exports . "code") (:results . "replace") (:session . 
>> "sva") (:hlines . "no")))
>>   ((lambda (result) (cond ... ... ...)) (funcall cmd body params))
>>   (setq result ((lambda ... ...) (funcall cmd body params)))
>>   (if (and (not arg) new-hash (equal new-hash old-hash)) (save-excursion 
>> (goto-char ...) (end-of-line 1) (forward-char 1) (setq result ...) (message 
>> ...) result) (message "executing %s code block%s..." (capitalize lang) (if 
>> ... ... "")) (setq result (... ...)) (org-babel-insert-result result 
>> result-params info new-hash indent lang) (run-hooks (quote 
>> org-babel-after-execute-hook)) result)
>>   (progn (fset (quote call-process-region) (function* ...)) (unless (fboundp 
>> cmd) (error "No org-babel-execute function for %s!" lang)) (if (and ... 
>> new-hash ...) (save-excursion ... ... ... ... ... result) (message 
>> "executing %s code block%s..." ... ...) (setq result ...) 
>> (org-babel-insert-result result result-params info new-hash indent lang) 
>> (run-hooks ...) result))
>>   (unwind-protect (progn (fset ... ...) (unless ... ...) (if ... ... ... ... 
>> ... ... result)) (if --cl-letf-bound-- (fset ... --cl-letf-save--) 
>> (fmakunbound ...)))
>>   (let* ((--cl-letf-bound-- ...) (--cl-letf-save-- ...)) (unwind-protect 
>> (progn ... ... ...) (if --cl-letf-bound-- ... ...)))
>>   (letf ((... ...)) (unless (fboundp cmd) (error "No org-babel-execute 
>> function for %s!" lang)) (if (and ... new-hash ...) (save-excursion ... ... 
>> ... ... ... result) (message "executing %s code block%s..." ... ...) (setq 
>> result ...) (org-babel-insert-result result result-params info new-hash 
>> indent lang) (run-hooks ...) result))
>>   (letf* ((... ...)) (unless (fboundp cmd) (error "No org-babel-execute 
>> function for %s!" lang)) (if (and ... new-hash ...) (save-excursion ... ... 
>> ... ... ... result) (message "executing %s code block%s..." ... ...) (setq 
>> result ...) (org-babel-insert-result result result-params info new-hash 
>> indent lang) (run-hooks ...) result))
>>   (flet ((call-process-region ... ...)) (unless (fboundp cmd) (error "No 
>> org-babel-execute function for %s!" lang)) (if (and ... new-hash ...) 
>> (save-excursion ... ... ... ... ... result) (message "executing %s code 
>> block%s..." ... ...) (setq result ...) (org-babel-insert-result result 
>> result-params info new-hash indent lang) (run-hooks ...) result))
>>   (unwind-protect (flet (...) (unless ... ...) (if ... ... ... ... ... ... 
>> result)) (setq call-process-region (quote 
>> org-babel-call-process-region-original)))
>>   (let* ((lang ...) (params ...) (cache\? ...) (result-params ...) (new-hash 
>> ...) (old-hash ...) (body ...) (cmd ...) (dir ...) (default-directory ...) 
>> (org-babel-call-process-region-original ...) (indent ...) result) 
>> (unwind-protect (flet ... ... ...) (setq call-process-region ...)))
>>   (progn (let* (... ... ... ... ... ... ... ... ... ... ... ... result) 
>> (unwind-protect ... ...)))
>>   (if (org-babel-confirm-evaluate info) (progn (let* ... ...)))
>>   (when (org-babel-confirm-evaluate info) (let* (... ... ... ... ... ... ... 
>> ... ... ... ... ... result) (unwind-protect ... ...)))
>>   (let ((info ...)) (when (org-babel-confirm-evaluate info) (let* ... ...)))
>>   org-babel-execute-src-block(nil ("sh" "echo In $(pwd):\nls *.org\n" 
>> ((:colname-names) (:rowname-names) (:result-params "replace") (:result-type 
>> . value) (:comments . "") (:shebang . "") (:cache . "no") (:noweb . "no") 
>> (:tangle . "no") (:exports . "code") (:results . "replace") (:session . 
>> "sva") (:hlines . "no")) "" nil 0))
>>   (progn (org-babel-execute-src-block current-prefix-arg info) t)
>>   (if info (progn (org-babel-execute-src-block current-prefix-arg info) t) 
>> nil)
>>   (let ((info ...)) (if info (progn ... t) nil))
>>   org-babel-execute-src-block-maybe()
>>   (or (org-babel-execute-src-block-maybe) (org-babel-lob-execute-maybe))
>>   org-babel-execute-maybe()
>>   call-interactively(org-babel-execute-maybe)
>>   (cond ((commandp org-speed-command) (setq this-command org-speed-command) 
>> (call-interactively org-speed-command)) ((functionp org-speed-command) 
>> (funcall org-speed-command)) ((and org-speed-command ...) (eval 
>> org-speed-command)) (t (let ... ...)))
>>   (cond ((and org-use-speed-commands ...) (cond ... ... ... ...)) ((and ... 
>> ... ... ...) (let ... ... ... ... ...)) (t (setq org-table-may-need-update 
>> t) (self-insert-command N) (org-fix-tags-on-the-fly) (if 
>> org-self-insert-cluster-for-undo ...)))
>>   org-self-insert-command(1)
>>   call-interactively(org-self-insert-command nil nil)
>> #+end_src
>>
>> ** Next times
>>
>> The second time I execute it:
>>
>> #+begin_src sh :session sva
>> echo In $(pwd):
>> ls *.org
>> #+end_src
>>
>> it never terminates, echoing:
>>
>> : executing Sh code block...
>>
>> in the minibuffer.
>>
>> The =sva= shell buffer contains:
>>
>> #+begin_src sh
>> echo In $(pwd):
>>
>> /address@hidden:...ples/Org-scraps 0$ In 
>> /cygdrive/c/home/sva/Examples/Org-scraps:
>> /address@hidden:...ples/Org-scraps 0$ echo In $(pwd):
>> In /cygdrive/c/home/sva/Examples/Org-scraps:
>> /address@hidden:...ples/Org-scraps 0$ ls *.org
>> Agenda-Sorting-Strategy.org
>> Clock-Report.org
>> org-beamer-fpu-rules.org
>> org-hist.org
>> /address@hidden:...ples/Org-scraps 0$ echo 'org_babel_sh_eoe'
>> org_babel_sh_eoe
>> /address@hidden:...ples/Org-scraps 0$
>> #+end_src
>>
>> ** Value of my Bash prompt
>>
>> It now is defined as such:
>>
>> #+begin_src sh
>> # my format of the prompt: mono-line and (often) colored
>> function my_prompt_command ()
>> {
>>     RET=$?
>>
>>     # replace the $HOME prefix by ~ in the current directory
>>     if [[ $HOME == ${PWD:0:${#HOME}} ]]; then
>>         NEWPWD="~${PWD:${#HOME}}"
>>     else
>>         NEWPWD=$PWD
>>     fi
>>
>>     # how many characters of the $PWD should be kept
>>     local pwd_max_length=15
>>
>>     if [[ ${#NEWPWD} -gt $pwd_max_length ]]; then
>>         local pwd_offset=$(( ${#NEWPWD} - $pwd_max_length ))
>>         NEWPWD="...${NEWPWD:$pwd_offset:$pwd_max_length}"
>>     fi
>>
>>     # prompt character
>>     if [[ $(whoami) = "root" ]]; then
>>         local PROMPTCHAR="#"
>>     else
>>         local PROMPTCHAR="$"
>>     fi
>>
>>     setenv PS1 "/address@hidden:${NEWPWD} ${RET}${PROMPTCHAR} "
>> }
>>
>> # shell prompt
>> setenv PROMPT_COMMAND my_prompt_command
>> #+end_src
>>
>> So, it now definitively is:
>>
>> - mono-line
>> - uncolored
>>
>> That should be the basic requirements I needed to meet, right?
>>
>> ** Variable comint-prompt-regexp
>>
>> Value of the =comint-prompt-regexp= variable in my shell, used by Org-babel 
>> to
>> digest output from the shell:
>>
>> #+begin_src emacs-lisp
>> (switch-to-buffer "sva")
>> (describe-variable 'comint-prompt-regexp)
>> #+end_src
>>
>> #+results:
>> #+begin_example
>> comint-prompt-regexp is a variable defined in `comint.el'.
>> Its value is 
>> "^[^#$%>\n]*[#$%>] *"
>> Local in buffer sva; global value is "^"
>>
>> Documentation:
>> Regexp to recognize prompts in the inferior process.
>> Defaults to "^", the null string at BOL.
>>
>> This variable is only used if the variable
>> `comint-use-prompt-regexp' is non-nil.
>>
>> Good choices:
>>   Canonical Lisp: "^[^> \n]*>+:? *" (Lucid, franz, kcl, T, cscheme, oaklisp)
>>   Lucid Common Lisp: "^\\(>\\|\\(->\\)+\\) *"
>>   franz: "^\\(->\\|<[0-9]*>:\\) *"
>>   kcl: "^>+ *"
>>   shell: "^[^#$%>\n]*[#$%>] *"
>>   T: "^>+ *"
>>
>> This is a good thing to set in mode hooks.
>> #+end_example
>>
>> This regexp should match my own prompt as it appears in your *shell*
>> buffers locally (product of my personal =.bashrc=) configuration.
>>
>> It does, right?
>>
>> ** What about shell-prompt-pattern?
>>
>> Browsing along, it seems there is another interesting value to look at:
>> =shell-prompt-pattern=.
>>
>> #+begin_src emacs-lisp
>> (switch-to-buffer "sva")
>> (describe-variable 'shell-prompt-pattern)
>> #+end_src
>>
>> #+results:
>> #+begin_example
>> shell-prompt-pattern is a variable defined in `shell.el'.
>> Its value is 
>> "^[^#$%>\n]*[#$%>] *"
>>
>> Documentation:
>> Regexp to match prompts in the inferior shell.
>> Defaults to "^[^#$%>\n]*[#$%>] *", which works pretty well.
>> This variable is used to initialize `comint-prompt-regexp' in the
>> shell buffer.
>>
>> If `comint-use-prompt-regexp' is nil, then this variable is only used
>> to determine paragraph boundaries.  See Info node `Shell Prompts' for
>> how Shell mode treats paragraphs.
>>
>> The pattern should probably not match more than one line.  If it does,
>> Shell mode may become confused trying to distinguish prompt from input
>> on lines which don't start with a prompt.
>>
>> This is a fine thing to set in your `.emacs' file.
>>
>> You can customize this variable.
>> #+end_example
>>
>> I don't know if this was important, but -- as you can see -- I've set it to
>> the same value as =comint-prompt-regexp=.
>>
>> ** Back to comint's prompt regexp!
>>
>> #+begin_src emacs-lisp
>> (switch-to-buffer "sva")
>> (describe-variable 'comint-use-prompt-regexp)
>> #+end_src
>>
>> #+results:
>> #+begin_example
>> comint-use-prompt-regexp is a variable defined in `comint.el'.
>> Its value is nil
>>
>> Documentation:
>> If non-nil, use `comint-prompt-regexp' to recognize prompts.
>> If nil, then program output and user-input are given different `field'
>> properties, which Emacs commands can use to distinguish them (in
>> particular, common movement commands such as `beginning-of-line'
>> respect field boundaries in a natural way).
>>
>> You can customize this variable.
>> #+end_example
>>
>> NIL!?  Important?
>>
>> It should not. The problems seem to be somewhere else... but where?  Can you
>> give me a hint?[1]
>>
>> Best regards,
>>   Seb
>>
>> Footnotes:
>> [1] Sorry for this looooong mail. Really. But I hope its Org form will be on
>> the "good" side...