Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])

[Matt]

If I followed correctly, the topic switched to discussing async generally 
within Babel.  I've started a new thread accordingly.

 ---- On Tue, 06 Feb 2024 20:24:13 +0100 Bruno Barbier

 > FWIW, I've been trying to use asynchronous blocks for everything, not only 
 > the source blocks that are based on the comint mode.

I've been trying to figure out how to make everything async by default.  I'm 
super interested to see what you've come up with.

 > I think it would be good if ob-core itself could provide an asynchronous API.

Fortunately or unfortunately, depending on how you look at it, Babel already 
does.

The challenge is that the Org Babel API has grown piecemeal over 14 years.  
It's been written by several authors with limited time and knowledge.  The 
result, while powerful and useful, is a bit of a hodgepodge.  A prime example 
is that the concepts of "persistence" and "synchronicity" are conflated.  
"Session" is often used to mean "asynchronous" even though the two ideas are 
orthogonal.  Emacs provides primitives that could make non-session blocks 
asynchronous.  It's historical accident that blocks aren't async by default.

For me, the issue is that the Babel API needs some high level perspective in 
order to make it consistent.  I see the following terms as guides.  If we can 
separate these concepts within the API, then Babel to *feel* like an API:

- "Session" means a shell environment is "persistent."  Each call is executed 
in the same environment.  State exists between calls.

- "Non-session" means a shell environment is "temporary."  Each call is 
executed in an independent environment.  State does not exist between calls.

- "Synchronous" means that execution prevents the user from editing the 
document while results are obtained.

- "Asynchronous" means that execution does not prevent the user from editing 
the document while results are obtained.

 > I've modified my Org so that it does have such an API.  This is work in 
 > progress; let me describe it.
 >
 > I've modified ob-core itself to allow asynchronicity.  In the asynchrosous 
 > case, instead of calling:
 >
 >   (org-babel-execute:LANG body params)
 >
 > I'm calling:
 >
 >   (org-babel-schedule:LANG body params handle-result)
 >
 > where `org-babel-schedule:LANG' is in charge of calling `handle-result' with 
 > the result (or the error) when it is known; `handle-result' takes care to 
 > call `org-babel-insert-result' at the correct place (and 
 > `org-babel-insert-result' is only called with a real result).
 >
 > While the execution is pending, I'm using the same technique that Org is 
 > using when a source block is being edited: the result is left untouched, but 
 > below an overlay.  The overlay is used to know where to insert the result 
 > and to display the status/progress of the execution.  If the file is closed 
 > and the execution fails, nothing is lost, the old result is still available.

The use of the overlay is a really cool idea!

I hesitate to say that's a good way to convey success or failure.  If a process 
failed, I want to see the output which tells me why so that I can correct it.  
Or, I might actually want the failure output.  Maybe I want to literally 
demonstrate what a code failure looks like.  Maybe I want to use that output in 
another block.  For example, shell blocks have multiple output types.  A shell 
process may return standard output/error or a failure code.  The result of the 
failure may trigger something else.

However, using an overlay to communicate "the process is still running" could 
be good.  We'd need to be careful about accessibility, though, and make sure 
the overlay is apparent, visually and otherwise.

 > If that technique looks safe enough and interesting, I can prepare a set of 
 > patches so that we can discuss it further and, maybe, add it in Org.

Please do!  I'm super interested.  I've put a lot of thought into how we might 
make Babel async by default.  I'm excited to see your interest in the topic and 
look forward to seeing what you've come up with.

--
Matt Trzcinski
Emacs Org contributor (ob-shell)
Learn more about Org mode at https://orgmode.org
Support Org development at https://liberapay.com/org-mode