Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])

[Bruno Barbier]

Hi Matt, Jack, Ihor,

Sorry for the late reply.  Cleaning the code took me longer than
expected.


Jack Kamm <jackkamm@gmail.com> writes:

> Bruno Barbier <brubar.cs@gmail.com> writes:
>
>> FWIW, I've been trying to use asynchronous blocks for everything, not
>> only the source blocks that are based on the comint mode.
>> ...
>
> Sounds interesting, a couple questions:
>
> 1. Which languages have you implemented/tested this on?

I'm not using it with official org backends (yet).  I'm using it with
several custom backends that I'm working on.  One of the backend
delegate the block executions to emacs subprocesses: so I have a kind of
asynchronous executions for free for any language, including elisp
itself.


> 2. Does it apply for sessions, nonsessions, or both?
>

The new API itself is more about how to wait for and display one block
result.  So, it's not really aware of sessions.

But, I usually try to think as "no session" as a "one shot" session
(like Matt wrote in his email).  So, in that sense, it works for both
anyway ;-)


> Also interesting, I think it's worth exploring/testing this overlay idea
> out. Does that mean that output is asynchronously printing into the Org
> buffer? It sounds cool but I wonder if it might cause problems while
> trying to edit another part of the buffer.

Currently, I've limited the progress feedback to fit on one line to
avoid anoying vertical display jumps.  When the computation is
successful, the result is inserted as usual, and, that may be annoying;
when it updates a previous result of the same height, it's ok.  Else, we
could fold it to stay on one line too (using the overlay), until the
user explicitly request to see it.

Jack Kamm <jackkamm@gmail.com> writes:
>
> That all sounds reasonable...if you work on this, let me know if you
> want any help with testing.

Thanks.  I'll definitely appreciate your help, to test the current
code with the Python backend or any other backend if you prefer.


Matt <matt@excalamus.com> writes:

> If I followed correctly, the topic switched to discussing async
> generally within Babel.  I've started a new thread accordingly.

Indeed. Thank you!


>  > FWIW, I've been trying to use asynchronous blocks for everything, not only 
> the source blocks that are based on the comint mode.
>
> I've been trying to figure out how to make everything async by default.  I'm 
> super interested to see what you've come up with.

Good to know.  Let's we make this happen! 


>  > I think it would be good if ob-core itself could provide an asynchronous 
> API.
>
> Fortunately or unfortunately, depending on how you look at it, Babel already 
> does.
>

> The challenge is that the Org Babel API has grown piecemeal over 14 years.  
> It's been written by several authors with limited time and knowledge.  The 
> result, while powerful and useful, is a bit of a hodgepodge.  A prime example 
> is that the concepts of "persistence" and "synchronicity" are conflated.  
> "Session" is often used to mean "asynchronous" even though the two ideas are 
> orthogonal.  Emacs provides primitives that could make non-session blocks 
> asynchronous.  It's historical accident that blocks aren't async by default.

> For me, the issue is that the Babel API needs some high level perspective in 
> order to make it consistent.

> I see the following terms as guides.  If we can separate these concepts 
> within the API, then Babel to *feel* like an API:
>
> - "Session" means a shell environment is "persistent."  Each call is executed 
> in the same environment.  State exists between calls.
>
> - "Non-session" means a shell environment is "temporary."  Each call is 
> executed in an independent environment.  State does not exist between calls.
>
> - "Synchronous" means that execution prevents the user from editing the 
> document while results are obtained.
>
> - "Asynchronous" means that execution does not prevent the user from editing 
> the document while results are obtained.

I mostly think the same.  Sessions (including the "none" session)
definitely need some generic API and some generic tests that all
backends could just reuse.

To execute Python blocks, using the proposed async API:

   - I've (re)implemented the "asynchronous with session" case (copying/pasting
     the relevant part from ob-python).
   
   - The "synchronous case" is just artificially blocking the user until
     the asynchronous result is known (which looks incredibly tricky to
     implement if even possible...).
   
   - The "no session" case is just about creating a new unique session
     and throwing it away immediately.

But, some users may rely on some particular behavior, of the current
implementations, that may be hard to implement in such a generic way.


> The use of the overlay is a really cool idea!
>
> I hesitate to say that's a good way to convey success or failure.  If a 
> process failed, I want to see the output which tells me why so that I can 
> correct it.  Or, I might actually want the failure output.  Maybe I want to 
> literally demonstrate what a code failure looks like.  Maybe I want to use 
> that output in another block.  For example, shell blocks have multiple output 
> types.  A shell process may return standard output/error or a failure code.  
> The result of the failure may trigger something else.

I'm not sure I fully understand what you mean. The API just assumes the
backend returns the outcome: either success or failure, where failure
means "no result" (the previous result, if it exists, is even preserved
in the document).  The backend is free to transform a failure into a
success to make that result available though.


> However, using an overlay to communicate "the process is still running" could 
> be good.  We'd need to be careful about accessibility, though, and make sure 
> the overlay is apparent, visually and otherwise.

You should be able to test my current implementation, see below.  It's
almost too visual in my opinion.  But, that's probably something that we
should make easy to configure.

>  > If that technique looks safe enough and interesting, I can prepare a set 
> of patches so that we can discuss it further and, maybe, add it in Org.
>
> Please do!  I'm super interested.  I've put a lot of thought into how we 
> might make Babel async by default.  I'm excited to see your interest in the 
> topic and look forward to seeing what you've come up with.

Good to know!


So, here we go.  You'll find attach a set of patchs.  It works for me with
Emacsc 30.50 and 9.7-pre (from today).

I didn't check yet that the code and the commits follow all the
guidelines.  This is just a preliminary version for feedbacks.
Corrections/critiques are welcome, but don't check the details until I
check them myself.


The 5 first patchs provide an API to handle asynchronous execution in
ob-core, i.e. an API to report progress and to insert results in the
asynchronous case.  The 5th one isn't really about asynchronicity; but
it adds a new keyword `:execute-with' that allows to delegate block
executions to some other package; it's useful for testing or plugging
other execution engines.

That's a cleaned-up version of what I been using myself for a while,
with 4 different values for `:execute-with'.

The remaining patchs are new code that I've just written to show how to
use this new API.

I tried first to use it for ob-python, as an example.  I just needed to
figure out where to place the callbacks ... well ... "just" ... :)


So, I decided to rewrite the whole thing, taking code from the
synchronous case (following `org-babel-python-evaluate-session').  I
also created a package that contains all the functions that should be
reusable for any language.  The patch [1] adds some generic functions to
help dealing with asynchronicity (process, comint, etc.). The patch [2]
shows a new possible way to execute python code blocks, both
synchronously and asynchronously, with or without sessions.  You should
just need to open [3] and follow what's written there, and execute the
existing bash and python code blocks.

Note that this will create a folder `scratch/bba-ob-core-async' in your
repository to place the temporary files there.  If you know a better way
to do this, let me know, thanks.

I think the first 5 patchs could be included almost as-is in org.  About
the emaining ones, I'm not sure exactly how we should proceed.

Let me know if you need help to test it.

Feedbacks, corrections, critiques, etc are most welcome!

Thanks!

Bruno




[1] lisp/org-elib-async.el: New package about async helpers
[2] scratch/bba-ob-core-async: Some temporary test files
[3] scratch/bba-ob-core-async/my-async-tests.org

0001-ob-core-async-Add-faces-1-5.patch
Description: 0001-ob-core-async-Add-faces-1-5.patch

0002-ob-core-async-Add-org-babel-async-tools-2-5.patch
Description: 0002-ob-core-async-Add-org-babel-async-tools-2-5.patch

0003-ob-core-async-Refactor-handle-result-3-5.patch
Description: 0003-ob-core-async-Refactor-handle-result-3-5.patch

0004-ob-core-async-Handle-nasync-param-4-5.patch
Description: 0004-ob-core-async-Handle-nasync-param-4-5.patch

0005-ob-core-async-Add-execute-with-5-5.patch
Description: 0005-ob-core-async-Add-execute-with-5-5.patch

0006-lisp-org-elib-async.el-New-package-about-async-helpe.patch
Description: 0006-lisp-org-elib-async.el-New-package-about-async-helpe.patch

0007-lisp-ob-core.el-Notify-when-execution-fails.patch
Description: 0007-lisp-ob-core.el-Notify-when-execution-fails.patch

0008-scratch-bba-ob-core-async-Some-temporary-test-files.patch
Description: 0008-scratch-bba-ob-core-async-Some-temporary-test-files.patch