Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])

[Matt]

 ---- On Fri, 16 Feb 2024 18:52:22 +0100  Bruno Barbier 

 > Sorry for the late reply.  Cleaning the code took me longer than expected.

No need to apologize, we're all volunteers here :)

 > Feedbacks, corrections, critiques, etc are most welcome!

Thank you for sharing!

If I understand correctly, there are several independent topics the code 
addresses:

| topic            | manner addressed                               |
|------------------+------------------------------------------------|
| execution status | using overlays to communicate execution status |
| locating results | using overlays to locate results               |
| blocking         | making all execution asynchronous              |
|------------------+------------------------------------------------|

I suggest these be discussed in separate threads.

 > > The use of the overlay is a really cool idea!
 > >
 > > I hesitate to say that's a good way to convey success or failure.  If a 
 > > process failed, I want to see the output which tells me why so that I can 
 > > correct it.  Or, I might actually want the failure output.  Maybe I want 
 > > to literally demonstrate what a code failure looks like.  Maybe I want to 
 > > use that output in another block.  For example, shell blocks have multiple 
 > > output types.  A shell process may return standard output/error or a 
 > > failure code.  The result of the failure may trigger something else.
 > 
 > I'm not sure I fully understand what you mean. The API just assumes the 
 > backend returns the outcome: either success or failure, where failure means 
 > "no result" (the previous result, if it exists, is even preserved in the 
 > document).  The backend is free to transform a failure into a success to 
 > make that result available though.

You can disregard my hesitation on this point.  I had not run your code yet and 
had misunderstood how it worked.

Since this thread is dedicated to blocking, let me share my thoughts on that 
subject.

 > To execute Python blocks, using the proposed async API:
 > 
 >    - I've (re)implemented the "asynchronous with session" case 
 > (copying/pasting the relevant part from ob-python).
 >    
 >    - The "synchronous case" is just artificially blocking the user until the 
 > asynchronous result is known (which looks incredibly tricky to implement if 
 > even possible...).
 >    
 >    - The "no session" case is just about creating a new unique session and 
 > throwing it away immediately.

This is an interesting idea, feeding all processes through the same mechanism.

Executing a shell block requires starting a 
[[https://www.gnu.org/software/emacs/manual/html_node/elisp/Processes.html][process]].

Processes are synchronous or asynchronous.

Three primitives exist in Emacs for making processes:

1. make-process (asynchronous)
2. call-process (synchronous)
3. call-process-region (synchronous)

There exist several convenience wrappers for these.  AFAIK, everything reduces 
to these three primitives.  For example, =async-shell-command= runs 
=call-process= and prevents blocking by appending a "&" to the command which 
tells the shell to run the command in the background and return control to the 
terminal.  This background-foreground distinction is called "job control".

Output from a process typically goes to a buffer.  This may be changed and 
instead handle output with a filter function.  =call-process= has an option to 
directly send output to a file.

Subprocesses inherent the =default-directory= and the environment from Emacs.  
The environment may be changed using =process-environment=.

There are two types of asynchronous connections: "pty" ("pseudoterminal") and 
"pipe".  The main difference is that "pty" provides a terminal-like connection 
which allows for things like job control (=C-c=, =C-z=, etc.).

In my previous message, I divided evaluation into 4 types:

  - non-persistent vs. persistent 
  - synchronous vs. asynchronous

I find the approach of feeding everything through, fundamentally, 
=make-process= interesting because if we make a chart of the 4 types, we see 
some ambiguities:

|              | non-persistent | persistent   |
|--------------+----------------+--------------|
| synchronous  | call-process   | ???          |
|--------------+----------------+--------------|
| asynchronous | ???            | make-process |
|--------------+----------------+--------------|

To make a non-persistent asynchronous process, the first thing that comes to 
mind is =async-shell-command=.  However, as the code shows, another option is 
to use =make-process= and throw away the state (the process buffer).

I'm not sure how we could make a persistent, synchronous process.  Persistence 
is achieved, currently, by a process buffer.  Is there another way persistence 
may be achieved?  Of course, this ignores whether a persistent, synchronous 
process is even desirable.  Given reliable asynchronous execution with 
persistence, I can't think of reason why someone would prefer a blocking 
operation.

All that is mainly academic.  The idea I think most interesting is using a 
single primitive to handle all evaluation.

It idea reminded me of exploration code I wrote a while back which uses 
=make-process= to run all code blocks asynchronously (attached).

It works as follows.  I defined a new Babel "language" called "blub".   Blub 
could be a shell, python, ruby, whatever.  I wanted to test that the 
implementation could work with different interpreters or compilers.  Note that 
"blub" doesn't have any relationship to Paul Graham's blub; I just needed a 
name for a generic language that could be swapped out.

Attached are two files, ob-blub.el and ob-blub-test.org.  Download both to the 
same directory.  Run the first block in ob-blub-test.org.  This imports 
ob-blub, loads it into Babel, and sets up blub to be whatever =shell-file-name= 
is (for example, bash).  If you want to try Python or Ruby, comment out the 
shell configuration, uncomment the Python or Ruby implementations, and evaluate 
the block again.  Hopefully ob-blub.el is documented sufficiently for you to 
experiment.

The blub implementation has the same shortcomings, at least for shells, as the 
current shell implementation.  It has a few ideas, such as everything being 
asynchronous and completely removing the prompt, that may prove useful for 
improving Babel generally.  The blub implementation is also simpler than 
related parts of Babel and may be useful for figuring out ways to solve the 
currently known shortcomings.  If you run into an error during execution, you 
will need to call (setq my-org-babel-comint--async-uuid nil).

The challenge I've found with Babel is figuring out how to make the changes.  
My current approach is to address bugs and to make changes that move us toward 
something like the ob-blub implementation.  I wonder if it might help to 
discuss the core ideas and use a minimal reference implementation that serves 
as a guide for the actual changes we make.

Curious to hear other people's thoughts!

--
Matt Trzcinski
Emacs Org contributor (ob-shell)
Learn more about Org mode at https://orgmode.org
Support Org development at https://liberapay.com/org-mode

ob-blub-test.org
Description: Binary data

ob-blub.el
Description: Binary data