Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])

[Bruno Barbier]

Hi Matt,

Thanks for your answer.

Matt <matt@excalamus.com> writes:

> ...
> If I understand correctly, there are several independent topics the code 
> addresses:
>
> | topic            | manner addressed                               |
> |------------------+------------------------------------------------|
> | execution status | using overlays to communicate execution status |
> | locating results | using overlays to locate results               |
> | blocking         | making all execution asynchronous              |
> |------------------+------------------------------------------------|
>
> I suggest these be discussed in separate threads.

Actually my patch is only about reporting execution status and
inserting results, in a generic way, in ob-core.  I provided some demo
code on top of them, just to show how it could be used.

Sorry about the confusion.  

> ...
>
> Since this thread is dedicated to blocking, let me share my thoughts on that 
> subject.

I guess I should start a new thread then :)


> ...
>
>
> Executing a shell block requires starting a 
> [[https://www.gnu.org/software/emacs/manual/html_node/elisp/Processes.html][process]].
>
> Processes are synchronous or asynchronous.
>
> Three primitives exist in Emacs for making processes:
>
> 1. make-process (asynchronous)
> 2. call-process (synchronous)
> 3. call-process-region (synchronous)
>
> There exist several convenience wrappers for these.  AFAIK, everything 
> reduces to these three primitives.  For example, =async-shell-command= runs 
> =call-process= and prevents blocking by appending a "&" to the command which 
> tells the shell to run the command in the background and return control to 
> the terminal.  This background-foreground distinction is called "job control".
>

> Output from a process typically goes to a buffer.  This may be changed and 
> instead handle output with a filter function.  =call-process= has an option 
> to directly send output to a file.
>

I've reached the same conclusion. As I wanted one simple
implementation, I was thinking about using 'make-process', but, as we
have to deal with the output manually and use a process sentinel, and,
as offering a REPL to the user is nice, my personal pick is to use a
plain shell with comint.

> Subprocesses inherent the =default-directory= and the environment from Emacs. 
>  The environment may be changed using =process-environment=.
>

One wonderful thing about using =default-directory= is that, if done
right, tramp will automatically handle executions on one or more remote
computers.

> There are two types of asynchronous connections: "pty" ("pseudoterminal") and 
> "pipe".  The main difference is that "pty" provides a terminal-like 
> connection which allows for things like job control (=C-c=, =C-z=, etc.).

I'm personally ignoring the difference between pty and pipe: as I
don't use the terminal features, it doesn't really matter to me.


> ...
> I'm not sure how we could make a persistent, synchronous process.  
> Persistence is achieved, currently, by a process buffer.  Is there another 
> way persistence may be achieved?

If I understand correcly, "persistent" here means to preserve some
state between executions.  They are definitely other way to preserve
and provide a state.

If you focus on processes only, it's probably OK. But, if you use
threads or some other kinds of asynchronicity, that might be too
limiting to require a buffer and a process.


> Of course, this ignores whether a persistent, synchronous process is even 
> desirable.  Given reliable asynchronous execution with persistence, I can't 
> think of reason why someone would prefer a blocking operation.
>

I'm not so sure.  If the success of an execution is really important,
and, intrinsically unreliable (like a network connection), and, it
only takes a few seconds, I'll probably choose synchronous execution,
just to see that everything is OK, before moving to my next task.



> ...
> Attached are two files, ob-blub.el and ob-blub-test.org.  Download both to 
> the same directory.  Run the first block in ob-blub-test.org.  This imports 
> ob-blub, loads it into Babel, and sets up blub to be whatever 
> =shell-file-name= is (for example, bash).  If you want to try Python or Ruby, 
> comment out the shell configuration, uncomment the Python or Ruby 
> implementations, and evaluate the block again.  Hopefully ob-blub.el is 
> documented sufficiently for you to experiment.

Thanks for sharing.  I didn't have time to try it yet.  But it looks
like you could use the 'execute-with' keyword that my patchs provide.
That way, you may redirect evaluation to use your "blub"
implementation, and still use the real language name for your code
blocks (that way, org knows how to fontify them, edit them, etc).  And, you
will not have to manually modify and reevaluate your elisp when
switching languages.

> The blub implementation has the same shortcomings, at least for shells, as 
> the current shell implementation.  It has a few ideas, such as everything 
> being asynchronous and completely removing the prompt, that may prove useful 
> for improving Babel generally.  The blub implementation is also simpler than 
> related parts of Babel and may be useful for figuring out ways to solve the 
> currently known shortcomings.  If you run into an error during execution, you 
> will need to call (setq my-org-babel-comint--async-uuid nil).

I'll try it. Thanks,


> The challenge I've found with Babel is figuring out how to make the changes.  
> My current approach is to address bugs and to make changes that move us 
> toward something like the ob-blub implementation.  I wonder if it might help 
> to discuss the core ideas and use a minimal reference implementation that 
> serves as a guide for the actual changes we make.
>
> Curious to hear other people's thoughts!

One think that comes to mind is to check that our testsuite covers
everything we might break.

If 'execute-with' gets added to org (from my patchs), that may provide
an easy way to test/compare several execution engines.  I'm thinking
about modifying my patch to provide a 'feedbacks-with' keyword so that
we can also test/compare different ways to report the execution status
and outcome.

Bruno