RFC: A framework for task management and execution in Emacs

[Jan Moringen]

Hi,

I would like to start a discussion regarding a topic that has bothered
me for some time now: management and execution of (long-running) tasks
in Emacs. Instead of properly explaining what I mean by "task", I will
just paste a table from my notes that has several examples:

| Component    | Execution          | Progress Indication | Cancelable
|--------------+--------------------+---------------------+-----------
| Tramp        | synchronous        | progress reporter   | no?
| VC           | sync, async        | mode-line           | sometimes
| Gnus         | synchronous        | ?                   | ?
| Compilation  | asynchronous       | mode-line           | yes
| URL          | synchronous, async | progress reporter   | ?
| Emacs Jabber | timers, fsm        | ?                   | ?
| SemanticDB   | idle-timers        | custom reporter     | on input

Each of these packages performs some kinds of potentially long running
operations. Originally, the table was intended to illustrate the
heterogeneous nature of solutions despite similar requirements and
use-cases.

The (hopefully) upcoming thread support will probably make the situation
better and worse. Better, because long-running operations could "simply"
run in their own threads, worse because actually implementing this is
usually not that simple. The number and complexity of solutions for task
execution would probably grow further once threads become available.

My proposal is to introduce a set of mechanisms that make task execution
more uniform for the user and easier for programmers. Here is the full
set of requirements/aspects I could come up with:

+ Scheduling/Execution
  + Resource Allocation (threads)
  + Synchronization (order of execution, retrieval of results)
  + Canceling
+ User Interface
  + Graphical Indicators
  + Progress/remaining Time Estimation
  + Error Reporting
+ Desktop Integration
  + Power Management inhibition
  + Desktop-wide Task Management
    (Example:
http://ssickert.wordpress.com/2010/05/09/introducing-my-gsoc project/)
+ Customization
+ Compatibility
  + Portability
  + Backwards Compatibility

Since there is so much potential for code duplication, reinventing the
wheel and divergent user interfaces, I think all of these issue should
be addressed in a general way.

Other Environments such as Eclipse or the Evolution mail client seem to
employ such generic mechanisms since their user interfaces contain lists
of currently running tasks, which can also be interacted with.

At this point, my general question is whether there are any plans or
related efforts regarding this topic. Of course, I would also like to
know whether people consider the approach worthwhile at all :)

The second part of this message is about a small prototype framework I
made to explore the issue practically.

The framework is structured like this:

Interface Layer
+--------------------------------+-----------------------------------+
|                                |                                   |
|         User Interface         |               Macros              |
|                                |                                   |
+--------------------------------+-----------------------------------+
Backend Layer
+------------------------------+-------------------------------------+
|                              |                                     |
|              Tasks           +---+            Execution            |
| +-----------+     +-----------+  | +-----------+     +-----------+ |
| | Strategy1 | ... | StrategyN |  | | Strategy1 | ... | StrategyN | |
| +-----------+     +-----------+  | +-----------+     +-----------+ |
+----------------------------------+---------------------------------+

These components are discussed below:

*Macros*

Basically just two macros with identical syntax:

({do,run}-as-task NAME (TASK-SPEC) (EXECUTION-SPEC)
  [DOCSTRING]
  BODY)

The difference being that the do- macro returns the result of BODY when
it is ready while the run- variant immediately returns an object from
which the result of BODY can be retrieved later.

Examples:

(do-as-task add-numbers () ()
  "Add some numbers."
  (dolist (i (number-sequence 1 1000))
    (+ 1 2 3)
    'my-result))

+ Blocks immediately
+ Body may or may not start executing immediately
+ Returns when the body finishes
+ Returns the value returned by the body

(let ((delayed (run-as-task add-numbers () ()
  "Add some numbers."
  (dolist (i (number-sequence 1 1000))
    (+ 1 2 3)
      'my-result))))
  ;; do other stuff
  (future-get delayed))

+ Does not block
+ Body may or may not start executing immediately
+ Returns immediately
+ Returns an object that implements a "future" protocol
+ Result of body can be retrieved from returned object

*Tasks Strategies*

>From the commentary section:

;; What a task is
;; + What gets executed
;; + Execution context (functions callable from task code)
;; + Meta information (name, description)
;; What a task is not
;; + how to execute (synchronous vs. asynchronous)
;; + when to execute (delayed, lazy etc)

(do-as-task add-numbers (progress) ()
  "Add some numbers, reporting progress."
  (dolist (i (number-sequence 1 1000))
    (+ 1 2 3)
    (progress i))
  'my-result)

(do-as-task add-numbers (cancelable) ()
  "Add some numbers, cancelable."
   (dolist (i (number-sequence 1 1000))
     (+ 1 2 3)
     (maybe-cancel))
   'my-result)

(do-as-task add-numbers (progress cancelable) ()
  "Add some numbers, reporting progress and cancelable."
   (dolist (i (number-sequence 1 1000))
     (+ 1 2 3)
     (progress i)
     (maybe-cancel))
   'my-result)

*Execution Strategies*

These control how and when a task is executed. The currently available
executions are (:thread is just a teaser, of course):

(run-as-task add-numbers () (:execution blocking)

(run-as-task add-numbers () (:execution idle)

(run-as-task add-numbers () (:execution (timeout :delay 5))

(run-as-task add-numbers () (:execution thread))

*User Interface*

There is also user interface code, but discussing it here would probably
provide little insight.

The code of the framework described above is available at

  http://bazaar.launchpad.net/~scymtym/etasks/trunk/

Feedback and suggestions would be greatly appreciated.

Kind regards,
Jan