Re: Comments on process template syntax

[Ricardo Wurmus]

Hi Roel,

> On Sun, 2020-02-02 at 11:20 +0100, Ricardo Wurmus wrote:
>> Hi,
>>
>> While looking at the examples at
>> https://www.guixwl.org/beyond-started,
>> I found that process templates might be difficult to understand, and
>> that they have no pretty syntax.
>>
>> --8<---------------cut here---------------start------------->8---
>> process: (list-file-template filename)
>>   name
>>     string-append "list-file-"
>>                   basename filename
>>   packages "gzip"
>>   inputs filename
>>   outputs
>>     string-append filename ".list"
>>   run-time
>>     complexity
>>       space 20 mebibytes
>>       time  30 seconds
>>   # { gzip --list {{inputs}} > {{outputs}} }
>> --8<---------------cut here---------------end--------------->8---
>>
>> The first line is easy to understand for lispers but it might look
>> weird
>> to people who come from other workflow languages or programming
>> languages.  This describes a procedure called “list-file-template”
>> that
>> returns a process parameterized on the argument “filename”.
>>
>> Nextflow has no concept of procedures that produce processes when
>> given
>> arguments.  It does however have a concept of data streams that can
>> be
>> fed into processes, which results in a process to be instantiated for
>> every element of the stream.  The stream may be created from a
>> directory
>> containing files.
>>
>> This implementation likely stems from the realization that the
>> “template
>> case” is the most common case for processes.  Rarely ever is it
>> necessary to define a process that does *not* require
>> parameterization
>> on its inputs.
>>
>> Can we make the common case simpler and easier to understand?
>
> Perhaps with some parentheses?  That it is a Lisp is a good thing, not
> something you'd rather hide.. :)  Like you've said; what you've defined
> above is a procedure, not a record.  That's a really cool "feature" of
> the GWL!

I agree.  I wouldn’t want to hide it e.g. by somehow “inferring”
procedure inputs via the inputs field.  I think it’s good that it
closely resembles a procedure, because that’s what it is.

I still think that the syntax is sub-optimal.  We support Wisp to
make the Lispiness a little easier to swallow for the skeptics.  But
the procedure case does not benefit much from Wisp — it would look worse
if we expressed it in the Wisp way:

    process : list-file-template filename

Note the space between “process” and the remainder.  It would be wrong
to remove the space after “process”.  That’s a pitfall stemming from a
familiarity with YAML that I’d rather avoid.  (That’s why I want to
rename “process:” and “workflow:”.)

The only reason why I know how to use “:” is because I know that I want
the remainder to be wrapped in parentheses…  People who only know the
sugary syntax would not have that knowledge and it would just seem like
an arbitrary thing.

The confusion disappears in my opinion when the colon does not follow
the first word.

    process list-file-template : filename
      name …
      inputs …
      outputs …

or when the colon is avoided altogether:

    process list-file-template (filename)
      name …
      inputs …
      outputs …

or in Lispy syntax

    (process list-file-template (filename)
      (name …)
      (inputs …)
      (outputs …))

Having a list of identifiers after the name of the procedure matches
Common Lisp and C-style languages.  I think it looks less confusing as
the difference between the “template case” and the “record case” becomes
merely a parenthetical list of free variables.

What do you think?  This can be accomplished with a tiny change to the
“process” macro in (gwl sugar).

> Maybe we can just clarify the feature better in the documentation.
> Here's my initial thought:
> ---
> When defining processes, they can be parameterized by turning the
> process definition into a procedure, which will form a template for
> processes to be defined later.  This is done by adding a name for the
> template, and its parameters directly after "process:".
> ---

I agree that emphasizing this in the manual is a good idea.

--
Ricardo