Re: Workflow management with GNU Guix

[Roel Janssen]

Ludovic Courtès writes:

> Roel Janssen <address@hidden> skribis:
>
>> I realize I never shared my proof-of-concept implementation.  I attached
>> my motivations for having a workflow language in Guix, and my code.
>
> Nice work, thanks for sharing!
>
>> The subcommand "guix workflow" does not work (yet) here.  I currently
>> execute a workflow directly from the REPL.
>>
>> A final point to note is that I would like to do a second attempt at
>> designing the workflow language, changing the way we can execute
>> programs.
>
> IIUC, (guix workflows) from the tarball you sent executes workflows in
> the current environment, as opposed to creating a derivation that would
> actually perform the workflow.  What motivated this approach?

The short answer:
Lack of time to implement it properly ;).

The slightly longer answer:
I want to avoid storing results in the store, because we could be
analyzing files of 100GB or more that we do not want to copy into the
store, neither do we want to store the results of the run in the store.

I now realize we could only put the derivation in the store, and not the
build output itself..

> Workflows could compiled to derivations, which in turn could be “built”,
> and their build result would be the workflow’s output file.
>
> I guess in practice it only works if users of the cluster can build
> derivations on the cluster and have them scheduled on compute nodes.
>
> Thoughts?

For building derivations, I think we need super user privileges, right?
Why can't the scripts "just" output the environment variables required as
@code{guix package --search-paths} provides, and then run the commands
with the newly set environment?

Kind regards,
Roel Janssen