Re: Processing large amounts of files

[Ricardo Wurmus]

Hi Liliana,

[-guix-devel@gnu.org, +gwl-devel@gnu.org]

(forgot to actually send this message a few days ago)

thanks for the report!

> I have a somewhat unusual workflow that requires me to do a number of
> processes on numerous, but small input files.  The original is a bit
> unwieldy and takes several minutes to compile, but I've managed to
> produce a more understandable and better performing example.  Note,
> that after a certain number of inputs, I get the following error:
>
> info: .16 Loading workflow file `meow.gwl'...
> info: 2.80 Computing workflow `cat'...
> run: 12.96 Executing: /bin/sh -c 
> /gnu/store/kmssbjcdcabg9fh4nxscwwpnlb4px30h-gwl-meow.scm …
> error: 13.01 Wrong type argument in position 1: #f

It is frustrating that there is no backtrace.  Reliable error handling
is hard.

When running with "-l all" I see this:

  info: .75 Computing workflow `cat'...
  debug: 3.13 Computing script for process `meow'
  guix: 3.13 Looking up package `bash-minimal'
  guix: 3.13 Opening inferior Guix at 
`/gnu/store/pb1nkrn3sg6a1j6c4r5j2ahygkf4vkv9-profile'
  guix: 4.27 Looking up package `guix'
  debug: 4.45 Generating all scripts and their dependencies.
  debug: 4.89 Generating all scripts and their dependencies.
  run: 6.73 Executing: /bin/sh -c 
/gnu/store/5idhbvhrwj3p53kkz2vikdn1ypncwj84-gwl-meow.scm '((inputs 
"/tmp/meow/0" ...
  process: 8.80 In execvp of /bin/sh: Argument list too long
  error: 8.80 Wrong type argument in position 1: #f

This at least tells us that the last error here is due to sh refusing to run.

> For comparison:
>   time cat /tmp/meow/{0..7769}
>   […]
>   
>   real        0m0,144s
>   user        0m0,049s
>   sys 0m0,094s
>
> It takes GWL 6 times longer to compute the workflow than to create the
> inputs in Guile, and 600 times longer than to actually execute the
> shell command.  I think there is room for improvement :)

Yeah, not good.  Do you have any recommendations?

-- 
Ricardo