Re: Parallel concatenate set of files

[Ole Tange]

I have seen people online doing it this way :-)

function local_cat()
{
  cat $1 $2 > $3
}

export -f local_cat

for ((i=0,i<=99,i+=4));do
 f1in=("${f1in[@]}" "${filelist1[*]:$i:4}")
 f2in=("${f2in[@]}" "${filelist2[*]:$i:4}")
 fout=("${fout[@]}" "${filelist3[@]:$i}")
done

parallel --xapply local_cat {1} {2} {3} ::: "${f1in[@]}" :::
"${f2in[@]}" ::: "${fout[@]}"

(Some people find Perl looks like line noise. I am not sure Bash is
any better - based on the above).

/Ole

On Tue, Apr 29, 2014 at 2:55 PM, Kostas Politis <kgpolitis@gmail.com> wrote:
> Hello All!
>
> I am trying to concatenate a set of files to another set of files. To be
> more specific I want to parallelize the following loop:
>
> for ((i=0,i<=99,i+=4));do
>  cat ${filelist1[@]:$i:4} ${filelist2[@]:$i:4} > ${filelist3[@]:$i}
> done
>
> So I decided to generate the list of files:
>
> for ((i=0,i<=99,i+=4));do
>  f1in=("${f1in[@]}" "`echo ${filelist1[@]:$i:4}`")
>  f2in=("${f2in[@]}" "`echo ${filelist2[@]:$i:4}`")
>  fout=("${fout[@]}" "${filelist3[@]:$i}")
> done
>
> and pass these as arguments to parallel:
>
> parallel --xapply cat {1} {2} ">" {3} ::: "f1in[@]" ::: "f2in[@]" :::
> "fout[@]"
>
> However the above generates an error since the first element of f1in
> contains blanks that are consider as characters by cat and as a result cat
> cannot find the files `echo ${filelist1[@]:$i:4}`. On the contrary a simple
> cat works as expected:
>
> cat f1in[0] f2in[0] > fout[0]
>
> There is something I'm missing here. Could you please help me or provide a
> workaround for what I'm trying to do? Is this a valid use of parallel?
>
> Thank you in advance.
>
> Kostas