Re: Problem with thousands of small jobs.

[Ole Tange]

On Sat, Feb 7, 2015 at 10:11 AM, xmoon 2000 <xmoon2000@googlemail.com> wrote:
> On 6 February 2015 at 20:42, Ole Tange <ole@tange.dk> wrote:
>> On Fri, Feb 6, 2015 at 3:56 PM, xmoon 2000 <xmoon2000@googlemail.com> wrote:
>>
>> My experience with Cygwin is limited, but I seem to remember that
>> spawning is expensive (in the order of 0.1 second), which is one of
>> the reasons for recommending MSYS. So if your jobs all finish at the
>> same time, GNU Parallel will need 3.2 seconds to start 32 new jobs.
>> Maybe you will have a better experience if you delay starting jobs by
>> 0.1 seconds:
>>
>>   cat /tmp/parList | parallel -j 28 --delay 0.1 --eta;
>>
>> What is your reason for not using all 32 cores?
>
> 1. Reason for not using 32 cores is then I can't do anything else. I
> need to do emails, check web, simple stuff whilst waiting for jobs to
> finish.

I tested this on a 4 core Microsoft Windows 8 machine and it is not
true there: The machine will be slower, but not useless. On a 32 core
machine I would expect this to be even less of a problem.

On GNU/Linux you would use --nice which would mean that all other
processes would get more CPU time than the jobs. In practice the jobs
will only use the spare CPU time. I have not tested if --nice gives
the same behaviour on Microsoft Windows.

If you simply assumed your machine would be slower, then I will
encourage you to try it out. You might be pleasantly surprised.

If it is possible for you to use MSYS instead of Cygwin it would be
interesting to see if that makes the problem go away.


/Ole