Re: parallel hanging when used with pipe

[Ole Tange]

Hyperthreading is too far away from GNU Parallel to be the cause. It
would surprise me very much.

I find it much more likely that it is some local configuration
(environment variable, .rc-setting, shell weirdness, other jobs
running on the system) that is the cause.

If you create a new clean user on the same system, do you see the
problem running as the new user?

Can you reproduce the problem running other programs under GNU
Parallel or other data?


/Ole

On Tue, Apr 9, 2013 at 11:35 PM,  <juncus@gmail.com> wrote:
> Thank for the response. The error did not occur on the virtualbox
> image, or on other servers I have at my disposal.  But the hang is
> reproducible on this one machine, for this dataset.  The server in
> question has hyperthreading enabled in the bios.  Could this have
> something to do with it?  Maybe hyperthreading doesn't play nice with
> parallel, in certain situations?
>
> I want to try it without hyperthreading, I just need to get into the
> server room so I can reboot and get into the bios settings.
>
> Thanks again
>
>
> On Sat, Mar 30, 2013 at 3:20 AM, Ole Tange <ole@tange.dk> wrote:
>> That very much looks like a bug caused by a race condition.
>>
>> Unfortunately I need it more reproducible to be able to fix it.
>>
>> So I would like to ask you to try to reproduce it on a virtual machine
>> from: http://sourceforge.net/projects/virtualboximage/files/ using
>> data that you are allowed to give me.
>>
>>
>> /Ole
>>
>>
>> On 3/30/13, juncus@gmail.com <juncus@gmail.com> wrote:
>>> I was using parallel to do blast searches on a large number of
>>> biological sequences.
>>>
>>> cat sequences.fasta | parallel --block 2k --recstart '>' --pipe -j 30
>>> "blastn -db nt -task blastn -evalue 0.00001 \
>>>   -outfmt '6 qseqid sseqid pident length mismatch gapopen qstart qend
>>> sstart send evalue bitscore ppos' \
>>>   -max_target_seqs 5 -query -" > results.txt
>>>
>>> It hung after a few hours.  Only about 40% of the input file had been
>>> processed.  No blastn processes were running, and the parallel process
>>> was in sleep (S) state.
>>>
>>> I killed the parallel process and got this message:
>>>
>>> parallel: SIGTERM received. No new jobs will be started.
>>> parallel: Waiting for these 0 jobs to finish. Send SIGTERM again to stop
>>> now.
>>>
>>>
>>> This message seems somehow pathological.  Parallel is waiting for zero
>>> jobs to finish.  This must surely be connected with why it was hanging
>>> (it didn't get the message that jobs were done, so it didn't spawn new
>>> jobs.)
>>>
>>> This is not happening all the time, so difficult for me to diagnose.  Any
>>> clues?
>>>
>>> Thanks very much,
>>> Owen
>>>
>>>
>>> $ parallel --version
>>> GNU parallel 20130222
>>> Copyright (C) 2007,2008,2009,2010,2011,2012,2013 Ole Tange and Free
>>> Software Foundation, Inc.
>>> License GPLv3+: GNU GPL version 3 or later
>>> <http://gnu.org/licenses/gpl.html>
>>> This is free software: you are free to change and redistribute it.
>>> GNU parallel comes with no warranty.
>>>
>>> Web site: http://www.gnu.org/software/parallel
>>>
>>> When using GNU Parallel for a publication please cite:
>>>
>>> O. Tange (2011): GNU Parallel - The Command-Line Power Tool,
>>> ;login: The USENIX Magazine, February 2011:42-47.
>>>
>>>