Re: [Fab-user] tricky parallel execution

[Jeff Forcier]

On Mon, Sep 10, 2012 at 7:41 AM, Dmitry Makovey <address@hidden> wrote:

> in case of postmaster - that is probably what I'll end up doing. However
> before commiting to that I decided to play with os.fork():
>
> <snip>
>
> which seems to work but something tells me this approach may bite me in the
> end if Fabric keeps track of connections by "reference counts" and does
> cleanup on SIGTERM - i.e. I'm destroying child and inadvertently terminating
> connection with the server
>
> So the question is: is above code really working or should I stay away from
> doing forks within tasks?

Re: handling kills/disconnects, Fabric keeps a module level dict
holding the SSH connections and closes them at process exit
(normally/imperatively, not via signal trapping).

So theoretically fork() will copy those objects for the child (if I
remember how it works correctly) but the sockets they represent would
be the same objects as the OS level -- so you'd get odd behavior or
"Socket closed" errors if the child closed a connection the parent was
using. (We got this in the existing parallel code until we cleared out
that connection dict at the top of the child processes so they'd make
their own, new sockets.)

That shouldn't matter a lot though, because run("command &") will
actually sit there and block forever (see
http://docs.fabfile.org/en/1.4.3/faq.html#why-can-t-i-run-programs-in-the-background-with-it-makes-fabric-hang),
so your sleep & kill calls should never actually get executed. Not
100% sure why it appeared to work for you.

I would definitely just try using a non-& backgrounding method on the
remote end, it'll be saner & quicker :)

-Jeff

-- 
Jeff Forcier
Unix sysadmin; Python/Ruby engineer
http://bitprophet.org