Re: [Fab-user] tricky parallel execution

[Dmitry Makovey]

On September 10, 2012 12:03:45 Jeff Forcier wrote:
> On Mon, Sep 10, 2012 at 7:41 AM, Dmitry Makovey <address@hidden> 
wrote:
> > in case of postmaster - that is probably what I'll end up doing. However
> > before commiting to that I decided to play with os.fork():
> > 
> > <snip>
> > 
> > which seems to work but something tells me this approach may bite me in
> > the
> > end if Fabric keeps track of connections by "reference counts" and does
> > cleanup on SIGTERM - i.e. I'm destroying child and inadvertently
> > terminating connection with the server
> > 
> > So the question is: is above code really working or should I stay away
> > from
> > doing forks within tasks?
> 
> Re: handling kills/disconnects, Fabric keeps a module level dict
> holding the SSH connections and closes them at process exit
> (normally/imperatively, not via signal trapping).
> 
> So theoretically fork() will copy those objects for the child (if I
> remember how it works correctly) but the sockets they represent would
> be the same objects as the OS level -- so you'd get odd behavior or
> "Socket closed" errors if the child closed a connection the parent was
> using. (We got this in the existing parallel code until we cleared out
> that connection dict at the top of the child processes so they'd make
> their own, new sockets.)
> 
> That shouldn't matter a lot though, because run("command &") will
> actually sit there and block forever (see
> http://docs.fabfile.org/en/1.4.3/faq.html#why-can-t-i-run-programs-in-the-ba
> ckground-with-it-makes-fabric-hang), so your sleep & kill calls should never
> actually get executed. Not 100% sure why it appeared to work for you.
> 
> I would definitely just try using a non-& backgrounding method on the
> remote end, it'll be saner & quicker :)

Thank you Jeff for your helpful comments.

With above semi-confirmation I think I'll stay away from forking in-Fabric and 
will attempt handling things on remote end (as in start processes in 
background, noting PID, etc. etc. etc.) bit more scripting to push over to 
remote end, but I guess you stability/sanity doesn't come cheap ;)



> 
> -Jeff
-- 
Dmitry Makovey
Web Systems Administrator
Athabasca University
(780) 675-6245
---
Confidence is what you have before you understand the problem
    Woody Allen

When in trouble when in doubt run in circles scream and shout 
     http://www.wordwizard.com/phpbb3/viewtopic.php?f=16&t=19330


-- 
    This communication is intended for the use of the recipient to whom it
    is addressed, and may contain confidential, personal, and or privileged
    information. Please contact us immediately if you are not the intended
    recipient of this communication, and do not copy, distribute, or take
    action relying on it. Any communications received in error, or
    subsequent reply, should be deleted or destroyed.
---