Re: [Fab-user] Calling fabric from within multithreaded program

[Jeff Forcier]

Hi John,

On Mon, Sep 24, 2012 at 8:39 AM, John Kelly <address@hidden> wrote:

>This works fine in the non-threaded version of the
> script, but in the multi-threaded version, it looks as if the threads
> aren't keeping up and are stomping on each other.
>
> <snip>
>
> What we're seeing is that, for the most part, the mkfs.xfs works fine on
> all machines, but the mkdir and mount commands are often run on the
> wrong machine.

Unfortunately, Fabric 1.x is not threadsafe, it makes heavy use of
global module variables for state :( That's what you're seeing -- e.g.
fabric.state.env.host_string determines the current connection
context, so multiple threads will be updating/reading that and causing
each other to connect to the wrong host.

Fab's own concurrency model uses the multiprocessing library to work
around this -- see
http://docs.fabfile.org/en/1.4.3/usage/parallel.html for details.
Depending on exactly what your parent codebase does, this is probably
the way to go.

(FWIW, Fab 2.x will take pains to avoid this limited architecture, but
that's a little ways off yet.)

Best,
Jeff

-- 
Jeff Forcier
Unix sysadmin; Python/Ruby engineer
http://bitprophet.org