Re: [Fab-user] How do I combine "local" with "remote" tasks in fabric 2?

[Michel Albert]

It's a bit unfortunate that both objects have different APIs. I also don't see an easy way to improve this other than slapping on some methods on both so they become interchangeable. But that seems really messy and is probably not with the effort.

And an invoke task with a "local" method on it is even more confusing.

On Tue, 2 Apr 2019, 21:11 Brandon Whaley, <address@hidden> wrote:

I think I see what the biggest point of confusion here is.  When fab
executes a task but no -H parameters are set and the decorator
specifies no host, the Connection object is instead an invoke Context
object, which has no .local.  I agree that that's not intuitive, but
I'm not sure where that choice is made in the fabric codebase.

On Tue, Apr 2, 2019 at 2:50 PM Michel Albert <address@hidden> wrote:
>
> Creating a new Context from the connection config did indeed work. So I am planning - for now - to go ahead with this.
>
> I hear you on the point of separating invoke from fabric tasks.
>
> To elaborate, and please don't read the following as a kind of rant, just as an elaboration where the need to mix remote & local tasks comes from.
>
> My current aim is to migrate an old, rather large and complex collection of fabric-1.x tasks to fabric-2.x And many of those tasks mix both remote and local commands. My aim is to keep everything available under the "umbrella" "fab" executable. Many tasks are shared among an ever growing team via a library made available to them. I really want to avoid having two executables where it may not be clear from the outside which task needs to be executed using invoke and which one needs fabric. The end-goal is to have a task runner which has the same core tasks for all of our projects. Namely:
>
> develop - to create a new development environment
> build - To build and bundle whatever is necessary for deployment
> test - To run unit-tests
> deploy - To deploy to a targeted environment
>
> We chose fabric in the past because it is really flexible, and lets us use it even for non-Python projects. And most of all it is extensible. So most projects have more than these 4 core tasks. Not being able to determine whether to use "inv" or "fab" would quickly become quite annoying.
>
> The library has grown over the past and it helped us a lot in our day-to-day business. Splitting the tasks now into an invoke- and fabric-library is definitely something I will take a closer look at, but I am afraid that some of the more complex tasks might lose in readability when doing so. I will see.
>
> In any case, your feedback helped me a lot. I wonder if this would be a useful example to put into the docs (that is,calling an invoke task from a fabric task). It does not seem to be something very common, but I had a lot of trouble finding a solution without falling back to this mailing list.
>
> On Tue, Apr 2, 2019 at 5:35 PM Brandon Whaley <address@hidden> wrote:
>>
>> Have you tried instantiating a context with the config from Fabric's
>> connection?  I haven't mixed invoke tasks and fabric tasks in the same
>> module before, I'm surprised the same task decorator will work for
>> both using `inv` on the command line and `fab` as well.  If you have a
>> task that only does local work inside your fabfile module, I'd
>> recommend using conn.local instead of making it an invoke task.  If
>> you have an existing invoke task codebase, I would import that module
>> from my fabfile module and run tasks with a locally created context,
>> since you're not calling the task from the invoke cli.
>>
>> ```
>> tasks.py
>> ====
>> from invoke.tasks import task
>>
>> @task
>> def build(ctx):
>>    ctx.run('make artifact')
>>
>> fabfile.py
>> ====
>> from fabric.tasks import task
>> from invoke.context import Context
>> from tasks import build
>>
>> @task(hosts=PROD)
>> def deploy(conn):
>>     ctx = Context(config=conn.config)
>>     build(ctx)
>>     conn.put('artifact', '/app')
>> ```
>>
>> On Tue, Apr 2, 2019 at 2:04 AM Michel Albert <address@hidden> wrote:
>> >
>> > I understand the differences. The only traceback I  got was the one I sent earlier. This traceback is correct, only maybe a bit cryptic.
>> >
>> > So far, I managed to get some basic tasks ported to fabric2.
>> >
>> > However, this morning I ran into a new problem:
>> >
>> > I have a remote task (deploy) which depends on a local task (build) and I want to be able to call the build step separately. But now I don't know how to call "build" from the "deploy" task. The docs state to simply call the function and forward the connection/context. But when calling from the "deploy" task, I have a "Connection" instance, but I need a "Context" instance to pass into the "build" task. Is there a way to get the Invoke context from the Connection instance? Example:
>> >
>> > @task
>> > def build(ctx):
>> >    ctx.run('make artifact')
>> >
>> > @task(hosts=PROD)
>> > def deploy(conn):
>> >     build(?)  # <- what do I need to pass onto build? "conn" won't work
>> >     conn.put('artifact', '/app')
>> >
>> > On Mon, Apr 1, 2019 at 9:33 PM Brandon Whaley <address@hidden> wrote:
>> >>
>> >> There is no local attribute on invoke contexts, only on fabric
>> >> connections (which inherits from invoke contexts).  If this isn't what
>> >> you're trying to do, could you provide some code examples that give
>> >> you the traceback you're seeing?
>> >>
>> >> On Mon, Apr 1, 2019 at 1:07 PM Michel Albert <address@hidden> wrote:
>> >> >
>> >> > Ok. I think I will get it to work like that.
>> >> >
>> >> > But after fiddling around with it I found out that the type of the first argument depends on whether a "hosts" argument was passed into the task decorator or not. This was quite confusing. And the error is a bit cryptic. When calling "local" on an invoke context, the following AttributeError is thrown:
>> >> >
>> >> > Traceback (most recent call last):
>> >> >   File "/usr/lib/python3.7/site-packages/invoke/config.py", line 113, in __getattr__
>> >> >     return self._get(key)
>> >> >   File "/usr/lib/python3.7/site-packages/invoke/config.py", line 178, in _get
>> >> >     value = self._config[key]
>> >> >   File "/usr/lib/python3.7/site-packages/invoke/config.py", line 169, in __getitem__
>> >> >     return self._get(key)
>> >> >   File "/usr/lib/python3.7/site-packages/invoke/config.py", line 178, in _get
>> >> >     value = self._config[key]
>> >> > KeyError: 'local'
>> >> >
>> >> > During handling of the above exception, another exception occurred:
>> >> >
>> >> > Traceback (most recent call last):
>> >> >   File "/home/exhuma/.local/bin/fab", line 10, in <module>
>> >> >     sys.exit(program.run())
>> >> >   File "/usr/lib/python3.7/site-packages/invoke/program.py", line 363, in run
>> >> >     self.execute()
>> >> >   File "/usr/lib/python3.7/site-packages/invoke/program.py", line 532, in execute
>> >> >     executor.execute(*self.tasks)
>> >> >   File "/usr/lib/python3.7/site-packages/invoke/executor.py", line 129, in execute
>> >> >     result = call.task(*args, **call.kwargs)
>> >> >   File "/usr/lib/python3.7/site-packages/invoke/tasks.py", line 128, in __call__
>> >> >     result = self.body(*args, **kwargs)
>> >> >   File "/home/exhuma/tmp/fabfile.py", line 8, in get_version
>> >> >     version = ctx.local('python setup.py --version').strip()
>> >> >   File "/usr/lib/python3.7/site-packages/invoke/config.py", line 125, in __getattr__
>> >> >     raise AttributeError(err)
>> >> > AttributeError: No attribute or config key found for 'local'
>> >> >
>> >> > Valid keys: ['connect_kwargs', 'forward_agent', 'gateway', 'inline_ssh_env', 'load_ssh_configs', 'port', 'run', 'runners', 'ssh_config_path', 'sudo', 'tasks', 'timeouts', 'user']
>> >> >
>> >> > Valid real attributes: ['cd', 'clear', 'config', 'cwd', 'from_data', 'pop', 'popitem', 'prefix', 'run', 'setdefault', 'sudo', 'update']
>> >> >
>> >> >
>> >> > On Mon, Apr 1, 2019 at 5:37 PM Brandon Whaley <address@hidden> wrote:
>> >> >>
>> >> >> Hi Mich,
>> >> >>
>> >> >> The connection object (you're using ctx in your examples) has a .local
>> >> >> method that is just a pass-through to invoke.run.  It's documented on
>> >> >> the connection object's page:
>> >> >> http://docs.fabfile.org/en/2.4/api/connection.html?highlight=local#fabric.connection.Connection.local
>> >> >>
>> >> >> On Mon, Apr 1, 2019 at 5:55 AM Michel Albert <address@hidden> wrote:
>> >> >> >
>> >> >> > Hi,
>> >> >> >
>> >> >> >
>> >> >> > Consider the following fabric-1 task. For illustration I kept it really short:
>> >> >> >
>> >> >> > @fab.task
>> >> >> > def sample():
>> >> >> >     version = fab.local('python setup.py --version')
>> >> >> >     fab.run('mkdir -p /snapshots/%s' % version.strip())
>> >> >> >
>> >> >> > This task needs to run a local and remote command. I am now trying to port this to fabric-2, and I can't figure out how I can implement this. If I define the "hosts" variable in the task, then the first line will be executed on the remote host as well, which I don't want. A naive aproach which won't work:
>> >> >> >
>> >> >> > @task(hosts=PROD)
>> >> >> > def sample(ctx):
>> >> >> >     version = ctx.run('python setup.py --version').strip()   # <- this won't work
>> >> >> >     ctx.run('mkdir -p /snapshots/%s' % version)
>> >> >> >
>> >> >> > At first I thought I would split the task into two, one for just local commands and one for remote tasks, but then I am forced to pass in the context, which will in turn cause it again to be run remotely:
>> >> >> >
>> >> >> > @task
>> >> >> > def get_version(ctx):
>> >> >> >     version = ctx.run('python setup.py --version').strip()
>> >> >> >     return version
>> >> >> >
>> >> >> > @task(hosts=PROD)
>> >> >> > def sample(ctx):
>> >> >> >     version = get_version(ctx)  # <- this won't work
>> >> >> >     ctx.run('mkdir -p /snapshots/%s' % version)
>> >> >> >
>> >> >> > How can I accomplish something like this? And where is it noted in the docs? In the current example on the "Upgrading from 1.x" page does not have a single task mixing local with remote commands in any way.
>> >> >> >
>> >> >> >
>> >> >> > Regards,
>> >> >> >
>> >> >> >
>> >> >> > Mich
>> >> >> >
>> >> >> > _______________________________________________
>> >> >> > Fab-user mailing list
>> >> >> > address@hidden
>> >> >> > https://lists.nongnu.org/mailman/listinfo/fab-user