Re: [Fab-user] What is the future vision of fabric/invoke?

[Jeff Forcier]

Hi Mich,

Thanks a lot for your detailed feedback; I appreciate the difficulty you're going through. The development status has been in limbo since my employer was acquired late last year, so I haven't had time to polish all these rough edges. This ought to change in the near future now that I've found new employment.

I'm in the middle of a cross-country move this week and next, so I can't be as detailed as I'd like, but some quick points:

> [The upgrade page] was not really helpful and very confusing. I ended up perusing the API docs to find what I needed, but this still involved some guesswork.

I've got my own ideas for how to improve that page's usability, but I'd love to hear about what you were expecting to find / what sorts of info (or style of organization) would have helped you here.

> I now believe that the "fab" executable is useless for us and that we could do everything with "invoke" and just import fabric connections in our "tasks.py" file.

The `fab` shim needs more work to avoid a lot of the rough edges (to be understated) you've run into, but...

 > it does not seem that this is the intended way to do it.

... this isn't accurate - using Invoke as the main interface/framework and then importing Fabric into that code *is* definitely a supported/intended use case. (A major goal here was to be less opinionated and more just providing building block APIs that advanced users can use however they need, without us having to anticipate every use case.)

I _do_ want to get to where v2's `fab` is more readily usable for folks coming from v1, as well, but I suspect for now an intermediate-to-advanced team would find more utility with the other approach.

> Dropping the concept of "roles" surprised me.

Roles have NOT been dropped purposefully, they just haven't been migrated yet - getting the lower level APIs set up (again, in the spirit of "let advanced users build what they need instead of making them wait for higher level stuff to get finished") was done first.

The upgrading page should have a link to the issue or issues around building a "v2" style functionality that fills the roles use case better (mostly by bridging configuration to the Group class). If you didn't run across that line, it's likely part of the previous issue re: searchability of that page.

> Running local tasks does not behave as expected due to the way the console and environment variables are handled.

This feature area ought to be an overall upgrade to how it worked in v1, so I'd definitely appreciate details here (if you posted them in another thread, just let me know, as noted I have not had time to pay close attention). It's possible you're running into things related to the v2 APIs having more fine-grained 'levers' for you to tweak, not sure.

> "put" no longer supports folders [...]

I think this was noted on the upgrading page, but the recursive put() had an absolutely dismal utility-to-bug-surface ratio (esp given it's just us poorly reinventing functionality that native Unix tools have had for decades). Is there a reason you're not able to use things like tar.gz?

My expectation is to make the 'ramp' to this easier (eg making it trivial for users to determine what native tools they have available - tgz, rsync, etc - or autoselection) without actually porting v1's bug-prone implementation.

> I am wary of the fact that I need this kind of scaffolding for something that was much easier with fabric-1

As you can probably guess from my opening and the rest, this is (IMO) due almost entirely to the fact that I wanted to get v2 out into the world instead of waiting until it was "fully ready" (combined with unfortunate timing in my professional and personal life) and does _not_ reflect the intended final state of things. Building those kinds of scaffolding are exactly where I intend the core dev to go next.

Best,

Jeff

On Thu, Apr 4, 2019 at 1:37 AM Michel Albert <address@hidden> wrote:

Hi,

In our development team we have been using fabric-1 as task runner for many years and we are really happy with it. Initially we chose it for the following reasons:

• Being Python, we can transfer our development logic to our task-runner as Python is our main development language
• Defining tasks is really easy and flexible
• It allowed us to create a library of shared tasks for the whole team making definition of new fabfiles a breeze for our projects
  
• It "just worked"

We are now planning to review and refresh our whole development process and environment. The task runner is part of this.

As some preparation work and to gain experience I've tried to move some personal projects to fabric-2 and ran into several problems:

• The first hurdle was the "migrating to fabric-2" document. It was not really helpful and very confusing. I ended up perusing the API docs to find what I needed, but this still involved some guesswork.
• The split of "invoke" and "fabric" is confusing. Especially since the argument passed into the tasks depends on how they are called. If called without host information, the first argument will be an invoke context, if host information is available it becomes a fabric connection, and both are not API compatible. After several attempts, I now believe that the "fab" executable is useless for us and that we could do everything with "invoke" and just import fabric connections in our "tasks.py" file. This way we have consistency with the first argument, but it does not seem that this is the intended way to do it.
  
• Dropping the concept of "roles" surprised me. It seems to me that this would be an integral part of a task-runner. This now forces me to implement "roles" logic in all of my task files.
• Running local tasks does not behave as expected due to the way the console and environment variables are handled. Using "pty=True" does not help with everything and "replace_env=False" is also a bit cryptic.
• "put" no longer supports folders and one is now reliant on "patchwork" to get the "rsync" task. I am worried that the docs state that patchwork is not a priority as we rely on the functionality to push folders onto remote machines. fabric-1 made this trivial.
  

I do like the configuration system and how task-arguments are handled in the CLI tool. This is a big step-up from fabric-1

After discussing this with the team another member stated that he made the same effort on some of his personal projects and ran into the exact same issues and has given up on fabric-2 as it makes task-definitions too complicated compared to fabric-1. Although in defense of fabric, I wonder if this is not foremost an issue with the documentation. As I mentioned, I fled to the API docs after reading the "prose" docs as they really inundated me details I did not need to know and was unable to find what I was looking for.

So in our team I am now faced with making a decision of picking a new task-runner. The options I see are:

• Switch to fabric-2 - I'm worried that the complexities will make it difficult for the less experienced developers in our team to define tasks. It also seems that fabric-2 is not really gaining traction which led me to the subject of this mail.
  
• Stick with fabric-1 and hope that fabric-2 is at some point on par with fabric-1 in terms of simplicity - This does not seem like a good long-term plan and would force us to install Python2 on our machines solely for fabric.
• Use Makefiles instead - This would solve most of our problems but makes it difficult to share common tasks and is definitely not as convenient as writing Python
• Find another task-runner - This is unknown/unexplored territory
  

All the points mentioned above are mainly inconveniences which can be mitigated by writing a bit of scaffolding around it. But I am wary of the fact that I need this kind of scaffolding for something that was much easier with fabric-1. The main goals for me is to have a task-runner where it is easy to define and execute tasks. Even for people that don't do Python development as their daily business.

In order to make an informed decision about our future task-runner the question in the topic is very important to me: What are the future plans for fabric-2?

BR

Mich

_______________________________________________
Fab-user mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/fab-user

-- 

Jeff Forcier
Unix sysadmin; Python engineer
http://bitprophet.org

On Thu, Apr 4, 2019 at 1:37 AM Michel Albert <address@hidden> wrote:

Hi,

In our development team we have been using fabric-1 as task runner for many years and we are really happy with it. Initially we chose it for the following reasons:

• Being Python, we can transfer our development logic to our task-runner as Python is our main development language
• Defining tasks is really easy and flexible
• It allowed us to create a library of shared tasks for the whole team making definition of new fabfiles a breeze for our projects
  
• It "just worked"

We are now planning to review and refresh our whole development process and environment. The task runner is part of this.

As some preparation work and to gain experience I've tried to move some personal projects to fabric-2 and ran into several problems:

• The first hurdle was the "migrating to fabric-2" document. It was not really helpful and very confusing. I ended up perusing the API docs to find what I needed, but this still involved some guesswork.
• The split of "invoke" and "fabric" is confusing. Especially since the argument passed into the tasks depends on how they are called. If called without host information, the first argument will be an invoke context, if host information is available it becomes a fabric connection, and both are not API compatible. After several attempts, I now believe that the "fab" executable is useless for us and that we could do everything with "invoke" and just import fabric connections in our "tasks.py" file. This way we have consistency with the first argument, but it does not seem that this is the intended way to do it.
  
• Dropping the concept of "roles" surprised me. It seems to me that this would be an integral part of a task-runner. This now forces me to implement "roles" logic in all of my task files.
• Running local tasks does not behave as expected due to the way the console and environment variables are handled. Using "pty=True" does not help with everything and "replace_env=False" is also a bit cryptic.
• "put" no longer supports folders and one is now reliant on "patchwork" to get the "rsync" task. I am worried that the docs state that patchwork is not a priority as we rely on the functionality to push folders onto remote machines. fabric-1 made this trivial.
  

I do like the configuration system and how task-arguments are handled in the CLI tool. This is a big step-up from fabric-1

After discussing this with the team another member stated that he made the same effort on some of his personal projects and ran into the exact same issues and has given up on fabric-2 as it makes task-definitions too complicated compared to fabric-1. Although in defense of fabric, I wonder if this is not foremost an issue with the documentation. As I mentioned, I fled to the API docs after reading the "prose" docs as they really inundated me details I did not need to know and was unable to find what I was looking for.

So in our team I am now faced with making a decision of picking a new task-runner. The options I see are:

• Switch to fabric-2 - I'm worried that the complexities will make it difficult for the less experienced developers in our team to define tasks. It also seems that fabric-2 is not really gaining traction which led me to the subject of this mail.
  
• Stick with fabric-1 and hope that fabric-2 is at some point on par with fabric-1 in terms of simplicity - This does not seem like a good long-term plan and would force us to install Python2 on our machines solely for fabric.
• Use Makefiles instead - This would solve most of our problems but makes it difficult to share common tasks and is definitely not as convenient as writing Python
• Find another task-runner - This is unknown/unexplored territory
  

All the points mentioned above are mainly inconveniences which can be mitigated by writing a bit of scaffolding around it. But I am wary of the fact that I need this kind of scaffolding for something that was much easier with fabric-1. The main goals for me is to have a task-runner where it is easy to define and execute tasks. Even for people that don't do Python development as their daily business.

In order to make an informed decision about our future task-runner the question in the topic is very important to me: What are the future plans for fabric-2?

BR

Mich

_______________________________________________
Fab-user mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/fab-user

--

Jeff Forcier
Unix sysadmin; Python engineer
http://bitprophet.org