Re: [Fab-user] Greetings and suggestions

[Niklas Lindström]

Hi Jeff!

Thanks a lot for sifting through my suggestions! :) And for the
detailed response. I'll go through it piece by piece.


== Namspacing mechanisms ==

>> 1. Making "_private" names in scripts accessible to the functions
>> (commands) you define.
>
> I can see two questions here, one being the desire for "helper"
> functions which are available inside commands but which don't become
> commands themselves; and the other being a desire to have the entire
> module-level namespace in your Fabfile, available to the commands. I
> like the former; we could e.g. make a decorator one would use to flag
> these helper functions so that load() knows not to make them actual
> commands. The latter desire ties somewhat into your next question.

Something like that would be useful. Although each command could be
explicitly marked as @command as well, that'd take away the
cleanliness of fabfiles; so I like this idea better. That or just
marking "helper" functions with a preceding "_", the pythonic way. :)


>> 2. In "load", use the optional globals and locals that execfile
>> supports to manage the names accessible in fabfiles.
>
> This is a good idea and one I was actually thinking of a while ago
> (but, of course, never got around to poking at). We would definitely
> want to retain the stuff we expect fabfiles to use, like the decorator
> mentioned above (and the other decorators already available that I've
> written, namely @hosts and @mode). Retaining modules like 're', I
> could go either way on -- yea, it's nice to have stuff already there,
> but it would feel like magic. I've been learning Rails lately and
> magic is hell when you're trying to debug or pick up someone else's
> project.

I agree, magic often eventually becomes unmanageable.

> Anyway, aside from stuff-in-fabric.py, there's also the earlier
> question of what from the Fabfile(s) to include/exclude -- I think
> that would require a higher level discussion of just what the
> execution environment inside Fab commands, is expected to be.

Definitely. This would be a rather large (but probably
straightforward) refactoring of the current code. It would also raise
questions of wether to use execfile, or thinking about __import__ or
similar.

.. But I've experimented with a patch for some of the suggestions
below, and come to realize that they're probably just enough to solve
my current way of using fabric without requiring even helper
functions. So I think this is less of a priority for me now.


>> 3. One useful approach may be to create Command objects for all
>> commands, which stores "private locals" from the file in which they
>> were created.
>
> I think Christian mentioned something along these lines at one point?
> I know we've already class-ified the connection object which makes
> certain tasks easier and more self-contained; doing this for commands
> might also be a good idea. You're right that in the case where one
> loads up >1 fabfile (a use-case I personally haven't had to think
> about much, unfortunately) it'd be great to try and avoid inadvertent
> shadowing of variables.

Sounds like we're on the same page. :) This would go into the larger
refactoring I mention above, something worthy of a joint effort to
focus on this alone.


== Straight-forward improvements ==

>> 4. Only run "load" once per filename (by storing the loaded filenames
>> to prevent from loading them twice).
>
> +1 on this.

Great. I can attempt a patch if you'd like. In fact, I'll go through
the rest here and mention what I'd like to contribute, and then write
a follow-up with my plan, along with a patch I've made for some of
this.


>> 5. Add a "depend" operator.
>
> Makes some sense, though I'm unsure what the best syntactical approach
> is, the only thing I can think of that doesn't have issues is to do
> something like depend('name_of_other_command'), which is slightly icky
> but not horrible. (It's also where Ruby's symbols seem like a nice
> idea, as they communicate the concept of something that's
> not-quite-a-variable and not-quite-a-string)

Indeed. But I think the way "provided_by" in "require" does it works
fine -- supply a function reference if you have it, or the name of it
if not.

> I also don't think "depend" is the right name, given that you'd be
> using this declaratively inside a command -- just doesn't sound verby
> enough. Maybe "call_once" or something like that.

Sounds reasonable.

> Alternately, we could set it up as a decorator, i.e.
> @depends('other_command', 'other_command2') which would then execute
> those other command(s) before actually executing the one being
> decorated (and only if they haven't already been called). The problem
> with this is that it doesn't allow you to call it declaratively, e.g.
> in a situation where the current command wants to set something up
> before 'depending' the other command.

See below. :)


>> 6. Extend require to take a varargs instead of just one name, so you
>> don't have to repeat calls to it if you have a bunch of variable names
>> you need to get from the "provided_by" commands.
>
> This would only work if it were made very clear that the optional
> kwargs would then apply to _all_ of the varargs, i.e. that they are
> all used for and/or provided by the same thing. If that's where you
> were going with this, then I guess that makes sense -- you're
> essentially refactoring aimed at the kwargs, so that a large pool of
> variable names would end up 'collected' by any common
> used_for/provided_by values.

That's precisely how I thought about it. I've done this, so this will
be part of my coming patch.


>> 7. Add decorator versions for require ("@requires(...)") and depend
>> ("@depends(...)"), to be used for declaratively binding commands
>> together.
>
> I already mentioned part of this above, so obviously I agree to some
> extent. Having things be used as either decorators OR commands feels a
> little icky to me, however (not to mention that the actual
> implementation of those functions would need to be pretty magical,
> detecting how they were used and changing behavior appropriately). I
> might prefer that they stick to being one or the other, or have
> slightly different names for the two behaviors so they're not
> literally the same function.

Absolutely. In fact, my suggested decorators end with "s", to
differentiate them from the imperative operators. :) I can add these
as well (aligned with your suggested "call_once" above).


>> 8. Support for filename globs (e.g. "components/*.py") in "load". This
>> might be less useful if number 4 is implemented.
>
> Globbing is almost always a welcome addition to programs that deal
> with file paths; so +1 from me.

I can do this as well.


>> 9. Extend prompt to repeat itself until "validate" is satisfied.
>
> I forget if this is in main Fabric but I am pretty sure I have some
> prompt repeating in the case of incorrect passwords, and having that
> for other prompting methods would be a good idea. +1.

Part of the patch.


>> 10. A regexp match complement to the current validate feature of
>> prompt.
>
> Not a bad idea; I actually think it might make the most sense to
> overload the validate keyword: if a callable, it's called, if a
> string, it's used as a regex match.

Good point; in the patch as well. ;D


>> 11. Optionally define variables on the commandline (e.g. "fab
>> tag_release -v tagname=release-1.0").
>
> How is this significantly different from the existing argument
> functionality? (serious question)

Heh, only in my mind. I had missed that completely, so scratch this
suggestion. I'm (very) happy with the existing take on it (though it
may need more some more documentation..).


>> 12. Support for supplying the fabfile to use with a named commandline
>> flag (e.g. "fab --use setup/vcs/tagging.py").
>
> Can you provide a use case where this isn't solved by using (as you
> mention) load() in your current fabfile? Or do you mean to use it in a
> directory where no fabfile exists?

Yes, it's for putting fabfile away from where fabric will actually
work. Mainly because I personally have a "setup" dir with the fabfile,
which currently uses a "basedir='../'" for a lot. One could argue that
I should put the fabfile in the root of course..


>> 13. Support for changing working dir via operator(s).
>
> Did you have an idea about how to implement this? Given that things
> are executed in turn and the remote shell is reopened each time, the
> only thing that really springs to mind is that Fabric would, in the
> background, be prepending 'cd /foo/bar ; ' to every run/sudo/etc
> command. Not terrifically clean; but not horrible either, and it would
> give the end-user a feeling of a persistent shell session, which might
> be nice.

In fact, that was my crude initial solution. Worth thinking more about though.

>> (Also note that this with-version would require e.g. a threadlocal for
>> [...]
>
> Check out my execution branch on Github, it would work with your
> 'with' version just fine if fab_mode is set to 'deep'. (I wrote it
> specifically to enable fab commands that could meaningfully make use
> of logic instead of just being a series of disconnected statements)

Interesting, I should go and study that in detail.


>> 14. The name "set" shadows the built-in set type.
>
> This is a minor peeve of mine too, but I can't think of any
> alternative name that communicates what it does nearly as well, which
> is why I haven't changed it. Pretty sure I've just dealt with sets by
> using the __builtins__ trick you mention, although that is kind of
> messy. Perhaps we could replace set() by exposing the mapping object
> set() operates on (directly or indirectly) so users would just do
> vars['foo'] = 'bar' instead of set(foo='bar').

Funny; I thought of the same variant. Unfortunately, "vars" is also a
builtin. ;) How about a "params" object, which would work like::

    # Define one or more variables (just as with current "set"):
    params(fab_user='me')
    params(fab_hosts=[...], fab_user='me')

    # Define one var can also be done like:
    params.fab_user = 'me'
    params['fab_user'] = 'me'

    # For getting them, same pattern:
    print params.fab_user
    print params['fab_user']


>> 15. Possibly adding Jython support. Take a look at
>> <http://code.google.com/p/robotframework-sshlibrary/> for a way to go
>> about that. I personally don't consider this important at the moment
>> though, but it may be useful in the future.
>
> Sadly Jython's not on my personal todo list at all, but once we get a
> long-term ticket tracker set up, I think that'd be a good candidate
> for a 'someday' milestone :)

Yup. Since Jython 2.5 won't be released until around the end of the
year, this is probably not that crucial for most people. (The wrapper
for two ssh backends in robotframework-sshlibrary just looked too
promising not to mention right away -- in case someone else wants to
take a look.)


> Thanks again for all this input!

Thank you as well; I'm only happy if I can contribute in some way.


Now, with this gone through, I will write the follow-up which details
of what I've done, and intend to do. And attach the patch of my
changes there (generated against the master branch).

Best regards,
Niklas