Re: Re: Re: [PATCH 0/4] Add import builtin

[Matheus Afonso Martins Moreira]

> If the purpose is to introduce a standard module system in Bash,
> I'll have to be opposed to the change because I think the current
> suggestion for `source -i' wouldn't help satisfy that purpose at all.

I think we were unable to reach an understanding because we were both
using the same words to talk about different things. I think I have a
different idea of what constitutes a "module/library system" and it's
causing everyone to think that I want more out of this patch than I
actually do.

By "library system" I just mean the exact mechanism through which
bash will load a "library". By "library", I mean ordinary scripts
whose purpose is to collect related functions and variables
for reuse by other scripts or interactive bash users.

A library is a container for functions/variables,
and the library system is the exact API scripts use
to fill up those containers with code and data.

So at least to me this is all about the "import" primitive.
Which particular form it takes is up to debate, it's something
that's worth discussing and getting right before it's released
due to the difficulty in changing such core APIs after the fact.
However, the fact is that I think many of those other considerations
are actually out of scope!

The include guard has potential to be a language feature.
This is a generally useful feature since libraries are usually
meant to be included only once. This allows the source statement
to be idempotent, meaning `source x; source x` does not run the
script twice and therefore produces no additional side effects.
An override via a --force option can be provided to allow control
over this in the cases where it is intended and desirable.
It should also be easy to implement, just need to make a set of
loaded scripts and only execute them if forced to or if the script
is not in the set, ensuring that it is added to the set later.
Bash already has hash table support so it should be a simple matter
of using the data structure.

On the other hand, central repositories are a completely external concern,
as is the installation and management of libraries. Those are problems better
solved by Linux distribution packagers or a custom package manager for bash.
I may be wrong but I believe that is the role many of those bash frameworks
are fulfilling. In order to do that, they _had_ to create an import primitive
for themselves because bash lacked such facilities. My intention is to fix
the lack of the import facilities, not to replace those frameworks.
Ideally, they'll be using bash's native import under the hood!

By introducing the "module/library system" I want to do the following:

  1. Add a builtin primitive that can be used to load libraries
  2. Establish a convention for where bash will look for libraries
  3. Separate the libraries from the commands/executables

The proposed import builtin accomplishes goal 1.
The BASH_IMPORT_PATH variable accomplishes goal 2.
The fact import will look for libraries _only_ in
BASH_IMPORT_PATH accomplishes goal 3.

Which paths are contained in BASH_IMPORT_PATH is completely undefined.
Linux distributions might add their own paths. Users might do the same.
Maybe the bash frameworks will take control over that variable in order
to run their commands so that they can ensure only their packages are
loadable by bash. Someone might make a "bashenv" tool for isolating
a set of bash libraries. All these usages are enabled by the builtin.

A uniform interface to library loading will make it easier to develop
library code: there is one bash native way to import a dependency,
not one specific import mechanism for each framework.

The frameworks can still manipulate the load paths however they want,
but the code will be more uniform. Maybe it's wishful thinking but
perhaps they'll even end up being portable between the frameworks.

> However, if the purpose is solely to solve the ambiguous search paths
> of the `source' builtin, I think a flag to the source builtin is a
> reasonable suggestion.

That is the purpose behind this patch!
That's what I meant by "library system".
A system that's built into bash through
which libraries can be loaded.

Solving the ambiguous search paths issue
is a prerequisite for it, but an issue of
equal importance is the separation of the
load paths for libraries and commands.
This will allow the library loading paths
to be freely manipulated without affecting
the way that scripts work.

> This is my personal preference for the name,
> but the current simple feature of restricting
> the search paths doesn't feel` import'.
> I'd think it can simply be BASH_SOURCE_PATH.
> The flag can also be something other than `-i'
> or` --import', like `source -l <name>' to mean
> the "library".

BASH_SOURCE_PATH and `source -l|--library` are good names.

BASH_LIBRARIES_PATH would be consistent with --library
as well as existing variables such as BASH_LOADABLES_PATH.
 
> I feel we should ask the package maintainers to include those files.

I think the fact they are called "examples" might be having a major impact
on how such files are perceived by packagers and distributors. "Examples"
sound like documentation, example code meant for learning how some bash
feature works. They generally copied and built upon, not used as libraries.

Adding useful functions to a collection of functions that are shipped along
with bash is more akin to a standard library... That would be an awesome
addition to bash but I think the library system should be perfected first.

> The different module systems do not need to cooperate with
> each other but should be allowed to coexist in a session and work
> without interfering with each other.

I understand.

> For example, when module X managed by system A requests another module
> Y, we want to find module Y provided by system A and exclude the
> modules provided by system B (even when no module is provided with the
> conflicting name Y in system B). The idea is the same as yours to
> separate the namespace of the executable files and the script files to
> source.

It would be reasonable for the framework to take over the BASH_SOURCE_PATH
variable while performing its own operations. They already define their own
import mechanisms, they can build upon a native bash builtin that way by
simply using the builtin and overriding BASH_SOURCE_PATH with a local
that is under their full control.

So instead of everyone having to define such a mechanism for themselves,
only the bash frameworks would have to. Users could be well served by
running `source -l some-file` directly in the terminal without being
forced to use a framework at all, but they can still choose to use one
if that's what they want.

> Yes, I agree with that. But everyone wants to sell their own module
> framework. The war begins. I think the war is a good thing to improve
> the language, but one needs to be prepared.

That's true. It's not my intention to go to war with anyone,
but I can't deny that I do have a vision for bash and how it
should evolve. One of the coolest things about free software
is how we can actually make it happen, that motivates me.

I don't think my vision is mutually exclusive with the vision of others.
I think it could even make their visions easier to achieve.
So I do hope we can reach a consensus.

> I think a subtlety in shell languages is how to count `a' language.
> One may want to have a unique module system for POSIX scripts, and
> another may want to have a unique module system for Bash scripts. They
> inevitably conflict if they want to be the unique system. Some
> frameworks even try to implement ``a new shell language'' on top of
> POSIX (Modernish is this) or Bash (Bash Infinity is kind of this).

I would like to clarify this: by "the" language I meant bash specifically,
not POSIX shell. It was my intention that this be a so called "bashism",
specific to bash shell scripts. I don't generally hold POSIX compliance
in high regard, although I accept that it is an important consideration
for others and that compromises might be required given bash's POSIX
compliance requirements.

Even so, I'm not opposed to allowing POSIX shell scripts to use the
library sourcing feature. I assumed it would not be acceptable but I
could be wrong about that. Someone who's more knowledgeable about
and invested in POSIX compatibility should make that call.

Thanks for the feedback,
  Matheus