Re: Minimal Guile

[Mark H Weaver]

Mike Gran <address@hidden> writes:

>> From: Mark H Weaver <address@hidden>
>
>> What is the advantage of including our own little read-only filesystem,
>> when every OS already provides this functionality?  Is it really
>> significantly easier to install 3 files than to install 300?
>> 
>> Admittedly, I can see how it might make a psychological difference.
>> Somehow, people get the feeling that a package is huge and bloated when
>> it contains a large directory structure, whereas a single file of the
>> same size (or even larger) seems significantly less obtrusive.
>> 
>> However, I'm not sure that this psychological difference is enough to
>> justify the reduced flexibility of such an approach.
>> 
>> Is there an advantage that's not merely psychological?
>
> No, there is no advantage beyond the psychological for any system that
> uses the standard Unix-like filesystem hierarchy and has a decent
> package manager.

How does the absence of a package manager affect the desirability of 3
files vs 300?  I would think that any archiver such as tar or zip would
be sufficient to make unpacking a directory of 300 files as easy as
installing 3 files.

It seems to me that the problem caused by lack of a decent package
manager has to do with Guile's external dependencies.  This is a
separate issue from 3 files vs 300.

The obvious solution is to provide pre-built Guile packages for popular
platforms that contain all dependencies bundled.  This is the standard
practice on MacOS for example, and probably Windows too I guess.

> There are some theoretical corner cases where it could be useful.
> Don't know if these would ever occur in practice.
> - Systems that don't use a Unix-like filesystem heirarchy

I also don't see how Unix-like filesystem semantics affect this
question.  If the semantics are significantly different, that might
require minor changes to Guile's loading of .go files, but I don't see
how it would require us to combine all the .go files into one file, as
long as the filesystem supports directories.

> - Programs that are distributed in a folder whose root location
>   can change

This is a real problem, I agree.  For example on MacOS, applications are
actually entire directories that are made to look like single files to
the user.  This is Apple's solution to the psychological problem of
programs with too many files :)

In any case, users are accustomed to being able to freely move these
applications to any folder they wish.  Therefore, we should make sure
that applications using Guile can include Guile within their application
directory, and that this directory can be moved without breaking
anything, without any special action on the user's part.

However, it seems to me that this problem is strictly orthogonal to the
question of 3 files vs 300.  The number of files doesn't matter here.
Even if we have to find only one file, we still need to solve the
problem of how to find this file if the user moves it.

> - Programs that use Guile as an extension but want to limit its
>   library for some reason.

I don't understand what you mean here.  By limiting the library, do you
mean removing some of the modules that are currently distributed with
Guile by default?  If so, this also seems like a separate issue.

> It is all branding, or marketing, I guess.  And fun, of course. I
> thought that, as a hack, it would be fun to try.

I agree that marketing is important, and perhaps this might help :)

I would recommend against tar format, because it is a stream format with
no index.  The only way to find an individual file in a tar file is to
read it from the beginning until you reach the file.

If this single file is to be built only during the package build
process, and never modified after that, then it should be super-easy to
invent your own little homebrew format.  If we assume that the library,
executable, and go-bundle files are always distributed together, that
also means that we're free to change the format at will.  I guess it
could be as simple as this:

* Concatenate all of the .go files together, making sure to align each
  .go file on a page boundary (4 kilobytes) so that each one can be
  individually mmapped.

* Generate a C file that includes the entire directory of files within
  an initialized array.  It could be as simple as an array of structs
  with three fields: module name, start offset in the big file, and
  length.  If you sort this array by module name before generating the
  file, then you could use binary search to find the file.

* Once the C file is generated, relink libguile with this file included.
  (The first time libguile is built, it could include a stub version of
  the C file with an empty directory, so that it knows to use the
  traditional module-loading mechanism).

* To allow graceful error messages in case the wrong go-bundle is used
  with the wrong library, there should be some kind of fingerprint put
  in both the go-bundle and the C file index.  Check the fingerprint
  before trying to load things from the go-bundle.

Just an idea.  Another possibility is to use something like TinyCDB
<http://www.corpit.ru/mjt/tinycdb.html>, but be aware that each .go file
should be aligned on a page boundary to allow fast mapping.  This might
require some changes to the format.

   Happy hacking,
       Mark