Re: [RFT PATCH v4 0/8] Sysroot series

[Ralf Wildenhues]

Hi Charles,

thanks for the elaborate explanations.

* Charles Wilson wrote on Thu, Aug 05, 2010 at 08:10:10AM CEST:
> On 8/4/2010 2:15 PM, Ralf Wildenhues wrote:
> > * Charles Wilson wrote on Tue, Aug 03, 2010 at 07:22:46AM CEST:
[...]
> >> Thanks to a quirk in window's path resolution, any
> >> unixish absolute path is interpreted as if it were a path from the drive
> >> root of the CWD.  So, if CWD is on C:, then /mingw/libexec is
> >> interpreted as C:/mingw/libexec.  Since ALL the unixish paths will start
> >> with /mingw, this will Do The Right Thing.
> > 
> > Sorry, but I don't think that is viable.  Say, C:/mingw is owned by the
> > administrator, but D:/mingw is domain of user A.  Starting your app on
> > D: as user B makes B susceptible to files which A owns.  In general, you
> > cannot leave away the drive letter when specifying the --prefix.
> 
> Then you are stuck with building twice.  Once for your sysroot
> installation, and once for the "real" installation on $host.
> However...that would STILL break:
> 
> libfoo.la
> a)  compiled for sysroot using the cross compiler, and installed there,
>     with --prefix=/something and --with-sysroot
> b)  compiled for $host using the cross compiler, and installed THERE,
>     with --prefix=E:/mingw or whatever.
> 
> Then, you compile libbar.la which depends on libfoo:
> a)  compiled for sysroot using the cross compiler, and installed there,
>     with --prefix=/something and --with-sysroot.  libbar.la(a)'s
>     dependency_libs will look like
>         -L=/something =/something/lib/libfoo.la
>     This is obviously useful for future compilation of libbaz.la.
> 
> b)  compiled for $host using the cross compiler, and eventually
>     installed there, with --prefix=E:/mingw or whatever.  However...
>     libbar.la(b)'s dependency_libs will look like
>         -L=/something =/something/lib/libfoo.la
>     too!  But that's clearly wrong, if you want to copy libbar.la(b)
>     directly over to $host and use it there, under E:/mingw.
> 
> Now, this is cross-compiler 101, for all architectures.  You really need
> to use the same "prefix" underneath your sysroot that you use on the
> actual $host.  Unfortunately, there is no such thing, on unix, as a
> prefix named "E:/somedir" (note the missing initial '/').
> 
> 
> But, that is untenable: on the one hand you want to assert that "thou
> shalt include X: in thy prefixes when configuring for $host=mingw" even
> when cross-compiling for any purpose,

No; I don't want to assert this.  Users may want to assert this, and I
don't want to libtool stand in their way if that is possible.  If some
user has only one drive, or is otherwise fine with not specifying a
drive prefix, I'm totally fine with that.

> but then there's this impossible
> requirement that the "prefix" underneath the sysroot be identical to the
> "prefix" on $host -- otherwise this whole cross compile thing falls
> apart, once you start chaining dependencies.
> 
> Now, if "/something" in the (a) cases above were "/mingw" (e.g.
> "matched" the drive-letter-stripped version from the (b) cases above),
> then it would actually work correctly -- but, you'd be open to exploits
> from D:/mingw.

> And this problem is not limited to libtool's .la files; if any path
> containing $prefix is compiled into a binary, or recorded anywhere,
> you'll have this problem.
> 
> The only real solution, as I've said, is:
> 
>   1) don't cross compile for $host=mingw. Ever. Always compile
>      natively, and use --prefix=X:/stuff WITH the drive letter.

We are at that position already, and would like to improve from there.

>   2) Fix *every* package that records ANY directory information
>      so that it is completely relocatable. Each binary or script, or
>      tool that parses a text file containing directory information,
>      should autodetect its installation location, and deduce relative
>      paths to its resources from that.

Relocatability is one possibility, and one that I think we should
expore eventually, because it is a conceptually good idea.

> #2 is a gigantic job, and it's not libtool's task to do that for the
> universe of software packages. Now, MAYBE, we might take a look at
> relocation support for .la files -- but I fear that would open us up to
> more security holes than you could shake a stick at.  Resolving all
> paths into relative paths from whatever is recorded in libdir=''? Oh, boy...

Why would it be clear that unreliable automatisms need to come into play
here?  Please, let's not confuse goals with flaws in conceived actual
implementations.

> Hence...I think the consensus is:

I don't think we need consensus on how people are going to use
autotools.  They can (and will) do whatever they like.  What we need
is consensus on which semantics we intend to (and can) support.


Just so that we don't misunderstand each other: I really intend for
this patch series to be merged, and by that I mean merged hopefully
soon after currently known issues are dealt with, even if we don't
have full test coverage and even if we can't foresee all the indented
and unintended semantics yet.  But the more we leave to later, the
more work it is going to be then.

So I really hope that my comments don't discourage you from working on
this.  You guys have done a great job so far, and I'm really trying to
not delay review; it would be sad to see this not fly just because we
talk past each other about some related questions.

> >> However, items built for case (2) -- direct deployment on the $host (or
> >> compiled "natively" ON the mingw host, that is MSYS/MinGW) -- are by
> >> convention configured as:
> >>    configure --prefix=`cd /mingw && pwd -W`
> >> which ends up as
> >>    configure --prefix=C:/MinGW
> >> and the installation is usually preformed as
> >>    make install prefix=/tmp/something/MinGW
> > 
> > I guess what I don't see is why the former case doesn't also use
> >   --prefix=C:/MinGW
> 
> Because if you then try to reconstruct the actual location on the disk,
> on $build under sysroot, you get:
>    /usr/i686-pc-mingw32/sysrootC:/MinGW/
> and now, the word "sysrootC:", besides being a sibling, rather than a
> child, of the ACTUAL sysroot returned by the compiler via
> -print-sysroot, is unsplitable: how can libtool extract the "C:/MinGW"
> part when it is geared towards splitting paths at path-delimiter characters?

What if we special-cased drive letters in the way that
  DESTDIR=C:/foo/bar prefix=D:/baz/zork

gets mapped to
  C:/foo/bar/D/baz/zork

?

This would need adjustments in Automake, Libtool, and an unknown number
of user-written rules, I guess.  And I haven't thought it through at
all; I'm merely throwing it out for the sake of the argument.

> > These two side-conditions mean that when I run the testsuite on all the
> > systems and in all the configurations that I have been testing
> > prereleases in the past, *none* of them will actually cover any of the
> > new sysroot functionality, because even the cross compilers that I test
> > do not use a sysroot.
> 
> Ah -- but gcc supports --sysroot even if it wasn't compiled with one (my
> cygwin native 4.3.4 doesn't have a -print-sysroot, tho; so I wonder if
> the --sysroot=/test/dir actually works for that compiler!)
> 
> However, my linux native compiler supports both (and, of course, wasn't
> actually compiled with a sysroot of its own):
> 
> $ gcc --sysroot=/bob -print-sysroot
> /bob
> 
> So, you could easily create a test that uses the native system compiler
> to "fake" a cross-toolchain "sysroot" test.

Exactly.  That's what I meant.

> But to make that work you'd
> still have to create a temporary fake sysroot underneath /bob as a
> symlink farm using /usr/lib and /usr/include -- which would quickly
> become painful, since most systems have LOTS of stuff under those two
> directories...
> 
> And then there's the fact that the toolchain ITSELF has bits installed
> under /usr/lib/gcc/*/...

These parts are what I don't know, because I have no experience in this
area.  Is creating such a link farm feasible?  Presumably, some subtrees
could just be created with one symlink to a directory?  Presumably the
symlink-tree script from Automake/GCC could help?

If such a test group proves to be very expensive, then I don't mind
disabling it by default; but just the fact that some automated way of
testing is fairly easily available, would be a big help.

> > Given the experience that new Libtool tests need on the average about
> > three or more iterations to get right portably, this makes me a bit
> > nervous.  We should really, really try to get test coverage high for
> > this feature on as many setups as possible.
> 
> Which definitely argues AGAINST making --with-sysroot the default, even
> if you ARE using a cross compiler, right away.

I don't think anybody argued that --with-sysroot should be the default.
Right?

> The key thing is, we want to ensure that
> 
>   1) native compilers don't break -- that's covered by the rest of the
>      test suite

Yep.

>   2) cross-compiler setups don't break, when libtool is used in its
>      default --without-sysroot (--with-sysroot=) mode.  I think this
>      is covered by "the rest of the test suite" when you build and
>      test libtool using a cross toolchain that has no sysroot.

Yep.

> Last -- and frankly least important in comparison to keeping EXISTING
> functionality working -- is the
> 
>   3) test the new --with-sysroot support in conjunction with cross
>      toolchains that actually have a sysroot.  This is covered by the
>      three new tests -- but requires toolchains with sysroot support.
> 
> The only answer for (3) is "Get Ralf Access To More Cross Toolchains".

Hehe.  ;-)

Cheers,
Ralf