Re: [lmi] Building with shared-library attributes

[Greg Chicares]

On 2006-1-6 22:23 UTC, Vadim Zeitlin wrote:
> On Sun, 01 Jan 2006 16:29:49 +0000 Greg Chicares <address@hidden> wrote:
> 
> GC> Then I rewrote all the makefiles to use a single directory for static and
> GC> dll builds: that saves build time, and it works with gcc, which can link
> GC> directly to a dll without using any import library.
> 
>  Unfortunately AFAIK this is the only Windows compiler which can do this
> and if we ever want to support another one, we really shouldn't rely on
> this.

I agree. In the past, I supported another compiler, and kept a pair of
directories for each compiler's object files:

  gcc/static/
  gcc/shared/

  borland/static/
  borland/shared/

Actually, that was the base of a hierarchy like

  gcc/static/debug
  gcc/static/release
  gcc/shared/debug
  gcc/shared/release

which can lead to a combinatorial explosion. Now I routinely work with

  gcc/debug
  gcc/release

only, which is perhaps going too far--an anticombinatorial implosion,
if you will. But it works quite well as long as I don't have another
compiler to build with, for now.

> GC> It is not my intention to make everything work with shared-library
> GC> decorations:

That seems overstated now. I didn't mean to say "I will never do this".
Instead, I meant only "here is a description of a particular limited
piece of work, and this is one of its limitations at this time". I was
trying only to add a test to make sure the decorations were correct
for the code we use in production, not to make "decorated" builds work
generally for all targets at this time.

>  I'd like to object to this: IMHO it's very confusing to use both static
> and shared libraries in the same build. Again, this might work without
> problems with gcc and its auto import feature and also because all code is
> relocatable under Windows but it wouldn't work with another Windows
> compiler or on another architecture where shared libraries have to be built
> with special options to generate relocatable code. So I think it would be
> much better to always link everything statically or dynamically but don't
> mix the two as this could easily hide the problems which would manifest
> itself in any other build configuration.

Yes, I agree. We shouldn't be tied to any compiler, not even gcc; and
we mustn't be tied to msw.

> GC>   unit tests
> GC> There are forty-four programs in this category, and more will be added.
> GC> They make it easy to experiment with different implementations, by
> GC> shortening the compile-link-test cycle for each isolated feature, so
> GC> each should build as quickly as possible. Using shared libraries here
> GC> would increase link time dramatically,
> 
>  Sorry, this is exactly contrary to my experience. Linking statically is
> usually quite slow because the linker needs to extract the required
> functions from the static library and create a usually huge final
> executable. Linking against a DLL is much faster and, even better, you can
> avoid relinking entirely if you've changed just the library and not the
> test itself.

Interesting...perhaps I'm mistaken here.

What I had in mind was not creating a static library and linking every
unit test with it. Instead, I just compile object files and link them
directly. That sounds like the same thing (with only the 'ar' step
omitted), but there's an important difference. Consider a library that
includes many objects:

  libx.a libx$(SHREXT): 1.o 2.o 3.o [...ninety-five more...] 99.o

and a test that requires only a couple of those objects:

  foo_test.exe: foo.o foo_test.o obj14.o obj73.o

Building foo_test that way is fast. But if I write the rule as

  foo_test.exe: foo.o foo_test.o libx$(SHREXT)

then 'make foo_test' requires linking the whole library, and I've found
that to be much slower.

Speed varies by context, though. If building the entire unit-test suite
is the most common task, then I think your analysis applies, and a
shared library may save a lot of overhead. OTOH, most typically I do
something like this:

  make foo_test; ./foo_test
Oops, compiler error--fix it and rerun:
  make foo_test; ./foo_test
Oops, a test failed--fix the code and rerun:
  make foo_test; ./foo_test
OK, that worked. Now add another test case and rerun:
  make foo_test; ./foo_test
Repeat the same steps, adding more test cases. I might spend hours
iterating this process for 'foo_test'. Anything I can do to make that
development cycle faster is a big help. And linking directly with just
a small handful of objects seems like the fastest way.

>  Also, 44 unit tests lunk statically can consume quite a lot of disk space
> and space saving allowed by using the shared libraries can be useful in
> this situation.

44 unit tests add up to 78.7 Mbytes right now. That is a lot, if all
of them are to be built--the case where, as you point out, a shared
library would save both time and space.

OTOH, if the hypothetical 'foo_test' is of average size, it's only
78.7 / 44 or about 2 Mbytes, in the important case where I spend all
day refining 'foo_test' in isolation.

Can you think of a good way to get the advantage of both approaches?
I don't immediately see how to do that without maintaining two
different sets of unit-test targets in a makefile, and making sure
they both remain valid. We want to unify these rules somehow:

  foo_test.exe: foo.o foo_test.o obj14.o obj73.o
  foo_test.exe: foo.o foo_test.o libx$(SHREXT)

What about this?

  foo_test.exe: foo.o foo_test.o $(test_library) obj14.o obj73.o

Then, would building with

  make test_library='libx$(SHREXT)'

find all unresolved symbols in the shared library, and ignore the
explicitly-named object files?

> GC> The production system is to be built only without defining
> GC> LMI_USE_SO_ATTRIBUTES.
> 
>  Ok, this is an important thing to know. I still am not sure if this is a
> final decision or subject to change in the future?

For the excellent reasons you point out, we'll have to do it differently
in the future. I said something more general than I meant: I meant to
say only that the people who build our production system and distribute
it to our end users should continue to build it as they always have,
without shared-library decorations. Once we find another compiler that
we can actually use (borland is impossible, but we might use msvc for
msw, or como for any platform), we'll need to make this change. But
right now I don't have the time to spend on that, so I won't personally
make the change yet.

I hope I've expressed myself more clearly now, and thanks as always for
disagreeing, because that helps me learn how to make things better.