Re: Open MPI keeps references to GCC, GFortran, etc.

[Dave Love]

Ludovic Courtès <address@hidden> writes:

> Hi,
>
> Dave Love <address@hidden> skribis:
>
>> Ludovic Courtès <address@hidden> writes:
>>
>>> Hello,
>>>
>>> Open MPI retains references to GCC, GFortran, etc., which significantly
>>> increases its closure size.
>>
>> My query about cycles from separating the lib output was from looking at
>> basically this.  There should be a runtime package for compute nodes and
>> a development one (as "openmpi" and "openmpi-devel" in Fedora).
>
> Interesting.  It’s not a “should” though IMO, in the sense that we add
> additional inputs only when we have a good reason to do so.

I think I was misunderstanding.  Is the intention actually to get rid of
dependencies on the compilers?  (I assume that should apply to C as well
as Fortran.)  I guess that's arguable, but at least the compilers are
used by mpicc etc., and Fedora and Debian development packages depend
on, or recommend the compilers.

Anyhow, surely the first things to purge are perl, python, gdb, and
others that are currently in the closure.

Looking at the packaging more closely, I think it needs, or should have,
various changes.  --enable-static clobbers dynamically-loaded MCA
components, which I think is is a non-starter.  One question I have is
why are builtin atomics turned on?  They normally aren't, and I don't
know what the consequences are.

You can reduce the closure with other changes I made to bring it roughly
into line with Debian and Fedora and how I'd build it otherwise.
Removing the obsolete vampirtrace support, and the devel headers (which
are only for building external MCA components, which I've never seen
done), and replacing the valgrind integration with the library wrapper,
brings the "size" output down to:

  store item                                                       total    self
  /gnu/store/dws3a11p4s2qhnmapc4p1nm7g36hr3p4-openmpi-1.10.7         438.6     
9.7   2.2%
  /gnu/store/vcx02xlp6s92bxrcafnpw7hnqa4kc32b-gfortran-5.4.0         262.9   
105.4  24.0%
  /gnu/store/zlyn8z138s2vv86bdx9d72kchjczv0xn-gcc-5.4.0              198.2    
92.1  21.0%
  /gnu/store/hdc5cqig58rc1093rx1l6ii8psdlgrav-hwloc-1.11.7-nogui      90.0     
2.9   0.7%
  /gnu/store/padj9dg2p0dl82fhyhp73z2nqs5pyfrv-ncurses-6.0             74.3     
5.7   1.3%
  /gnu/store/sg7rcm6s5iqjsmnviq9gcjw85bjzf5cn-mpc-1.0.3               73.2     
0.4   0.1%
  /gnu/store/dxmkc0cq22dqnvidi0lqf1acrifn8l02-gfortran-5.4.0-lib      73.0    
34.5   7.9%
  /gnu/store/w1mrskd2ddgvkr727r9241g8dlkf0rlf-gfortran-5.4.0-lib      73.0    
34.5   7.9%
  /gnu/store/p5vnjcz57mgg20461c4ml71chy5b79rq-mpfr-3.1.5              72.8     
1.6   0.4%
  /gnu/store/62d6n9qw68m37xavyv79g6nzpij5795k-gmp-6.1.2               71.2     
2.6   0.6%
  /gnu/store/yd7bplsvf9nj72wn2z6n38rq9hfmjgd9-zlib-1.2.11             69.0     
0.4   0.1%
  /gnu/store/4xkbjwf1iqmcz5is5apzlj6qhk47bz7d-numactl-2.0.11          68.9     
0.3   0.1%
  /gnu/store/9sgscigfvrf6wklirgnmhnjqhajljhbv-libpciaccess-0.13.5     68.7     
0.1   0.0%
  /gnu/store/m4svrffkzxg88av9rv4gzcpzl2cnvg8r-gcc-5.4.0-lib           68.6    
30.1   6.9%
  /gnu/store/dhc2iy059hi91fk55dcv79z09kp6500y-gcc-5.4.0-lib           68.6    
30.1   6.9%
  /gnu/store/58nhnjff2yahyxa1yjh2m8k93fb955z5-bash-4.4.12             50.9     
5.4   1.2%
  /gnu/store/aiazy6r7flh8jk8fdaxp049g5lvsnkix-readline-7.0            45.5     
1.3   0.3%
  /gnu/store/zhg9ds8madp5nk3pb9krabagr1vffygm-ncurses-6.0             44.2     
5.7   1.3%
  /gnu/store/yz9yzv9nlvjpyw4vzjklhgk8dpl7hd7f-zlib-1.2.11             38.9     
0.4   0.1%
  /gnu/store/ybpgv1v7606xw7mafda66w10hiynpiw2-glibc-2.25              38.5    
37.1   8.5%
  /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25              38.5    
37.1   8.5%
  /gnu/store/02426nwiy32cscm4h83729vn5ws1gs2i-bash-static-4.4.12       1.4     
1.4   0.3%
  total: 438.6 MiB

I don't understand why gfortran and gfortran-lib are so large anyway --
they seem to duplicate C stuff, and gcc-lib bundles things I wouldn't
expect it to.  The RHEL gcc-5 equivalents are ~10 and 2 MB intrinsically.

I'll send the changes I have for consideration.

If you don't mean you want to remove the compiler dependencies, do you
mean outputs, rather than inputs above?  I was assuming a development
package depends on compilers.  Then the reason for separation is that
compute nodes, the way I've run systems, don't generally have a
development environment past what's required for dkms.  Also a user
installing some program in home shouldn't pull in the build environment,
which might clash with what they actually want.

I assume the store is intended to be on a shared filesystem which
compute nodes don't duplicate, which helps with space, but I don't think
that should be required.  The stateless systems I've set up used a
separate compute node image which was much smaller than the login node
one by omitting non-runtime rpms.

> For
> example, in most cases, a “doc” output would only save you a few KiB of
> man pages, representing .1% of the package size, so we don’t add a “doc”
> output in those cases.  This is different from the Fedora/Debian
> approach where packages are systematically split in several outputs.
> The “Submitting Patches” section of the manual mentions this.

I don't remember the Debian guidelines, but Fedora says to split out doc
files if they're "large" in total (~1MB?).  There is a requirement for a
separate -devel package for libraries (and for things like MPI with
compiler wrappers), and debug output is automatically separated.

> In the case of Open MPI, adding a “lib” output wouldn’t help: “out”
> would still refer to gfortran via ‘ompi_info’, and “lib” would refer to
> gfortran via opal_config.h.  Hence my suggestion to simply remove the
> reference to gfortran & co.

Yes, I now understand what the cycles are (thanks!), but I don't
understand why some of the strings are used in the source, or even what
the compiler paths are meant to be -- the package build (cross?)
compiler or the one invoked by mpifort?  [Cross-compilation might just
involve optimizing for a different micro-architecture, of course.]
Perhaps I should ask on the openmpi list, but I doubt I'll get far.

I think it's fine to remove the path from the point of view of (not)
breaking things, and other information strings could go, like overall
romio configure options.  (The relevant info about romio from ompi_info
is just the filesystem types supported.)

>> I'm confused what the wrappers are doing,
>
> The two commands return the actual compiler names, not the names of
> wrappers:
>
> --8<---------------cut here---------------start------------->8---
> $ $(guix build openmpi)/bin/ompi_info

I actually meant mpicc et al, which invoke the compiler.  I don't think
ompi_info should be in any separated runtime package, although it is in
RHEL.

> AIUI, this is “purely informational”.  Replacing the absolute file names
> with just the base name won’t break anything.  Except that there’s
> always the possibility of scripts or tools out there parsing the output
> of ‘ompi_info’.

I'd expect them to use "mpifort --showme" etc. for any configuration,
not ompi_info -- especially as it's not clear what that path means, or
why it matters.

> Thus my question was: do people really parse the output
> of these commands, or can we safely assume that third-party programs
> don’t rely on these absolute file names?

I'd say they shouldn't, and I can't think of any that do that I've
built, but I haven't tried to search lots of source.

>> I was intending to look at parameterizing the build on gfortran version,
>
> I suppose you could to:
>
>   (define openmpi-with-gfortran7
>     (package
>       (inherit openmpi)
>       (name "openmpi-gfortran7")
>       (inputs `(("gfortran" ,gfortran-7)
>                 ,@(alist-delete "gfortran" (package-inputs openmpi))))))

Right.

> (That said, if the .mod files are compatible among gfortran versions, it
> probably doesn’t make sense to do this.)

But they're not compatible, which is a real problem I'd have tried hard
to avoid if I was a maintainer.  All the gfortran versions I can see in
Guix have incompatible module formats.  It would probably be good to
provide symlinks like "gfortran-6", and even "gcc-6", as Debian does.

By the way, I don't want to be an HPC bigot, but HPC requirements seem
to be largely a superset of most others, and applicable in other areas.

Hope that helps a bit.