Re: New module nproc

[Glen Lenker]

On Thu, Mar 26, 2009 at 12:42:56AM +0100, Bruno Haible wrote:
> > +/* Return the total number of processors.  The result is guaranteed to
> > +   be at least 1.  */
> > +unsigned long int
> > +num_processors (void)
> > +{
> > +#ifdef _SC_NPROCESSORS_ONLN
> > +  long int nprocs = sysconf (_SC_NPROCESSORS_ONLN);
> > +  if (0 < nprocs)
> > +    return nprocs;
> > +#endif
> 
> It appears that this code is right according to POSIX, but does not catch
> the entire reality on Linux. (A comment in libgomp/config/linux/proc.c says:
> "Count only the CPUs this process can use.")
> And on mingw, a different API should be used. See the implementation of the
> function omp_get_num_procs() in libgomp, part of GCC:
> - for POSIX systems:
>   
> http://gcc.gnu.org/viewcvs/trunk/libgomp/config/posix/proc.c?revision=138078&view=markup
> - for Linux:
>   
> http://gcc.gnu.org/viewcvs/trunk/libgomp/config/linux/proc.c?revision=138078&view=markup
> - for mingw:
>   
> http://gcc.gnu.org/viewcvs/trunk/libgomp/config/mingw32/proc.c?revision=138078&view=markup
> 
> This leads to the question: Why not use the AC_OPENMP macro, and then use the
> following?
> 
> unsigned long int
> num_processors (void)
> {
> #ifdef _OPENMP
>   return omp_get_num_procs ();
> #else
>   ... existing implementation ...
> #endif
> }
> 
> This would get the count right on Linux, mingw, and on POSIX systems, i.e.
> nearly everywhere.

Hi Bruno,

When we first looked at parallelizing sort, it was actually our first
choice to use glthreads. We were on the fence though between pthreads
and glthreads because we were concerned that Pth might complicate our
changes if glthreads defaulted to Pth in the absence of pthreads. We
eventually decided to just use pthreads when we posted to coreutils
and Jim Meyering said the he would prefer pthreads. You are right, Its
an easy change (we changed the two letters before), and to be honest I
don't see anything wrong with using glthreads instead of just
pthreads.

We also considered using libgomp's omp_get_num_procs before, but
decided to go along with just sysconf. I looked at the Linux
implementation and from what I could understand, saw that libgomp
would either use sysconf itself, or return the number of cpus
available its affinity masks. Since we don't use libgomp anywhere else
in the code, I didn't think it would make sense to use their use
libgomp to detect the number of processors and then just create the
threads myself, since even if it did whatever information it had on
running tasks and processors, it probably wouldn't be different from
the number returned from sysconf.

As far as mingw is concerned, I was hoping that someone else could
help with this (I don't have a Windows machine). How easy would it be
to get the nproc module to work under Windows?

> Also, the omp_get_num_procs() function has the advantage that you can
> influence it through an environment variable, so that users have the ability
> to let multithreaded programs access only, say, 3 out of 4 CPU cores, if
> a responsive machine is more important to them than fast execution.

I like this. This would be a very useful ability, and I am sure there
would be a lot of other perks for using OpenMP instead of using
pthreads. I am just not sure that this should be the main reason for
including OpenMP into the code.

Ultimately we didn't want to include OpenMP and not use it except
checking for the number of threads. Is this not as big of a deal as we
are making it?

-- 
Glen Lenker