Re: [Bug-gnulib] xalloc.h proposed fix to detect potential ptrdiff_t overflow

[Bruno Haible]

Paul Eggert:
>       When checking for size_t overflow, check for ptrdiff_t overflow too.

I disagree with this for four reasons:

1) This patch makes it impossible to allocates arrays of more than 2 GB,
   on 32-bit machines, even when the OS allows it.

   This is not a theoretical problem; PCs with 1 GB or 2 GB of RAM are
   become popular right now, and when a GNU program cannot allocate
   a matrix of size 20000x20000 because of an arbitrary check in xmalloc,
   it'd be a pity.

2) The standards have designated size_t, not ptrdiff_t, as the type that
   measures the extent of a single block of memory. Lots of APIs taking
   a size_t argument exist, from malloc() to mbrtowc(). No APIs other
   than morecore() take a ptrdiff_t.

   malloc() has nothing to do with ptrdiff_t.

3) We should reduce, not increase, the use of 'ptrdiff_t'.

   You pointed out that

       When you are subtracting two pointers into the same
       object, the result is of type ptrdiff_t; but if the resulting value
       does not fit in ptrdiff_t, the behavior is undefined.

   This means that ptrdiff_t is ill-defined by design, if it has the same
   width than size_t. The ideal ptrdiff_t would be at least 1 bit wider
   than size_t.

4) The problem you are trying to solve, namely undefined behaviour resulting
   from "ptr1 - ptr2" in some cases, can practically be solved by two
   alternative, much simpler, techniques:

     a) Avoid pointer subtraction entirely. Always use indices. This not
        only allows indices from 0 to SIZE_MAX to be represented without
        problem, it also avoids the division by the element type size
        (which is not always a power of 2).

     b) Do pointer subtraction only when you a priori know which is the
        bigger and which is the smaller pointer, then cast the result to
        size_t:
             (size_t) (ptr1 - ptr2)
        This will work because if sizeof(*ptr1) == 1, the machine code
        for this is just a subtraction, and it will do the right thing
        despite the standard's "undefined behaviour" wording, whereas
        if sizeof(*ptr1) >= 2, the result will be < SIZE_MAX/2 and
        the behaviour will be well-defined.

In summary: Don't make xmalloc() less useful just because pointer
subtraction is ill-defined on platforms where ptrdiff_t and size_t have
the same size!

Bruno