Re: [ft-devel] [PATCH RFC] src/sfnt/ttsbit0.c: fix broken pointer overflow checks

[Xi Wang]

On 1/25/13 11:48 AM, Sean McBride wrote:
> And clang can detect signed overflow at runtime if you enable the debug flag 
> "-fsanitize=undefined" described here:

Yeah, if you have input data to trigger the overflows.

> Could be useful to build freetype with that flag and see what it finds...

I do find some signed overflows, using a homemade static checker.

src/cache/ftcbasic.c:360
src/cache/ftcbasic.c:695

  if ( (FT_ULong)(type->flags - FT_INT_MIN) > FT_UINT_MAX )

gcc optimizes this check away.  Try the simplified code.

#include <stdint.h>
#include <limits.h>
#define FT_Int32        int32_t
#define FT_ULong        unsigned long
#define FT_INT_MIN      INT_MIN
#define FT_UINT_MAX     UINT_MAX
void bar(void);
void foo(FT_Int32 flags)
{
        if ( (FT_ULong)(flags - FT_INT_MIN) > FT_UINT_MAX )
                bar();
}

$ gcc -S -o - -O2 t.c
foo:
.LFB0:
        .cfi_startproc
        rep
        ret
        .cfi_endproc

There is another possible overflow that I don't understand.

src/raster/ftraster.c:3052

  if ( e1 > e2 || ...)

Is e1 > e2 only possible on signed overflow?

- xi