Re: segfault in gmp p-1 factorer

[Alexander Kruppa]

Kevin Ryde wrote:
> 
> Alexander Kruppa <address@hidden> writes:
> >
> > I've written a simple P-1 factorer called gpm1. When I try to factor a
> > large number on a K6-3 450Mhz, the program segfaults. The segfault
> > occurs in mpn_sqr_basecase. The program runs fine on a P-II 400 Mhz, and
> > it runs fine for smaller numbers.
> 
> Perhaps you're running out of stack space.  gmp uses the stack for
> some temporaries, you can configure with --disable-alloca to stop it
> doing that.

I dont think that's it. The ulimit is the same on the Pentium-II and on
the K6-3, raising the ulimit to 32 MB did not help, either. I get the
segfault even with a ~400 decimal number as input, and the stage 1 of
P-1 uses very little memory, in case of the 400 digit number maybe a few
KB, but not MB.

But I found out something different. The code that segfaults was built
with a gmp-mparam.h generated by tuneup, which has 

#define KARATSUBA_SQR_THRESHOLD     65

If I build gmp with the default gmp-mparam.h, which has 

#define KARATSUBA_SQR_THRESHOLD  
34                                            

the code works without segfault.

I'll have to do some more testing next week, I'll keep you updated.

Ciao,
  Alex.