[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
RE: precision of floor(a/b)
From: |
Tim Rueth |
Subject: |
RE: precision of floor(a/b) |
Date: |
Mon, 5 Apr 2010 17:33:51 -0700 |
Yes, that fixed it, thanks! Learning something new every day.
--Tim
> -----Original Message-----
> From: Ben Abbott [mailto:address@hidden
> Sent: Monday, April 05, 2010 5:01 PM
> To: address@hidden
> Cc: address@hidden
> Subject: Re: precision of floor(a/b)
>
>
> On Apr 5, 2010, at 7:52 PM, Tim Rueth wrote:
>
> > If I do floor(6/1), I correctly get 6. If I do
> floor(0.6/0.1), I get 5. If I do floor((0.6/0.1 +
> 0.00000001)) I correctly get 6.
> >
> > I assume this has to do with the precision of the division?
> I read in the manual that Octave stores numbers as double
> precision, so this surprises me. What's the best way to deal
> with this imprecision so I can always get the correct answer,
> or is something else going on here?
>
> Will floor(0.6/0.1*(1+eps)) work for you?
>
> Ben
>
>
- precision of floor(a/b), Tim Rueth, 2010/04/05
- Re: precision of floor(a/b), Ben Abbott, 2010/04/05
- RE: precision of floor(a/b),
Tim Rueth <=
- RE: precision of floor(a/b), John W. Eaton, 2010/04/05
- Re: precision of floor(a/b), Scott Carter, 2010/04/05
- RE: precision of floor(a/b), Tim Rueth, 2010/04/06
- RE: precision of floor(a/b), CdeMills, 2010/04/06
- Re: precision of floor(a/b), Martin Helm, 2010/04/06
- Re: precision of floor(a/b), Dr.-Ing. Torsten Finke, 2010/04/06
- Re: precision of floor(a/b), Jaroslav Hajek, 2010/04/06
- RE: precision of floor(a/b), Tim Rueth, 2010/04/07
Re: precision of floor(a/b), Scott Carter, 2010/04/05