Re: Single/Double precision equality

[Rik]

On 09/26/2014 04:35 PM, address@hidden wrote:

Subject:

Single/double precision equality question

From:

Daniel J Sebald <address@hidden>

Date:

09/26/2014 04:34 PM

To:

address@hidden

List-Post:

<mailto:address@hidden>

Content-Transfer-Encoding:

7bit

Precedence:

list

MIME-Version:

1.0

Message-ID:

<address@hidden>

Content-Type:

text/plain; charset=ISO-8859-1; format=flowed

Message:

6

Should the following equality behavior be explained in "help =="?

format bit
single(0.1)

ans = 0011111110111001100110011001100110100000000000000000000000000000

double(0.1)

ans = 0011111110111001100110011001100110011001100110011001100110011010

single(0.1) == double(0.1)

ans = 0011111111110000000000000000000000000000000000000000000000000000

I guess I'm OK with it.  The test demotes the double to a single.  Try:

double(single(0.1)) == double(0.1)

ans = 0000000000000000000000000000000000000000000000000000000000000000

However, the demoting rule doesn't hold true for all classes.  Try:

format
v1 = pi;
v2 = int8(v1);
v1 == v2

ans = 0

Whatever the case, it probably would be good to have this documented for the user.  Perhaps it is, but I don't know where to look beyond "help ." and "help ==".

Look at section 4.7 of the manual "Promotion and Demotion of Data Types".  I'm sure there's still room for improvement there if you want to reword things.

Basically, when different types are used in a mixed operation such as A + B, then integer trumps single which trumps double.  Thus, single (0.1) == double (0.1) should demote the RHS to single and then do the comparison which equals true.

When doing mixed integer/floating point operations the integer is temporarily promoted to floating point and the final result is then cast back to integer (at least that is the way Matlab does it).  This explains the second example because 3 (int8) is promoted to double which != pi (double).

--Rik