Re: Rethinking octave_idx_type

[Rik]

On 11/25/2016 10:28 AM, address@hidden wrote:

Subject:

Rethinking octave_idx_type

From:

"John W. Eaton" <address@hidden>

Date:

11/25/2016 09:21 AM

To:

Octave Maintainers List <address@hidden>

CC:

address@hidden

List-Post:

<mailto:address@hidden>

Content-Transfer-Encoding:

7bit

Precedence:

list

MIME-Version:

1.0

Message-ID:

<address@hidden>

Content-Type:

text/plain; charset=utf-8; format=flowed

Message:

2

Currently we define octave_idx_type to be the same as the integer type used by BLAS, LAPACK, and other functions that have a "Fortran" interface.  By default, this means 32-bit integers even on 64-bit systems and arrays are limited to less than 2^31 elements.  This affects all of Octave, even if you don't actually want to pass large arrays to the functions that actually have these size limits.

Instead, it seems that we could define octave_idx_type to be ssize_t (or ptrdiff_t, I think they are equivalent in practice).  Then things like fread, fwrite, or simple element-by-element array operations that don't require BLAS or LAPACK functions could work on larger arrays.

Then we would also define something like fortran_integer_type for the Fortran-style function interfaces and check array sizes and limits when we actually call the Fortran-style functions.

Does anyone see a reason NOT to do this?

Are there any performance implications?  Most people will still not need regular access to matrices with more than 2^31 elements.  By potentially doubling the pointer size, you could increase the critical processor L1, L2 cache requirements.  If there are still systems where Octave is run that have 32-bit busses then there is another big hit because each pointer will require two fetches.  In general, the march of progress is towards 64-bit systems and bigger caches so this will sort itself out, but have we crossed over the point where 50% of Octave systems are 64-bit?

Since one can compile with --enable-64 today it is possible to check some of these assertions about the resident set size of the binary.  And separately the performance might be checked with "tic; __run_test_suite__; toc"

Other than performance, I think it does make sense to become more compatible with the STL.  I don't think we have to switch to an unsigned type though.  2^63 = 9 x 10^18 elements and if each element is an IEEE double that would be 70 exabytes.  I don't know of any personal PCs that would need the extra bit of addressing from an unsigned type because they actually have 140 exabytes of memory.

--Rik