[Octave-bug-tracker] [bug #54788] Implementation of memmapfile

[Guillaume]

URL:
  <https://savannah.gnu.org/bugs/?54788>

                 Summary: Implementation of memmapfile
                 Project: GNU Octave
            Submitted by: gyom
            Submitted on: Fri 05 Oct 2018 01:01:02 PM UTC
                Category: Octave Function
                Severity: 3 - Normal
                Priority: 5 - Normal
              Item Group: Feature Request
                  Status: None
             Assigned to: None
         Originator Name: Guillaume
        Originator Email: 
             Open/Closed: Open
         Discussion Lock: Any
                 Release: dev
        Operating System: Any

    _______________________________________________________

Details:

Matlab has a class to perform memory-mapping on a file from disk:

https://www.mathworks.com/help/matlab/ref/memmapfile.html

I attach two files, memmapfile.m and __memmapfile_handle__.m, as an initial
attempt at implementing this functionality in Octave and would welcome
feedbacks and help to finish it.

As far as I can see, the code in memmapfile.m is feature complete. The
internal handle class in __memmapfile_handle__.m is where further work is
required. An extra handle class has to be defined because memmapfile is a
value class and, given that the memory mapping can take place at any time
including when the data is first accessed for reading, a mechanism has to be
in place to store the mapping pointer. At the moment, memory mapping isn't
actually taking place and fread/fwrite is used to mock the real behavior (I
know, disappointing!). For example, instead of having:
  this.handle = fopen (filename, writable);
there should be a call:
  [this.handle, err, msg] = __mmap__ (filename, length, offset, writable);
to a low-level (oct-file or core?) __mmap__ that would open/mmap/close the
file and return the address of the mapping as a double. This function would
also have to deal with page size for the offset parameter so it would probably
need to have extra output variables containing the eventually modified offset
and length of the mapping.
Same thing with munmap instead of fclose. I have a toy example showing how
memory mapping would work but I need help/guidance in specifying and writing
the DEFUN_DLD function(s).

Other aspects to consider:
* On Windows, CreateFile/CreateFileMapping/MapViewOfFile will have to be used
instead of mmap:
  https://docs.microsoft.com/en-us/windows/desktop/memory/file-mapping
* There are problems with the 'end' keyword. One is a separate bug #54783 that
prevents m.Data(end) = 0; to work. But, more generally, the use of the 'end'
keyword is problematic here in the context of memory mapping because its
evaluation will try to access to all of the mapped data. Until a satisfactory
solution is found, I would document this and discourage its use.
* I noticed that some of the unit tests for errors do not report a bug if an
error is not raised.
* Care will have to be taken if a memmapfile object is copied or saved into a
mat-file such that the mapping is deleted first.
* The local function get_dimensions() in memmapfile.m could be cached so that
its evaluation is only done once for a given mapping but it is very light
anyway so I thought it was preferable than adding extra private properties.

Thanks to jwe and carandraug for their help.



    _______________________________________________________

File Attachments:


-------------------------------------------------------
Date: Fri 05 Oct 2018 01:01:02 PM UTC  Name: memmapfile.m  Size: 22KiB   By:
gyom

<http://savannah.gnu.org/bugs/download.php?file_id=45149>
-------------------------------------------------------
Date: Fri 05 Oct 2018 01:01:02 PM UTC  Name: __memmapfile_handle__.m  Size:
3KiB   By: gyom

<http://savannah.gnu.org/bugs/download.php?file_id=45150>

    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?54788>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/