[Octave-bug-tracker] [bug #42118] COW memory issues when extracting small slices from large arrays

[Jan Dohl]

URL:
  <http://savannah.gnu.org/bugs/?42118>

                 Summary: COW memory issues when extracting small slices from
large arrays
                 Project: GNU Octave
            Submitted by: derjan
            Submitted on: Mon 14 Apr 2014 04:49:55 PM GMT
                Category: Octave Function
                Severity: 3 - Normal
                Priority: 5 - Normal
              Item Group: Crash
                  Status: None
             Assigned to: None
         Originator Name: 
        Originator Email: 
             Open/Closed: Open
         Discussion Lock: Any
                 Release: 3.8.1
        Operating System: GNU/Linux

    _______________________________________________________

Details:

I'm porting a Matlab script to Octave that receives a large amount of data
points (hundrets of thousands to several million) every second. The task of
the script is to extract the interesting part of the data which is smaller by
orders of magnitude (10-100 samples). Those interesting parts are then to be
saved. The script is supposed to run for several hours at a time.

A mockup version of the script is as follows:

WARNING: Do not run this on a machine which runs important processes. It will
very quickly use all available memory which might cause your machine to become
slow or even crash if no limits are in place.


datas = cell(10000, 1);

for i=1:10000
  % Acquire big vector of data
  rxdata = randn(300000, 1);

  % Extract small amount of data
  datas{i} =rxdata(1:10);

  % Uncommenting this circumvents the problem
  %a = rxdata(1);
  %datas{i}(1) = a;

end


As said before, this script will very quickly use up all available memory and
eventually cause Octave to crash. However, if the process is aborted before
the crash, 'whos' will just show a very small memory usage by the existing
variables which I expected.

>From IRC I've got the idea that this might be a COW problem and datas{i} =
rxdata(1:10) actually creates a reference to the whole rxdata-array and keeps
it from going out of memory. Hence, not even clearing rxdata after each run
will cause the problem to go away. Overwriting one element of the copied slice
with a copy of itself will however trigger the copy mechanism, remove the
reference and the program works as expected.

Problem was verified with self-built Octave 3.8.1. as well as 3.6.2. from
Debian Stable.




    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?42118>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/