Re: Finding Octave's memory use breakdown

[Daniel J Sebald]

On 04/25/2016 11:31 PM, LachlanA wrote:
> John W. Eaton wrote
> > On 04/24/2016 11:42 PM, LachlanA wrote:
> > > Is there a way to find out where Octave is using memory?  I often find
> > > that
> > > Octave is using say 2GB, but "whos" shows only a few hundred megabytes in
> > > variables.
> >
> > What are you looking at to determine that the overall memory usage is
> > 2GB?  If I start Octave (running the GUI and built from current
> > development sources with debug symbols) it looks to me like it takes
> > about 100 MB to run, with 80 MB of that being shared.  That's the
> > resident and shared memory sizes reported by top.  The total virtual
> > size is a little over 1 GB.  That doesn't seem terribly large.  Are you
> > concerned that it is?
>
> I often deal with large data sets.  I just ran an example which gave
>
> debug>  whos
> Variables in the current scope:
>
>     Attr Name                  Size                     Bytes  Class
>     ==== ====                  ====                     =====  =====
>    a     argn                  4x4                         16  char
>          capFactor1         3000x365x31             271560000  double
>      f   cor_solar          3000x17520                  70084  double
>      f   data               3000x365x48             210240000  single
>          disconnected        365x3000                 1095000  logical
>          first_solar_data      1x1                          8  double
>          first_solar_day    3000x1                      24000  double
>          gap                   1x1                          8  double
>      f   ind                   1x3000                      24  double
>          k                     1x1                          8  double
>          last_solar_day     3000x1                      24000  double
>      f   meta                  1x1                     164158  struct
>          not_all               1x3000                    3000  logical
>      f   s                     1x1                  655798080  struct
>          s_su                  1x3000                   24000  double
>          solar_correction   3000x365x31             271560000  double
>          solar_user_range      1x3000                      24  double
>      f   su                    1x3000                   24000  double
>          u                     1x15                        60  double
>
> Total is 174126036 elements using 1410586470 bytes

64 bit?

1410586470 - 174126036 * 8 = 17578182

or about 1.3% overhead for internal book keeping.  Am I thinking of that 
correctly?



> There was a similar amount in another couple of stack frames, though much of
> it would point to the same "rep"s of octave_values.  After "dbquit; clear
> all", top reports a virtual size of 2,685,748 bytes (down from 3.7G) and an
> RSS of 1.032G (down from 2G), with 52792 shared.

I don't have the GUI built on my system, so in octave-cli I've done:

x = rand(1,174126036);

and memory jumps to 1.3 GiB.  Then

clear all

and memory goes back down to 9.6 MiB, which is where it started out.

You are thinking that somehow the act of parsing code is consuming 
memory?  A possible memory leak?  As I see it, parsing should be a 
temporary consumption of memory, that eventually gets cleaned up with 
successive returns to the command line (i.e., end of processing).

Can we come up with a script to compare across systems?  Something 
simple that illustrates what seems strange to you?  For example, I'd 
think that "plot" parses a lot of code, so I've done:

for i=1:20
  figure(i);
  plot(1:5000);
end
close all

and the memory goes from 9.6 MB to 21.1 MB and doesn't come back down 
with the close all, and whos shows but a single variable.  It might be 
that the extra overhead is internal properties for the figures still 
hanging around.  In fact, running the above script again still results 
in 21.1 MB, so how about changing to:

for i=21:40
  figure(i);
  plot(1:5000);
end
close all

Nope, still stays at 21.1 MB without increasing, so I doubt that theory 
(internal figure properties) is true.  Maybe it is parse tree, or maybe 
a leak, don't know.  But in that regard Rik and John have been working 
on memory leaks as of late so it would have to be the most recent 
version of code to work from.

Dan



> It may be that some of that has gone back to the malloc/new pool but not
> been returned to the OS, possibly because of fragmentation, in which case
> the RSS would probably drop without performance penalty if another
> application grabbed it.  However, I have no idea how much has been consumed
> by the parse tree of the scripts I was running, or any other overheads.  I'm
> guessing it is much less than 1GB, but that is just a guess.
>
> If there is no way to find out where the memory is going, would there be
> value in extending  whos  to report on memory use other than variables, such
> as those I listed above?
>
> (As an aside, I entirely agree that printf etc. don't need to be moved.  I
> was just thinking that the only benefit of converting them to a .m file was
> memory, which reminded me of this issue.)
>
> Cheers,
> Lachlan
>
>
>
> --
> View this message in context: 
> http://octave.1599824.n4.nabble.com/Finding-Octave-s-memory-use-breakdown-tp4676477p4676506.html
> Sent from the Octave - Maintainers mailing list archive at Nabble.com.

--

Dan Sebald
email: daniel(DOT)sebald(AT)ieee(DOT)org
URL: http://www(DOT)dansebald(DOT)com