Memleaks or not?

[Reinhold Kainhofer]

Running lilypond on a lot of files in one run, I observe that lilypond's 
memory usage slowly goes up with time, i.e. it seems that lilypond does not 
properly free all memory used for one score, before it starts with the next 
one. 

In particular, running lilypond on all 1010 regtests the output of ps is (as 
time goes on):
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
reinhold 28946  127  4.7 108432 97856 pts/3    R+   19:27   0:01 lilypond 
reinhold 28946 74.4  5.4 121564 110988 pts/3   R+   19:27   0:18 lilypond 
reinhold 28946 72.6  5.9 137168 123036 pts/3   R+   19:27   0:39 lilypond 
reinhold 28946 73.4  6.4 151544 133308 pts/3   R+   19:27   0:59 lilypond 
reinhold 28946 73.8  6.7 159784 139320 pts/3   S+   19:27   1:11 lilypond 
reinhold 28946 72.1  8.1 199532 166512 pts/3   R+   19:27   1:54 lilypond 
reinhold 28946 72.6 11.2 271932 230260 pts/3   R+   19:27   2:28 lilypond 
reinhold 28946 72.0 12.0 316108 247232 pts/3   R+   19:27   2:56 lilypond
reinhold 28946 72.0 12.6 341956 259276 pts/3   R+   19:27   3:25 lilypond
...
reinhold 28946 72.1 15.4 423400 315956 pts/3   S+   19:27   5:35 lilypond
reinhold 28946 72.4 18.8 526744 387140 pts/3   R+   19:27   7:47 lilypond
reinhold 28946 72.5 27.9 747168 572740 pts/3   R+   19:27   9:59 lilypond


Using the memcheck tool of valgrind, there are some lost memory warnings that 
might be leaks. Can anyone confirm that these are really leaks?

However, the leaks reported by valgrind do not explain why lilypond's memory 
increases on average per regtest file by ~650kB VSZ and ~475kB RSS.

All valgrind warnings in this mail are obtained by running valgrind on just 
two files. Most of the reportedly lost memory numbers go up practically linear 
in the number of files processed.



1) In Pango_font::text_stencil we have
    PangoLayout *layout = pango_layout_new (context_);
but after using the layout, we never call g_object_unref (layout).


==28530== 4,080 bytes in 2 blocks are possibly lost in loss record 6,178 of 
6,263
==28530==    at 0x402517B: memalign (vg_replace_malloc.c:581)
==28530==    by 0x40251D8: posix_memalign (vg_replace_malloc.c:709)
==28530==    by 0x43B3546: ??? (in /lib/i386-linux-
gnu/libglib-2.0.so.0.2800.6)
==28530==    by 0x43B4A4C: g_slice_alloc (in /lib/i386-linux-
gnu/libglib-2.0.so.0.2800.6)
==28530==    by 0x43B4B74: g_slice_alloc0 (in /lib/i386-linux-
gnu/libglib-2.0.so.0.2800.6)
==28530==    by 0x432C8C6: g_type_create_instance (in /usr/lib/i386-linux-
gnu/libgobject-2.0.so.0.2800.6)
==28530==    by 0x4309674: ??? (in /usr/lib/i386-linux-
gnu/libgobject-2.0.so.0.2800.6)
==28530==    by 0x430CCE6: g_object_newv (in /usr/lib/i386-linux-
gnu/libgobject-2.0.so.0.2800.6)
==28530==    by 0x430DA3F: g_object_new (in /usr/lib/i386-linux-
gnu/libgobject-2.0.so.0.2800.6)
==28530==    by 0x42224C5: pango_layout_new (in /usr/lib/i386-linux-
gnu/libpango-1.0.so.0.2800.4)
==28530==    by 0x81D02D3: Pango_font::text_stencil(Output_def*, std::string, 
bool) const (pango-font.cc:312)
==28530==    by 0x8292E68: 
Text_interface::interpret_string(scm_unused_struct*, scm_unused_struct*, 
scm_unused_struct*) (text-interface.cc:79)
==28530==    by 0x82932BE: 
Text_interface::interpret_markup(scm_unused_struct*, scm_unused_struct*, 
scm_unused_struct*) (text-interface.cc:95)



2) In includable-lexer.cc we use
  yy_switch_to_buffer (yy_create_buffer (file->get_istream (), YY_BUF_SIZE));
and in Source_file::~Source_file we have "delete istream_;". On the other 
hand, running valgrind with 2 and with 3 .ly files shows that the memory 
recorded as possibly lost really increases.
Still, valgrind reports:


==28530== 118,852 bytes in 18 blocks are possibly lost in loss record 6,257 of 
6,263
==28530==    at 0x402641D: operator new(unsigned int) 
(vg_replace_malloc.c:255)
==28530==    by 0x44B89F7: std::string::_Rep::_S_create(unsigned int, unsigned 
int, std::allocator<char> const&) (in /usr/lib/i386-linux-
gnu/libstdc++.so.6.0.14)
==28530==    by 0x44BAC33: char* std::string::_S_construct<char const*>(char 
const*, char const*, std::allocator<char> const&, std::forward_iterator_tag) 
(in /usr/lib/i386-linux-gnu/libstdc++.so.6.0.14)
==28530==    by 0x44BAD20: std::basic_string<char, std::char_traits<char>, 
std::allocator<char> >::basic_string(char const*, unsigned int, 
std::allocator<char> const&) (in /usr/lib/i386-linux-gnu/libstdc++.so.6.0.14)
==28530==    by 0x44B2B27: std::basic_istringstream<char, 
std::char_traits<char>, std::allocator<char> 
>::basic_istringstream(std::string const&, std::_Ios_Openmode) (in 
/usr/lib/i386-linux-gnu/libstdc++.so.6.0.14)
==28530==    by 0x824A340: Source_file::get_istream() (source-file.cc:163)
==28530==    by 0x8136886: Includable_lexer::new_input(std::string, Sources*) 
(includable-lexer.cc:93)
==28530==    by 0x814B3A5: Lily_lexer::new_input(std::string, Sources*) (lily-
lexer.cc:269)
==28530==    by 0x82D26AF: Lily_lexer::yylex() (lexer.ll:305)
==28530==    by 0x82E4599: yylex(YYSTYPE*, Input*, void*) (parser.yy:2784)
==28530==    by 0x82D854B: yyparse(void*) (parser.cc:2486)
==28530==    by 0x82E3880: Lily_parser::do_yyparse() (parser.yy:2557)
==28530==    by 0x814D089: Lily_parser::parse_file(std::string, std::string, 
std::string) (lily-parser.cc:120)




3) There are many, many warnings about possibly lost memory in 
pango_layout_get_lines calls, but I don't see how they can be real memory 
leaks from the pango docs. Still, the numbers go up linearly in the number of 
files...

==28530== 5,032 (512 direct, 4,520 indirect) bytes in 2 blocks are definitely 
lost in loss record 6,200 of 6,263
==28530==    at 0x402695A: realloc (vg_replace_malloc.c:525)
==28530==    by 0x42E4118: ??? (in /usr/lib/i386-linux-
gnu/libfontconfig.so.1.4.4)
[...]
==28530==    by 0x421BE52: pango_itemize_with_base_dir (in /usr/lib/i386-
linux-gnu/libpango-1.0.so.0.2800.4)
==28530==    by 0x4224557: ??? (in /usr/lib/i386-linux-
gnu/libpango-1.0.so.0.2800.4)
==28530==    by 0x4226780: pango_layout_get_lines (in /usr/lib/i386-linux-
gnu/libpango-1.0.so.0.2800.4)
==28530==    by 0x81D0303: Pango_font::text_stencil(Output_def*, std::string, 
bool) const (pango-font.cc:314)
==28530==    by 0x8292E68: 
Text_interface::interpret_string(scm_unused_struct*, scm_unused_struct*, 
scm_unused_struct*) (text-interface.cc:79)
==28530==    by 0x82932BE: 
Text_interface::interpret_markup(scm_unused_struct*, scm_unused_struct*, 
scm_unused_struct*) (text-interface.cc:95)



4) In all-font-metrics.cc we have 
  PangoFontMap *pfm = pango_ft2_font_map_new ();
  pango_ft2_fontmap_ = PANGO_FT2_FONT_MAP (pfm);
And in All_font_metrics::~All_font_metrics we have:
  g_object_unref (pango_ft2_fontmap_);

Still, valgrind reports that pango_ft2_fontmap_ is possibly lost, and the 
number of bytes goes up linearly in the number of files...

==30677== 32,768 bytes in 2 blocks are possibly lost in loss record 6,899 of 
6,919
==30677==    at 0x4026864: malloc (vg_replace_malloc.c:236)
==30677==    by 0x424C7BC: ??? (in /usr/lib/i386-linux-
gnu/libfreetype.so.6.6.2)
==30677==    by 0x425136A: ft_mem_qalloc (in /usr/lib/i386-linux-
gnu/libfreetype.so.6.6.2)
==30677==    by 0x42513C2: ft_mem_alloc (in /usr/lib/i386-linux-
gnu/libfreetype.so.6.6.2)
==30677==    by 0x42528C2: FT_New_Library (in /usr/lib/i386-linux-
gnu/libfreetype.so.6.6.2)
==30677==    by 0x424CBA8: FT_Init_FreeType (in /usr/lib/i386-linux-
gnu/libfreetype.so.6.6.2)
==30677==    by 0x41E82DD: pango_ft2_font_map_new (in /usr/lib/i386-linux-
gnu/libpangoft2-1.0.so.0.2800.4)
==30677==    by 0x8063731: All_font_metrics::All_font_metrics(std::string) 
(all-font-metrics.cc:48)
==30677==    by 0x80661B5: ly_reset_all_fonts() (all-font-metrics-
scheme.cc:29)



5) In relocate.cc (line 50) we use putenv (where we can't deallocate the 
memory). Is there any reason why we can't use setenv, which would create a 
copy? 
It's 50 bytes, so practically negligible, but still, we even have a comment in 
the source code:

      int retval = putenv (s);
      /*
        unfortunately, we can't portably free S here,
        due to various bugs in glibc prior to 2.1.1
      */


Cheers,
Reinhold

-- 
------------------------------------------------------------------
Reinhold Kainhofer, address@hidden, http://reinhold.kainhofer.com/
 * Financial & Actuarial Math., Vienna Univ. of Technology, Austria
 * http://www.fam.tuwien.ac.at/, DVR: 0005886
 * LilyPond, Music typesetting, http://www.lilypond.org