[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[bug-recutils] Optimizing memory consumption
From: |
Jose E. Marchesi |
Subject: |
[bug-recutils] Optimizing memory consumption |
Date: |
Tue, 14 Aug 2012 22:27:04 +0200 |
User-agent: |
Gnus/5.13 (Gnus v5.13) Emacs/24.0.92 (gnu/linux) |
Hi all.
I recently got a very interesting report: doing a recsel in a 200Mb
recfile takes more than 4Gb of RAM, with the result of the process being
killed by the kernel:
[60738.722562] Out of memory: Kill process 20130 (recsel) score 780 or
sacrifice child
[60738.722570] Killed process 20130 (recsel) total-vm:4841960kB,
anon-rss:3509936kB, file-rss:628kB
I used Valgrind's massif tool in order to profile the usage of the heap
by recsel, and got the following results running recsel on a file
containing 20000 records (3.8 Mb):
--------------------------------------------------------------------------------
Command: recsel 20000-records.in
Massif arguments: (none)
ms_print arguments: massif.out.3560
--------------------------------------------------------------------------------
MB
120.8^ #
| ::::::::::::::::#
| @:: #
| @:: #:
| :@:: #:
| :@:: #:
| ::@:: #:
| ::@:: #:
| :::@:: #:
| ::::@:: #::
| @::::@:: #::
| @@@:@::::@:: #::
| @@@@@@@@ @::::@:: #::
| @@@@@@@@@@@@@ @::::@:: #::
| @:@@@ @@@@@@@@@@@ @::::@:: #::
| @@@@@:@@@ @@@@@@@@@@@ @::::@:: #:::
| @@@@@@@@@@:@@@ @@@@@@@@@@@ @::::@:: #:::
| @@@@@@@@@@@@@@:@@@ @@@@@@@@@@@ @::::@:: #:::
| @@@@@@@@@@@@@@@@@@@:@@@ @@@@@@@@@@@ @::::@:: #:::
| @:@@:@@@ @@@@@@@@@@@@@@@:@@@ @@@@@@@@@@@ @::::@:: #:::
0 +----------------------------------------------------------------------->Gi
0 1.901
0 0 0 0 0 0
1 23,918,655 1,065,632 633,598 432,034 0
[...]
59 1,460,090,910 119,858,072 72,510,893 47,347,179 0
60 1,476,532,025 125,330,256 75,860,862 49,469,394 0
*61 1,942,067,951 126,676,744 76,685,098 49,991,646 0
62 1,958,509,017 108,537,280 65,788,890 42,748,390 0
63 1,974,950,089 90,398,536 54,762,626 35,635,910 0
64 1,991,391,167 72,259,752 43,658,413 28,601,339 0
65 2,007,832,250 53,735,576 32,429,103 21,306,473 0
66 2,024,273,329 35,380,312 21,481,457 13,898,855 0
67 2,040,714,408 17,025,096 10,401,197 6,623,899 0
There are no memory leaks. As you can see, the peak indicates a total
of memory used of around 121Mb in the heap. This is simply too much for
a 4 megabytes recfile, so we must come with some optimizations in terms
of memory consuption.
For example, an obvious optimization is to use a pool of strings to
store the field names. The pool would be shared by all the records
pertaining to some record set.
More ideas?
--
Jose E. Marchesi http://www.jemarch.net
GNU Project http://www.gnu.org
- [bug-recutils] Optimizing memory consumption,
Jose E. Marchesi <=