[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: sort: memory exhausted with 50GB file
From: |
Paul Eggert |
Subject: |
Re: sort: memory exhausted with 50GB file |
Date: |
Fri, 25 Jan 2008 15:13:27 -0800 |
User-agent: |
Gnus/5.11 (Gnus v5.11) Emacs/22.1 (gnu/linux) |
Leo Butler <address@hidden> writes:
> sort rapidly chews up about 40-50% of total physical memory
That's weird. It shouldn't do that. It doesn't do that on my machine
(Debian stable x86, coreutils 6.10, compiled with GCC 4.2.2). Memory
usage goes up to 250 MB (as requested) and stays there. 'sort'
creates temp files of size 201,061,873 bytes, each containing
2,545,087 lines. Here's the command I used to try to reproduce the
problem:
awk 'BEGIN {for (i=0;i<467289720;i++) print "-16 -2 -14 -5 1 1 0
0.3080808080808057 0 0.1540404040404028 0.3904338415207971"}' |
sort -S 250M -k 6,6n -k 7,7n -k 8,8n -k 9,9n -k 10,10n -k 11,11n -T /tmp -T
$HOME/junk -o /tmp/foo
I suppose it could be a locale problem. What's the output of the
"locale" command?
Can you run 'sort' under GDB, and see what the stack backtrace looks
like when 'sort' fails?
Are you compiling for x86 or x86-64?
- sort: memory exhausted with 50GB file, Leo Butler, 2008/01/25
- Re: sort: memory exhausted with 50GB file, Bob Proulx, 2008/01/25
- Re: sort: memory exhausted with 50GB file, Leo Butler, 2008/01/25
- Re: sort: memory exhausted with 50GB file, Paul Eggert, 2008/01/25
- Re: sort: memory exhausted with 50GB file, Jim Meyering, 2008/01/26
- Re: sort: memory exhausted with 50GB file, Leo Butler, 2008/01/26
- Re: sort: memory exhausted with 50GB file, Jim Meyering, 2008/01/26
- Re: sort: memory exhausted with 50GB file, Jim Meyering, 2008/01/26
- Re: sort: memory exhausted with 50GB file, Leo Butler, 2008/01/26
- Re: sort: memory exhausted with 50GB file, Paul Eggert, 2008/01/27
Re: sort: memory exhausted with 50GB file,
Paul Eggert <=