[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
sort with a large buffer size (-S)
From: |
Ryan Morin |
Subject: |
sort with a large buffer size (-S) |
Date: |
Fri, 29 May 2009 16:41:13 -0700 |
I routinely use sort on very large files and generally use a buffer size of
1500M to accomplish this. I have some large memory 64-bit machines but am not
able to utilize this memory to improve sorting efficiency. In other words, I
would like to be able to set -S to allow sort to consume much more memory to
save the necessity of writing large temp files to disk (unless necessary). I
presume there is a way to compile sort to allow it to utilize more than 4G of
memory on an x86_64 machine but have not been able to determine how. Here is
what I am currently encountering:
ls -l temp
-rw-rw-r-- 1 rmorin users 807714374 May 29 16:15 temp
sort -S 4000M temp > temp.sort
sort(45777) malloc: *** mmap(size=4194308096) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
I work on x86_64 Macs and also x86_64 machines running RHEL.
I would appreciate any advice on the matter.
Kindest regards,
Ryan Morin
- sort with a large buffer size (-S),
Ryan Morin <=