Out of memory when doing --pipe --block ?

[hubert depesz lubaczewski]

Hi,
I'm trying to work with parallel to split tarball, and then
compress/encrypt/upload-to-s3, but it's failing me.

I'm using parallel 20180122 on 64 bit linux.

My mem looks like:
             total       used       free     shared    buffers     cached
Mem:       4046856    2483552    1563304    1355128       5488    2111912
-/+ buffers/cache:     366152    3680704
Swap:            0          0          0

Not very beefy, as it's just one small test virtual server.

data to be split is ~ 33G.

My command is, more or less:
tar cf - /var/lib/postgresql | parallel -j 5 --pipe --block 360M --recend '' 
handle_single_part /tmp/test.pass us-east-1 "${s3_tarball_part}"

where handle_sigle_part is shell function that does:

handle_single_part() {
    enc_key_file="${1:-}"
    s3_region="${2:-}"
    s3_dest="${3:-}"
    temp_file="$( mktemp --tmpdir "$( basename "$0" ).$$.XXXXXXXXXXXXX" )"
    trap 'rm -f "${temp_file}"' EXIT
    gzip -9c - | \
        openssl enc -pass "file:${enc_key_file}" -aes-256-cbc > "${temp_file}"
    while true
    do
        aws s3 --region "${s3_region}" cp --quiet "${temp_file}" "${s3_dest}" 
&& break
    done
    exit 0
}

The thing is - parallel fails with "Out of memory!" message.

I did strace, and the error comes from "main" parallel process. Couple of 
strace lines above:

14:21:07.654093 rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1], [], 8) = 0
14:21:07.655150 clone(child_stack=0, 
flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, 
child_tidptr=0x7fd9c5bbaa10) = 9984
14:21:07.674230 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
14:21:07.674307 close(29)               = 0
14:21:07.674446 mmap(NULL, 377491456, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
14:21:07.674480 brk(0x19514000)         = 0x2cf4000
14:21:07.674497 mmap(NULL, 377622528, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
14:21:07.674517 mmap(NULL, 377491456, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
14:21:07.674535 brk(0x19514000)         = 0x2cf4000
14:21:07.674550 mmap(NULL, 377622528, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
14:21:07.674568 mmap(NULL, 134217728, PROT_NONE, 
MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7fd9bb8b5000
14:21:07.674586 munmap(0x7fd9bb8b5000, 7647232) = 0
14:21:07.674608 munmap(0x7fd9c0000000, 59461632) = 0
14:21:07.674625 mprotect(0x7fd9bc000000, 135168, PROT_READ|PROT_WRITE) = 0
14:21:07.674649 mmap(NULL, 377491456, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
14:21:07.674682 write(2, "Out of memory!\n", 15) = 15

The memory it's trying to allocate is available. ulimit is set to "unlimited".

What am I missing?

Best regards,

depesz