GNU Parallel Bug Reports Distribution of args skewed on input run with -X -j 4.

[Henrik Mühe]

This was observed both in

GNU parallel 20130922

as well as

GNU parallel 20170622

Repro is for the latter (2017 version), the only difference is the
count given in tail that triggers the behavior between the two
versions. For brevity, let me give the repro without the data and then
at the end of the email show the full repro with the data. I tried to
find a repro with just seq but failed.

echo "Qlpo...more data see below" | base64 -d | bunzip2 - | head -n
1500 | parallel -X -j 4 "echo {} | wc -w"

output
375
375
375
375

echo "Qlpo...more data see below" | base64 -d | bunzip2 - | head -n
1550 | parallel -X -j 4 "echo {} | wc -w"

output:
7
7
7
1523
6

I would have expected for both to be evenly distributed among 4 tasks.
I glanced at the code and it looks like the arg_list variable is a lot
shorter than expected (the total of the smaller 4 numbers) but we
weirdly spawn and additional process which gets the bulk of the work
and then got lost.

Please help me understand if this is a feature that I fail to
understand or a bug.

Full commands to repro:

Good case:
echo 
"QlpoOTFBWSZTWbfZQ+wAI11/gABf//////////////EAAEBgCByn1rKFs0rShSqV7s3bajtGuIp4hMjI0CaYARR6NJkDKAwDUGBGAAcZME0MhkZGTQ0AaDIwgGg0aZDENACKeKE9U9PTU20kyGSGmjTTTQaAAAAGgAARU/QAJkMgYlNlKe1TeknqemggaBoABoyBo9DjJgmhkMjIyaGgDQZGEA0GjTIYhoANNIIaAACNBqnpok8k2oAAAAAAGTvwUwauqmTpmggdspnda8maZZvmc/VmyxNr40zqg+BzVRX4bXC9UcSYZv3gGPUvPTG40rk7uutvIOD9iJQ4WZFitKZPlco53aSPkpdC+PLA2LJjhr3KdtKzhEHPREC+fUDELcCIUDOl5B1prGKvESVIb4MbTjw6LyY0YUBuJdCIkMVcVtVNruhdjrGRc1jWPATEGJbEvkgAx4MFqKi2A54JzlJiqytpmzdayNqvbweKqcIrJK5HKNzyKs1AnIy9Sx7b5odMhcZvNcEVSuMOu+DCtQq0/HCApgqonb2b9xqBuLyD0kALxA4ybhsSFES5tk1Yqae/FOsuwyNyNZLXk1FLJmtUXOkHWkcCqIoTC1ZDVNhDoutUtGqhNxNhiGzJgvK3G98prDmbdLZ1t0n0ipzjv42FYJe1UgbSAwpiXGG69mZGET9W2TrVVW2rGRuBWK+fCK2zlpTYXazsCtCrWR0OjVQcCBoQIgCEwxNbJ6pzTkaCVYLvZehg4Mg4pNXZcZVQ+DImmsk65OxjfrblM5MyBPconL+xUmMRAiIBxxbhSrZjVFrR9edpM7rBtbRU3XG/TOWw0O3nTkCBIWyqKTaWkmQoMRmqUFJuML2TFyYuMVM72unbJpRFdFIaNKC6oeOLhRWIc6Y+MTsQNxAkKbijCE0tVL6sWC4jzkdXm/ZtK04ivbU3hlrZavuLMtgMJN7D4zvAcDtdEIXUAIZcs8juGQ8FC7UgQIYuLBDDBYsqIMjJzttFMOe0KogIYbe7l+/UrN9F7dGqU0y2QDEoSN5QCuNRoRfG5iBhGZZHXlHq9dFky5PdO7lLYyeHbdd3ayRmOPHJZZ1EvOwC0oOBAvPeeY7bkcIBWLiFm7ptMsbML7WijngyqmPQvTIkY5purz7Yq3oi334eJponk52ZB9CieRQkpEMlAERE5AuqXD2VY593dGLtQzJcYsnvtkplkmmx+VYU8ZV8MAlBTEgOKRmMjtPC0QWogBkIa5oLILmpNyGZTa2l9Fh0UmbWwyvPOZmKJ1bbOZ4NueHLQ3aLgy+14jKWb1AVVdaoZTcwZEHLFYNu0tx7BxA51QVy5XptR0UdYoZyZyGAtZHsKniOYrG/yFt/mrnangdxEsagTtWY7FGESFE6cIMTlKrUdx4Ol6P76IXzuc9uPW/fL8ldhzI5/ParMi9ogQnIjHoFECOBGkFA7OSwXXF4RC2mVYigOZDp+TtMxB/LJqx205sTTN/G7FxT+LpfeysMTnRtaxma+EUjOwgVVLaeBQKQl/NXm8heNtVWHOV09OnGV9+6uLu8tcsXrMhFTz0Ke/zVBcramkhrFAHjI7vbESKloNzamKRjTyvhRpnjo+ExFGipl3iK4OMZA3UKl5CPqUDMjIi71amCvg/k2TLlhYKBZ4P6e3JLTZOLfJ1i4HhUgWPBaz7ccpwWBGUcGhUt0SSuhtK4r69Ql4m3XWaJm8zgDuD5uqoLcim8p1IzppYGFHkRlkV7W9nJcGdWPeskd0eEu3l3O26deQbdFCh1nVmro2c3Llg23NQMiOE6tnXMoSkLFtPofUlQhjZQ66uQvPIeMQOOKLOLu9tIacxegYFtKMYxAn41vtXG9l5blE3d4876GT8mQfPaVbKypvm1oYA0ERl58xSmkHTsS6ic4M+ac88LtJgXWsZWuKDjJ10QoR5sOSfeAbxzHaWSa8ZnAeO5YGs2nfcqTY+ozRtWTIRy0s5eMFxWWA6FxQkNKE5kaZrmW6q/z32sfX24pq5zJL6anyhrtVrqbOYv1buFW/MTDcISkORhGcxvCUoR0HZhDbUXoG1ld2PXhlfVwKKaS6KRllKjeVXT6q3I/3llslz8ndv6S0abMtyHZ0wtoERDLEJBEbAuEIbV43qsWvQvPuqYtD22oL/k3E5CRan0vMhlNOXDzCHB2TuJgSjkrPXZb9WOWdVmjAkqV9pG9Xi8OYACbiTYGDLTUQOpRLc1tYqOpO1cPIxu6bQ0tpm+LQUTPqz7WySkWlKzz7YBiYRM1i36Rtpso4tsu2zaatt5MpQYlIBVcNdx03yiXKkHELN2d0mjinZDhobbIggeEAMQMbooLSjEOCkhahJZ30nkcTyY0XVDB2hyuadUtHIYXJUJz1rdqinBwspnZtZX7hEpTAF6qGFNsFbB1OfBa9LL5/6w/B0JhA6kLzaCYILpcaWicONb6DoV95UEsq7eKOqK1YTcAMtRGVQW4JyVy6ECOO6K/ORS4aK7I349RusRBnWyNEIoERTS5CPWY3LR+FPS2eOSNd8qBrP81k8chTLiJ6oINGa2cz7BVUbZKDASvECeGUvSj2DA0QtGC55NjIEtUpDsqE9nZLsnMe/ebmEBZEQIw5Ue2wkLPhSfzRSBwx8IE8SUOaCUP91aJCYbeknLYDyYIQ0zvCvSO6LNssLUX/FGGeSZUJlk7NT0Jhq7/HluRAltp5FGCL1u66HV1+vH50FrQTmsGDqZ21NPsoKaN0LLRqAqQgdbqwL9fliZMMommC5B710RonI+hhOl+dqphCYV5hKhhsUtCTjdVWTrs2STqkQKsjOSi6v2VdhL0LU/EdFz7dOcdDg3vncXgLE2AUqpYe4dQk+wvMPJBi+7lvkug7YsHS1yHnRFXumbNbBCladLH3ce0TfHb55GkFvYJA22unjaAGEKCpiTcqeS8UZuhhs1RFdZ2AS4J5DkfwhCHvKjPJJT32c1Gis3WUz/dvTWOqX7EQ3xNKraXvyJ7NL+lzN24+wdIH4yCyVgvmWLXAaJD95FP508pPR6cvQiIAiIiAJ6QgRERAE/8XckU4UJC32UPsA="
| base64 -d | bunzip2 - | head -n 1500 | ./parallel -X -j 4 "echo {} |
wc -w"

Not so good case:
echo 
"QlpoOTFBWSZTWbfZQ+wAI11/gABf//////////////EAAEBgCByn1rKFs0rShSqV7s3bajtGuIp4hMjI0CaYARR6NJkDKAwDUGBGAAcZME0MhkZGTQ0AaDIwgGg0aZDENACKeKE9U9PTU20kyGSGmjTTTQaAAAAGgAARU/QAJkMgYlNlKe1TeknqemggaBoABoyBo9DjJgmhkMjIyaGgDQZGEA0GjTIYhoANNIIaAACNBqnpok8k2oAAAAAAGTvwUwauqmTpmggdspnda8maZZvmc/VmyxNr40zqg+BzVRX4bXC9UcSYZv3gGPUvPTG40rk7uutvIOD9iJQ4WZFitKZPlco53aSPkpdC+PLA2LJjhr3KdtKzhEHPREC+fUDELcCIUDOl5B1prGKvESVIb4MbTjw6LyY0YUBuJdCIkMVcVtVNruhdjrGRc1jWPATEGJbEvkgAx4MFqKi2A54JzlJiqytpmzdayNqvbweKqcIrJK5HKNzyKs1AnIy9Sx7b5odMhcZvNcEVSuMOu+DCtQq0/HCApgqonb2b9xqBuLyD0kALxA4ybhsSFES5tk1Yqae/FOsuwyNyNZLXk1FLJmtUXOkHWkcCqIoTC1ZDVNhDoutUtGqhNxNhiGzJgvK3G98prDmbdLZ1t0n0ipzjv42FYJe1UgbSAwpiXGG69mZGET9W2TrVVW2rGRuBWK+fCK2zlpTYXazsCtCrWR0OjVQcCBoQIgCEwxNbJ6pzTkaCVYLvZehg4Mg4pNXZcZVQ+DImmsk65OxjfrblM5MyBPconL+xUmMRAiIBxxbhSrZjVFrR9edpM7rBtbRU3XG/TOWw0O3nTkCBIWyqKTaWkmQoMRmqUFJuML2TFyYuMVM72unbJpRFdFIaNKC6oeOLhRWIc6Y+MTsQNxAkKbijCE0tVL6sWC4jzkdXm/ZtK04ivbU3hlrZavuLMtgMJN7D4zvAcDtdEIXUAIZcs8juGQ8FC7UgQIYuLBDDBYsqIMjJzttFMOe0KogIYbe7l+/UrN9F7dGqU0y2QDEoSN5QCuNRoRfG5iBhGZZHXlHq9dFky5PdO7lLYyeHbdd3ayRmOPHJZZ1EvOwC0oOBAvPeeY7bkcIBWLiFm7ptMsbML7WijngyqmPQvTIkY5purz7Yq3oi334eJponk52ZB9CieRQkpEMlAERE5AuqXD2VY593dGLtQzJcYsnvtkplkmmx+VYU8ZV8MAlBTEgOKRmMjtPC0QWogBkIa5oLILmpNyGZTa2l9Fh0UmbWwyvPOZmKJ1bbOZ4NueHLQ3aLgy+14jKWb1AVVdaoZTcwZEHLFYNu0tx7BxA51QVy5XptR0UdYoZyZyGAtZHsKniOYrG/yFt/mrnangdxEsagTtWY7FGESFE6cIMTlKrUdx4Ol6P76IXzuc9uPW/fL8ldhzI5/ParMi9ogQnIjHoFECOBGkFA7OSwXXF4RC2mVYigOZDp+TtMxB/LJqx205sTTN/G7FxT+LpfeysMTnRtaxma+EUjOwgVVLaeBQKQl/NXm8heNtVWHOV09OnGV9+6uLu8tcsXrMhFTz0Ke/zVBcramkhrFAHjI7vbESKloNzamKRjTyvhRpnjo+ExFGipl3iK4OMZA3UKl5CPqUDMjIi71amCvg/k2TLlhYKBZ4P6e3JLTZOLfJ1i4HhUgWPBaz7ccpwWBGUcGhUt0SSuhtK4r69Ql4m3XWaJm8zgDuD5uqoLcim8p1IzppYGFHkRlkV7W9nJcGdWPeskd0eEu3l3O26deQbdFCh1nVmro2c3Llg23NQMiOE6tnXMoSkLFtPofUlQhjZQ66uQvPIeMQOOKLOLu9tIacxegYFtKMYxAn41vtXG9l5blE3d4876GT8mQfPaVbKypvm1oYA0ERl58xSmkHTsS6ic4M+ac88LtJgXWsZWuKDjJ10QoR5sOSfeAbxzHaWSa8ZnAeO5YGs2nfcqTY+ozRtWTIRy0s5eMFxWWA6FxQkNKE5kaZrmW6q/z32sfX24pq5zJL6anyhrtVrqbOYv1buFW/MTDcISkORhGcxvCUoR0HZhDbUXoG1ld2PXhlfVwKKaS6KRllKjeVXT6q3I/3llslz8ndv6S0abMtyHZ0wtoERDLEJBEbAuEIbV43qsWvQvPuqYtD22oL/k3E5CRan0vMhlNOXDzCHB2TuJgSjkrPXZb9WOWdVmjAkqV9pG9Xi8OYACbiTYGDLTUQOpRLc1tYqOpO1cPIxu6bQ0tpm+LQUTPqz7WySkWlKzz7YBiYRM1i36Rtpso4tsu2zaatt5MpQYlIBVcNdx03yiXKkHELN2d0mjinZDhobbIggeEAMQMbooLSjEOCkhahJZ30nkcTyY0XVDB2hyuadUtHIYXJUJz1rdqinBwspnZtZX7hEpTAF6qGFNsFbB1OfBa9LL5/6w/B0JhA6kLzaCYILpcaWicONb6DoV95UEsq7eKOqK1YTcAMtRGVQW4JyVy6ECOO6K/ORS4aK7I349RusRBnWyNEIoERTS5CPWY3LR+FPS2eOSNd8qBrP81k8chTLiJ6oINGa2cz7BVUbZKDASvECeGUvSj2DA0QtGC55NjIEtUpDsqE9nZLsnMe/ebmEBZEQIw5Ue2wkLPhSfzRSBwx8IE8SUOaCUP91aJCYbeknLYDyYIQ0zvCvSO6LNssLUX/FGGeSZUJlk7NT0Jhq7/HluRAltp5FGCL1u66HV1+vH50FrQTmsGDqZ21NPsoKaN0LLRqAqQgdbqwL9fliZMMommC5B710RonI+hhOl+dqphCYV5hKhhsUtCTjdVWTrs2STqkQKsjOSi6v2VdhL0LU/EdFz7dOcdDg3vncXgLE2AUqpYe4dQk+wvMPJBi+7lvkug7YsHS1yHnRFXumbNbBCladLH3ce0TfHb55GkFvYJA22unjaAGEKCpiTcqeS8UZuhhs1RFdZ2AS4J5DkfwhCHvKjPJJT32c1Gis3WUz/dvTWOqX7EQ3xNKraXvyJ7NL+lzN24+wdIH4yCyVgvmWLXAaJD95FP508pPR6cvQiIAiIiAJ6QgRERAE/8XckU4UJC32UPsA="
| base64 -d | bunzip2 - | head -n 1550 | ./parallel -X -j 4 "echo {} |
wc -w"


Thanks a lot for this fantastic tool and for all helpful input!
Henrik