GNU Parallel on Amazon Linux

[Osborne, Darryl]

Ole,

 

First, thanks for developing and maintaining GNU Parallel. It’s an essential tool and I recommend it in just about every customer call, talk, or presentation I give. I’m an AWS Storage Specialist Solutions Architect for Amazon EFS and I work directly with customers to help them build solutions to leverage the distributed data storage design of EFS and evangelize EFS to the public. It’s mentioned on our AWS Summit and re:invent sessions on EFS and its included in a number of my EFS tutorial CloudFormation templates (creating a file system with dummy data, and the performance and scale-out walk-throughs) - https://github.com/aws-samples/amazon-efs-tutorial/.

 

Second, I don’t know if you’re aware but parallel is now included in the Amazon repository so AWS customers running Amazon Linux can easily install it with a simple “sudo yum install parallel –y” command.

 

One of my sample tutorial results sums it up well. In this example (attached) when I combined GNU parallel and standard Linux copy commands like cp and cpio, I achieved a throughput increase of 690% and transferred files 7.9 times faster when compared to using the copy commands alone.

 

Have you had a chance to play around with Amazon EFS? I’d be curious to hear your feedback.

 

I’d like to stay in touch and keep this dialogue going to see where we can team up and possibly work together on a few engagements.

 

Thanks,

 

Darryl S. Osborne

Storage Specialist | AWS Solutions Architecture

o: 469.283.5878

c: 206.817.2906

efs_performance_tutorial_sample_results.png
Description: efs_performance_tutorial_sample_results.png