I previously published a Ruby script that essentially duplicates an Amazon S3 bucket. I use it to copy my production bucket into a testing bucket that I can then use in my staging environment, without fear of killing off something important.
Anyway, this script used to take around 18 minutes to complete a full copy of a bucket with about 2,000 keys. What a pain when I had to run this script multiple times for some testing!
So I finally got around to rewriting it today, and wow, what an improvement!
I upgraded the delete and copy operations to run multithreaded, and boy does it fly.
On my main bucket (with about 2,000 keys) it now only takes about 1.5 minutes to complete! That’s 12x faster than it used to be!
So if you are looking for a multithreaded Amazon S3 bucket copier or duplicator, check it out!