Limiting S3 Sync Bandwidth

Out of the box the aws s3 application is suppose to limit bandwidth by setting the “s3.max_bandwidth” option.

Example:

aws configure set default.s3.max_bandwidth 5MB/s

Problem

aws s3 sync is using more bandwidth than the ~/.aws/config file has specified.

Replication Steps

Run the following command to set a max bandwidth limit for the s3 application

aws configure set default.s3.max_bandwidth 5MB/s

Start the sync

aws s3 sync /foo s3://bar

Monitor the bandwidth with nload

nload

Expected Result

nload should show that no more than 5MB/s (40 Mbps) of bandwidth is being used

Actual Result

The s3 application is consuming almost 37.5 MB/s (300 Mbps)

Solution

The reason that this is happening is that the aws config option to limit bandwidth is thread based.

You must change your concurrent threads to 1:

aws configure set default.s3.max_concurrent_requests 1

Leave a Reply

To create code blocks or other preformatted text, indent by four spaces:

    This will be displayed in a monospaced font. The first four 
    spaces will be stripped off, but all other whitespace
    will be preserved.
    
    Markdown is turned off in code blocks:
     [This is not a link](http://example.com)

To create not a block, but an inline code span, use backticks:

Here is some inline `code`.

For more help see http://daringfireball.net/projects/markdown/syntax

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>