Known Issues

Overview

S3cmd cannot download large files

Problem Statement: When using s3cmd to manage your data in the firstcolo Stack Object Storage, you may run into an issue trying to download large files (whose size exceeds 100 GiB). You will receive a 503 response asking you to slow down even when reducing download speed to a minimum.

Solutions: We suggest to download the file in multiple chunks using the HTTP Range header. s4cmd supports this out of the box using the --max-singlepart-download-size option:

s4cmd get --max-singlepart-download-size=$((50*1024**2)) --multipart-split-size=$((50*1024**2)) s3://BUCKET_NAME/FILE_NAME

The value of 52428800 Bytes = 50 GiB specified in this example is actually the default value for those parameters, so that using s4cmd without specifying those parameters should already circumvent the mentioned problem by doing multipart transfers.

409 Conflict when accessing public files

This issue only occurs for buckets that get accessed without authentication.

Problem Statement: When accessing a public bucket with no authentication you'll receive 409 Conflict Errors for some of the files.

Solutions: Please check in the horizon dashboard or via. the cli if there is a BUCKET_NAME+segments container created. If yes make sure its also set to allow public access.

The reason for this container to be automatically created is, that large objects (> 2GB) get split automatically by the swift storage during upload. When a large object is accessed, the client fetches an index file from the bucket and retrieves a list of all segments. It then downloads all segments and combines them to the full object.