In classical literature we are told the Titan, Prometheus, held one of the most important secrets in ancient mythology. He knew who would overthrow Zeus and bring an end the reign of the Olympian gods.
In the Greek tragedy, Prometheus Bound, Zeus has Prometheus chained to a rock due to Prometheus’s patronage of humanity. Though having information Prometheus could use to barter for his freedom he refuses to reveal what he knows to the cast of characters who visit to offer him sympathy or scorn. Prometheus’s secret was so important he chose torment rather than reveal it to others and risk that knowledge changing the future.
The story of Prometheus is myth and while none of your secrets might have cosmic implications keeping those secrets secure is important. A secret might not be something you would use to barter for your freedom, but it could be the intellectual property underpinning your business or the personally identifiable information of your customers.
In this blog, I discuss securely sharing files using Amazon S3 pre-signed URLs. I discuss the presign command and will go through the process of generating a pre-signed URL. Having shown how you generate a link for distribution I finish on how you might automate the creation of pre-signed URLs after you upload objects to an S3 bucket.
Overview
Accidental disclosure of intellectual property or customer personally identifiable information could damage the market position of your organisation or be financially ruinous if legal penalties are involved. Nonetheless there may be times where you are required to share information securely. When sharing information stored on Amazon S3 with someone outside of your organisation it is inefficient to generate unique AWS credentials for that person. What you can do, provided you have permissions yourself to access an S3 object, is you can generate a pre-signed URL that provides short term access to that object.
In this case a user with permissions to access the object would use their programmatic access to login to their AWS account and use the S3 presign command to generate a time limited URL. This URL could then be shared with those outside of the organisation who have a need to access that data. At no point is the S3 bucket open to the public and when the expiry time set on the URL lapses the URL is rendered unusable.
The S3 presign command
To examine the pre-signed URL feature of S3, in the following example Prometheus’s secret has been uploaded to an S3 bucket as an inbox punishing 19MB PowerPoint presentation. If I were sharing this with one person, or a hundred people, each could be provided with the same URL to download the presentation.
Here I will use the AWS CLI to login to my AWS account, list what buckets are accessible, list their contents, and then generate a pre-signed URL to enable external parties to download an object of my choosing.
aws configure
AWS Access Key ID [None]: <keyid>
AWS Secret Access Key [None]: <secretaccesskey>
Default region name [None]: us-east-2
Default output format [None]:
Listing the buckets available we see the S3 bucket where the presentation is located and can then list the bucket contents.
aws s3 ls
2020-07-30 14:18:29 bucketofbigsecrets
aws s3 ls s3://bucketofbigsecrets
2020-07-30 14:18:58 20622958 Prometheus.pptx
Having identified the bucket and the name of the presentation to be shared I can then generate a pre-signed URL. By default, all pre-signed URLs expire in one hour (3600 seconds) unless specified otherwise.
aws s3 presign s3://bucketofbigsecrets/Prometheus.pptx
https://bucketofbigsecrets.s3.us-east-2.amazonaws.com/Prometheus.pptx?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIA44VY4PBGN427ZUG7%2F20200730%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20200730T132253Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEGYaCXVzLWVhc3QtMSJGMEQCICcRKN0s%2BJwdEBRCgKfdWvEtECVFavMosaqDa30zNa4%2FAiAHPRB5o26vJkQPdAFBE2eQtiUaLxLK%2F2u%2Bu9u4mN%2BHASqeAgguEAEaDDg4NjIyNDc0NjU3MiIMo0jv1qSaG0%2BJHWiEKvsBiIs468kc56%2F8Jop71abvDQku34%2BVkAGQnHoYokowXEjVhfWlPEP80HdK9y78eUfYJBoX8CGATvajBbhmvrquNgxeUDImWDKsBYUu7VnhibK03vr9npDo44AOfqOUbL1Ulf%2BThnzMfpQzWIiL29XknuvInD7Qvi8C2fjyZ2sLSY3pSI8hyAJXP5RCDQlXYWrG5SD6vsNxpyJd2ErkK8575ONBWZXqCoNlSOsYEMvdWr%2FtURC2CiG6Bh0FLnx98vrI1aSBm2bXCjyFOHenbEjxgY2uAjuFPKPHVGnbNf9edoNiCwBnp4h3ZGgYa9w044%2BSZXTklTxap3cxGDwwz4%2BL%2BQU6ngGY6jOxtjwJSwQS5JmQ8%2BY%2BLKCixFXv3P37KbyswnAjRMcRbFEGllz4HXyEomLBWZKjJuAlO1r44YL5HV4ItcPfvYEU778s6BVuoHNSwGFJyQXNfrDnhdf1bQqKt2KGQhpExdBhFr4RHJSsPAxmkQvNA3wKZ5%2F%2F2dHxnW76o5vTQb9XY52sUeIZyX9iZwMstREkV9xxNfwQFDo7Jhrsrw%3D%3D&X-Amz-Signature=5f11113908741d8a2a81a639e593f4ea28b9d7a9354faf1790f2ab93d6b3a5ab
Using a web browser to access this link I can now download the file.
The default expiry time can be modified by appending the option --expires-in to the command. For example, if I wanted the pre-signed URL to expire after two hours, I would state that as 7200 seconds.
aws s3 presign s3://bucketofbigsecrets/Prometheus.pptx --expires-in 7200
When the pre-signed URL expires any attempt to access the object will be denied with a “Request has expired” string returned as part of the error message. I can check that from a web browser.
In this way using pre-signed URLs you can distribute information to other people without providing public access to the S3 bucket or having to grant end-user credentials to people who should not have them. As the access token tied to the pre-signed URL has an expiry time you do not have to worry about revoking access later.
Pre-signed URL uploads with Lambda automation
Were someone to share data with us and would like to place an object in our S3 bucket the process is more involved and requires writing code using a supported AWS SDK. As the receiver it would involve us creating a user with programmatic access in IAM. Attaching the correct IAM policies to that user, ensuring that GET object and PUT object permissions have been selected.
Having provided the credentials to the party uploading the data they would then use one of the AWS SDKs to generate a URL for the item to be uploaded and put that object into our S3 bucket. While beyond the scope of this blog post Lambda could be used to generate pre-signed upload URLs for distribution to those looking to upload data to a specific S3 bucket. You can find further information about uploading objects using pre-signed URLs here.
Conclusion
By default when you create a bucket all public access is blocked unless you choose to disable that. AWS identity and access management (IAM) policies can be used to enforce who has read/write access to an S3 bucket, allowing you to control who can access your data. S3 access control lists (ACLs) are attached to every bucket and object, they specify which AWS accounts groups are granted access and what type of access they have.
Combined these constitute the multi-layered security model for S3, and pre-signed URLs allow that model to be kept intact while facilitating data sharing with people outside of your organisation.
As I discussed earlier, in antiquity a secret was a privilege of power and a sign of access to that power. Today, our secrets can be how we run our businesses and serve our customers. In this post, I looked at using S3 pre-signed URLs as a facilitator for information sharing in a secure manner between people. I then used the Amazon S3 pre-signed to show how you can generate a temporary credential which you can distribute as an expiring URL.
With pre-signed URLs there is no need to open buckets to the public. You can share data with whomever you choose without compromising the multi-layered S3 security model.
To learn more about sharing S3 objects, visit the Amazon S3 documentation here.
Photo by Christian Paul Stobbe on Unsplash