Is it secure to store EC2 User-Data shell scripts in a private S3 bucket?

Question

I have an EC2 ASG on AWS and I'm interested in storing the shell script that's used to instantiate any given instance in an S3 bucket and have it downloaded and run upon instantiation, but it all feels a little rickety even though I'm using an IAM Instance Role, transferring via HTTPS, and encrypting the script itself while at rest in the S3 bucket using S3 Server Side Encryption. 
 

The Setup

• Created an IAM Instance Role that gets assigned to any instance in my ASG upon instantiation, resulting in my AWS creds being baked into the instance as ENV vars
• Uploaded and encrypted my Instance-Init.sh script to S3 resulting in a private endpoint

In The User-Data Field

I input the following into the User Data field when creating the Launch Configuration I want my ASG to use:

#!/bin/bash

apt-get update

apt-get -y install python-pip

apt-get -y install awscli

cd /home/ubuntu

aws s3 cp s3://super-secret-bucket/Instance-Init.sh . --region us-east-1

chmod +x Instance-Init.sh

. Instance-Init.sh

shred -u -z -n 27 Instance-Init.sh

The above does the following:

• Updates package lists
• Installs Python (required to run aws-cli)
• Installs aws-cli
• Changes to the /home/ubuntu user directory
• Uses the aws-cli to download the Instance-Init.sh file from S3. Due to the IAM Role assigned to my instance, my AWS creds are automagically discovered by aws-cli. The IAM Role also grants my instance the permissions necessary to decrypt the file.
• Makes it executable
• Runs the script
• Deletes the script after it's completed. 

The Instance-Init.sh Script

The script itself will do stuff like setting env vars and docker run the containers that I need to be deployed on my instance. Kinda like so:

#!/bin/bash

export MONGO_USER='MyMongoUserName'

export MONGO_PASS='Top-Secret-Dont-Tell-Anyone'

docker login -u <username> -p <password> -e <email>

docker run - e MONGO_USER=${MONGO_USER} -e MONGO_PASS=${MONGO_PASS} --name MyContainerName quay.io/myQuayNameSpace/MyAppName:latest

 

Very Handy

This creates a very handy way to update User-Data scripts without the need to create a new Launch Config every time you need to make a minor change. And it does a great job of getting env vars out of your codebase and into a narrow, controllable space (the Instance-Init.sh script itself).

But it all feels a little insecure. The idea of putting my master DB creds into a file on S3 is unsettling, to say the least. 

The Questions

1. Is this a common practice or am I dreaming up a bad idea here?
2. Does the fact that the file is downloaded and stored (albeit briefly) on the fresh instance constitute a vulnerability at all?
3. Is there a better method for deleting the file in a more secure way?
4. Does it even matter whether the file is deleted after it's run? Considering the secrets are being transferred to env vars it almost seems redundant to delete the Instance-Init.sh file.
5. Is there something that I'm missing in my nascent days of ops?

Answer 1

What you are describing is almost exactly what we are using to instantiate Docker containers from our registry (we now use v2 self-hosted/private, s3-backed docker-registry instead of Quay) into production. FWIW, I had the same "this feels rickety" feeling that you describe when first treading this path, but after almost a year now of doing it -- and compared to the alternative of storing this sensitive configuration data in a repo or baked into the image -- I'm confident it's one of the better ways of handling this data. Right now we are looking at using Hashicorp's new Vault software for deploying configuration secrets to replace this "shared" encrypted secret shell script container (say that five times fast). We are thinking that Vault is going to be the equivalent of outsourcing crypto to the open source community (where it belongs), except for configuration storage.

In fewer words, we haven't run across many problems with a very similar situation we've been using for about a year, but we are now looking at using a third-party open source project (Hashicorp’s Vault) to replace the normal method.

Hashicorp's vault link - https://www.hashicorp.com/blog/vault-announcement