How to use AWS SSM Parameter Store easily in Python
AWS just announced a new service AWS Secret Manager in SF Dev Summit (I was there at the announcement 😇), which is a cool service to help you to manage and rotate your secrets securely.
But actually, this is not something new. There is also a less-well-known service AWS Simple System Manager (SSM) that provides a similar feature to Secret Manager. Today I would like to write a post to show you about this service and how you can use it in Python easily.
AWS SecretManager and AWS SSM Parameter Store
AWS Secret Manager helps you to store, distribute, and rotate credentials securely. You can use it to store credentials for RDS, databases or any type of secrets (token, api secret, etc). It also provides secret rotation to allow you to change the secret and also an audit trail when then secret is rotated. You can use IAM to control access to secrets.
Secret Manager has a very nice integration with RDS. It will help you to populate relevant information about RDS such as hostname, port, … I can’t find the integration for Redshift though. And when you delete a secret, it enforces you to wait at least 7 days before the secret will go away. It will cost you $0.4/month for each of secret you store using the service.
Secret Manager is not the only way you can store secrets on AWS. There is also an another service names AWS Simple System Manager (SSM) that comes with a similar feature, namely Parameter Store. SSM Parameter Store is similar to Secret Manager, but it also allows you to store non-secret things. Below is the description of the service from AWS.
AWS Systems Manager Parameter Store provides secure, hierarchical storage for configuration data management and secrets management. You can store data such as passwords, database strings, and license codes as parameter values. You can store values as plain text or encrypted data. You can then reference values by using the unique name that you specified when you created the parameter. Highly scalable, available, and durable, Parameter Store is backed by the AWS Cloud. Parameter Store is offered at no additional charge.
So why use Parameter Store? Because it’s flexible, you can organize your parameters in a hierarchy. It has revision control, so you can keep track of your configuration and secrets history. It supports encryption via KMS, and also permissions control via IAM. And it’s completely free! The only missing feature is secret rotation. But you can always use Lambda + Cloudwatch timer to rotate the secret in the parameter store.
Personally, I prefer Parameter Store. Not only that i don’t have to pay $0.4/month for each of my secret, but also I can use it as a generic Configuration Management tool within AWS. And in fact, if you are in AWS, you should have used this already.
At the moment, there are limit of 10,000 parameters per account, but you can lift the limit by contacting the support team.
You can read this article by Matt Adorjan. It is intuitive and very easy to understand if you are new to SSM Parameter Store.
How to use this easily in Python?
Once you have the parameters in Parameter Store. You can use AWS Python SDK to query
import boto3ssm = boto3.client('ssm')
parameter = ssm.get_parameter(Name='/Prod/Db/Password', WithDecryption=True)print(parameter['Parameter']['Value'])
But AWS SDK is really verbose and very inconvenient to use, i never feel like to use it 😕. Can we do better?
What If we could just access the parameters as it is a Python dictionary? So if we have a key /Prod/Database/Password then we should be able to query it store['Prod']['Database']['Password']. Because the parameters are organized as in a hierarchy manner, accessing it like a dictionary inside a dictionary feels more reasonable to me.
With that motivation, I go ahead write a simple class. You can find it on this gist. Then you can use it as following:
from ssm_parameter_store import SSMParameterStorestore = SSMParameterStore(prefix='/Prod')# access a parameter under /Prod/ApiSecret
my_secret = store['ApiSecret']# check if a parameter available in the store
'MyKey' in store# list available parameters
print store.keys()# access parameter under Db prefix
db_store = store['Db']# this will query /Prod/Db/Password
my_db_password = db_store['Password']# or you can also do
my_db_password = store['Db']['Password']
For performance, there is also a caching feature to help you cache the SSM queries. Just create a store with ttl parameter to enable caching
# create a store that cache the result up to 10s
store = SSMParameterStore(ttl=10)This store is readonly. If you allow arbitrary change to configuration, you will have many surprises. But if you really want, you can always extend the class to allow write back to the store.
One last thing to note that the store will use AWS credentials available in system such as EC2 Instance Role or environment variables. In case you want to bring your own keys, usually during development on your machine, you can do as following. Using Instance Role is recommended for production though.
import boto3# configure your ssm client here, such as AWS key or region
ssm_client = boto3.client('ssm',
region_name="us-east-1",
aws_access_key_id="XXX",
aws_secret_access_key="YYY"
)# then pass to the store
store = SSMParameterStore(ssm_client=ssm_client)
That’s it. Now you have an easy way to integrate your Python application with SSM Parameter Store. There should be no excuse to hard code the secrets or configuration in your application anymore.
