28 Mar, 2023
However, I have seen cases where organisations tend to ignore recommended security practices for storing these credentials. Data and IP thefts are all side effects of such ignorance. I have worked with clients where they had SSH keys kept in a shared network drive so all devs could freely gain access to the servers. Some people had also allowed `0.0.0.0/0` in their production database’s security group just so the devs could connect the database from their home. Obviously, this also made their database publicly reachable while exposing risks to be hacked by potential attackers.
It's crucial to take security seriously in this regard to avoiding businesses from going through reputation damage or even financial loss in some cases. Let's take a closer look at this and prepare our tech stacks to adopt some of these security patterns mentioned below. A small disclaimer, I'm considering AWS as a preferred cloud provider for this article. I will just provide a blueprint & obviously, you can find similar offerings on other cloud platforms.
Gone are the days when IT Admins used to create a file on their desktop called password.txt which contained access to various systems the business used to rely on. We are living in an era where humans prefer to use Password Managers (like LastPass, 1Password ...) & OAuth (eg. OneLogin, Okta ...) instead. An Important thing to note, these systems are designed for 'humans', not computers. Below are a few ways to help you set up credentials access for your app:
AWS SSM parameter store, for storing encrypted strings & provide a parameter path to your app so it can download it on demand
AWS Secrets Manager is also a service you can consider to fully automate the workflow.
Passing secrets as environment variables or storing them in some vault or S3 bucket which is then set as environment variables to your app.
In some cases, though it’s not ideal, databases are shared across multiple systems may be by design or due to legacy architecture. Option (3) may not be suitable in such cases as it would require you to redeploy all upstream applications if those credentials get rotated. The other two options, (1) & (2), are very much suitable for this use case because your secrets are stored in a centralised system whose access can now be managed by IAM roles & policies. The moment these credentials are rotated, the upstream systems can automatically receive the change.
AWS has taken the secrets manager to a next level by providing direct integration with your RDS databases. It supports secrets rotation out of the box and can automagically update credentials in an integrated database server.
Deploying secrets
It's very important to build a strategy on "how" your credentials will be deployed so your app can easily access them. In terms of AWS Secrets Manager, you always have the option to auto-generate, auto-rotate & auto-update your secrets. You may choose to trigger a lambda each time the secrets manager auto-rotates the secret & use lambda to perform certain tasks as per your app's needs. So, deployment of secrets using Secrets Manager takes a lot of overhead away.
Here is a CloudFormation snippet (example) that creates an RDS instance with an auto-generated password.
# Auto generate secure password & store it in SecretsManager
Secret:
Type: 'AWS::SecretsManager::Secret'
Properties:
Name: !Ref SecretName
GenerateSecretString:
SecretStringTemplate: '{"password": "%s"}'
GenerateStringKey: 'password'
PasswordLength: 16
Tags:
- Key: 'Name'
Value: !Ref SecretName
# Creates RDS instance using the above password
RDSInstance:
Type: 'AWS::RDS::DBInstance'
Properties:
DBInstanceIdentifier: !Ref DBInstanceName
Engine: 'postgres'
EngineVersion: '12.4'
DBInstanceClass: !Ref DBInstanceClass
MasterUsername: !Ref MasterUsername
MasterUserPassword: !Join ['', ['{{resolve:secretsmanager:', !Ref SecretName, ':SecretString:', !Ref SecretUsernameKey, '}}']]
DBSecurityGroups:
- !Ref RDSSecurityGroup
And here is an example of auto-rotating RDS passwords:
RDSInstanceSecretAttachment:
Type: 'AWS::SecretsManager::SecretTargetAttachment'
Properties:
SecretId: !Ref SecretName
TargetId: !Ref RDSInstance
TargetType: 'AWS::RDS::DBInstance'
RotationSchedule:
Type: 'AWS::SecretsManager::RotationSchedule'
Properties:
SecretId: !Ref SecretName
RotationLambdaARN: !GetAtt [LambdaFunction, Arn]
RotationRules:
AutomaticallyAfterDays: 90
Duration: 1h
With regards to the other two options (1) & (3), it will be your responsibility to create, store, and rotate the secrets; not to forget, potentially redeploy an app in case of option (3). So, it's important to have a process in place to perform these tasks in a repeatable, reproducible manner with minimising human errors. I am hoping that you are not manually storing secrets in the SSM Parameter store. It's always a good practice to pre-encrypt your secrets using AWS KMS & store them in code if you intend to use either option (1) or (3). Storing them in an encrypted form with hardened IAM policies can provide an additional layer of protection ensuring only authorised apps (and people) can decrypt during runtime.
Ideally, you must avoid putting secrets in the code and rely on managed services like Secrets Manager instead. However, in case this is not an option for you; then here are few concepts you can follow to keep your secrets in the code without sacrificing security. Regardless of whether your goal is to use an SSM parameter store or an S3 bucket, you must build an automated system to put those credentials in there to avoid any human errors. But storing secrets in code could be challenging because it might impact your roll-back/forward strategies.
Storing creds in the same repo alongside your code can save you from managing a separate repo; but depending on the nature of your roll-back/forward deployment strategy, it might make sense to manage these creds into its separate repo.
I worked with a client who used a third-party SaaS product called Hubspot for many of their business processes. The developers had rotated Hubspot’s API key by following their yearly password rotation policies. They did all the right things by KMS-encrypting a new key & updating it into their codebase. Few days later, they discovered a major bug in the app and they had to rollback the previous release. However, this is no longer an option because older version had an old API key that is no longer valid. If they would have used SSM parameter store instead of storing the secrets in codebase, they would have been able to roll-back without having to loose connectivity with Hubspot. Even a standalone repo (separate from your app's codebase) for storing & deploying these credentials/keys could work in this case.
No one likes their passwords being published on the internet. It’s important to harden access to your secrets by using IAM policies with limited privileges. If you are intending to use KMS for encrypting your secrets, you must avoid using AWS provided KMS keys and create your own key (called CMK) instead. CMK allows you to have full control over it’s access policies. This also comes really handy when you have to migrate your infra (e.g. RDS) between AWS accounts.
Having a CMK with granting kms:* access to everyone is not ideal either. As a good security practice, you should nominate a management IAM role that has administrative kms:* access to your key while granting specific permissions like kms:Encrypt to other or specific IAM roles to allow usage. To further tighten the access, you can always make use of "IAM Conditions" in the IAM Policy to ensure the requests are made from approved AWS Accounts.
Here is a CloudFormation snippet for example:
# IAM Policy statement
Sid: Allow AWS Services permission to describe a customer managed key for encryption purposes
Effect: Allow
Principal:
AWS: arn:aws:iam::$ACCOUNT_ID:root
Action:
- kms:DescribeKey
Resource: "*"
Condition:
StringEquals:
kms:CallerAccount:
- $ACCOUNT_ID
Bool:
aws:ViaAWSService: "true"
I hope these insights were useful to help you create a secure & more reliable environment.
.png){kind=logo}
