How newcomers can avoid and prevent unwanted charges on their AWS accounts
Wed, 09 Jun 2021

Lately I saw many posts and tweets about people racking up moderate to huge bills on AWS without realizing it, a lot of heated discussions, but nothing on how to mitigate those problems. So I have decided to share my experience and my tools to make sure you won't end up owning 80.000$ to AWS.

Take in consideration that this is a "starter kit", the more your usage grows the more sophisticated methods you will have to adopt, but as a starting point is enough.

I would say there are two categories when it comes to this matter:

  • Costs control

  • Security

Costs control

Costs control is just the act of monitoring your current and forecasted bill and make sure it won't exceed your expectation and budget. We can monitor costs through:

  • Budget and alerts

  • General view of all the resources deployed

  • Daily cost notification

Budget

The first thing you should do after you sign up for an AWS account is setting up the budget alters. If you are lazy but you still want to avoid huge bills, this is the only thing you should do. Navigate to AWS budget with your root account, click on create budget, select Cost budget, here you can setup your costs, after that click on configure threshold. Here I usually suggest to have two thresholds, one with Actual and the other with Forecasted costs, in this way you can have a visibility on your current and future bill. Remember to insert your email under Set up notifications.

You can check this video tutorial for more detailed instructions.

General view

So, did you spin up a super computer cluster in some obscure region and forgot it about it? There is a little "trick" to search for all the deployed resources in one place, it is called AWS Resource Groups and tag editor, probably it is not meant for searching resources, but you can use it for this purpose.

Navigate to the AWS Console Resource Groups and Tag editor and from here you can access the tag editor, select All regions and if needed Resources types (For example: AWS::EC2::Instace), then press Search resources.

Remember this works only at account level, if you have multi-account setup you will have to switch role and repeat the same thing.

Daily costs notifications

This is more of an optional step, but sometimes budgets might be not enough, at least in my opinion. Budgets, in fact, only alert you when the threshold is breached or it is forecasted to be breached.

What I usually do, is to have a lambda that gets triggered every day and sends me an SNS notification directly to my inbox with both the forecasted and total current bill; in this way I can check if there have been any abnormality and immediately take action.

Just recently I have decide to open source it and make it available in the Serverless Application Repository. To start using it you just need to deploy it into your account, feel free to propose new features or open a new issue if you find any in the github repository.

Security

Security can also have a big impact on costs. What can happen If a malicious actor can access your account? Well you know, credit card, bitcoin mining, DDOS attacks, etc...

  • Avoid accidentally pushing credentials to public repository

  • Activate Multi-factor authentication

  • Periodically rotate passwords and IAM credentials

  • Restrict the AdministratorAccess policy

Avoid pushing credentials to public repository or share them

This is the scariest thing that can happen to us, if you push credentials to a public repository chances are that in few minutes some bot will discover and grab them. Let's face it, it might happens, and the remediation is fairly simple.

One of the tool I am using is git-secrets by awslabs. This tool will install a pre-commit hook that will make your commit fail if it recognize a particular patter, like aws credentials secret keys, in your committed files.

Multi-factor authentication

This should be another must and done immediately after sign up. You should activate Multi-factor authentication for both root account and any IAM user account you create. You can read more here. You can use the Google Authenticator app, however these days I prefer using my YubiKey.

Passwords rotation

I highly suggest periodic password rotation. You can simply enable password expiration in IAM password policy, in this way you will be forced to periodically change it. Navigate to your IAM account settings and check Password Policy, check the Enable password expiration and set the expiration you desire. Here you can also enforce some password requirements.

IAM credentials rotation

You should periodically rotate your local AWS keys. This is often very overlooked, I saw personal credentials active for years and never deleted or inactivated when a member leaves. Changing IAM credentials regularly should be a must in any organization, it reduces the impact if the key become compromised.

The process to rotate the credentials requires some steps:

  1. Create a second access keys pair

  2. Download those keys

  3. Substitute them from your ~/.aws/credentials file

  4. Deactivate the previous access keys

  5. Test that everything is working

  6. Delete the old keys

But we can automate this process using the AWS SDK or API. This is why I have built a small go utility that helps you rotate local credentials called LocalKeyRotation, not much fantasy in naming, but it does the job. You can simply run the command or create an anacron job to run it periodically.

Feel free to suggest any feature or open a new issue if you find any. Alternatively, if you are using aws-vault, you can use their rotate feature

Limit IAM permission

Note: do not create access keys for your root account, instead create a IAM user with limited permission and then create the access keys for that user.

This is a more of an advanced topic, but nonetheless useful. After you have created your first IAM user, instead of giving him AdministratorAccess, you can make a custom policy using IAM conditions; you can read more about IAM condition here.

One thing I would do is to limit the regions in which you can provision resources:

"Condition": {
                "StringEquals": {
                    "aws:RequestedRegion": "us-east-1"
                }
            }

Since a lot of services have regional quotas, for example the base limit of EC2 instances per region is 20, this could somehow tamper the effect of compromised credentials. You can also limit the type of instances size you can deploy:

"Condition": {
                "ForAnyValue:StringNotLike": {
                    "ec2:InstanceType": [
                        "*.nano",
                        "*.small",
                        "*.micro"
                    ]
                }
            }

You can also limit the resources you can work with:

        {
            "Effect": "Allow",
            "Action": [
                "sns:*",
                "s3:*",
                "cloudwatch:*",
                "apigateway:*",
                "lambda:*",
                "dynamodb:*"
            ],
            "Resource": "*"
            }
        }

In this way you will make sure nothing else can be deployed other than those services listed in the actions.

So your custom policy could be something like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "*",
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "aws:RequestedRegion": "us-east-1"
        },
        "ForAnyValue:StringNotLike": {
          "ec2:InstanceType": [
            "*.nano",
            "*.small",
            "*.micro"
          ]
        }
      }
    }
  ]
}

Conclusion

I really hope this post can help someone getting started on AWS without getting unwanted nasty surprises, always be vigilant, AWS is not a toy and money is money. Cheers!

Thanks for reading!