Imagine that you have a small project in the cloud. For six months now, you have been paying 20 cents a month for it. Basically, nothing special. But one morning, suddenly a large bill of $ 2,700 comes in.





In the cloud, we pay for storage, computing and other services as we use them. No need to set up your own server. However, the disadvantage is that you can accidentally spend more money than is in your wallet. This is especially difficult with serverless solutions that automatically scale with incoming traffic.



Accidentally leave an expensive VM unattended or your lambda functions will spiral out of control - and this can lead to an unexpected score.





In this article, we'll take a look at how billing works and how to prevent unexpected invoices from appearing.

Small business, small bills

This article is about personal projects and small companies with relatively small accounts. Some large corporation may not notice the $ 3,000 difference.

There is no perfect solution

Unfortunately, there is no perfect solution here. As Corey Quinn explains in his podcast , AWS billing system is several hours behind, and in some cases up to 24 or 48 hours behind. As a result, over-quota alerts can be triggered several hours or days after a significant overrun occurs. Alerts are still a great tool to guard against unnecessary costs, but only in cases where you forgot something for a few days, for example, did not stop an expensive EC2 instance after a machine learning workshop.



It's up to you how much time to spend protecting against unexpected expenses, but I highly recommend spending two minutes setting up budget alerts !

Defense mechanisms

There are many mechanisms to protect against unexpected bills. Consider security, alerts, corrective actions, and improving visibility.

1. Protect your account with multi-factor authentication

This is the first thing you should set up when creating a new AWS account.





Here's the official guide from AWS on setting up Multi-Factor Authentication (MFA), which provides an additional barrier against attackers.

2. Budget alerts

This is the second thing to configure when creating a new AWS account.



Budget Alerts are the most popular way to keep track of your spending. For example, you receive an email notification that the limit has been exceeded. You can further customize your notifications using Amazon SNS or AWS Chatbot .



Here's a short video (52 seconds) on creating your first budget alert. Ryan Lewis made a longer video with more detail, budget alert details, and many ways to customize them.



If you are already using the CDK, the aws-budget-notifier package is best...

How much to start with?

Start with an amount that is slightly higher than your current expenses and that you are comfortable with. For example, $ 10. If you already have workloads that have been running for months, then take the average and add 50%.



I also recommend setting up multiple billing alerts with different thresholds :

1. Convenient warning: the amount you are willing to put up with, but then study the bill.
   
   
2. Dangerous warning: you no longer feel comfortable and want to close the service as soon as possible. If the comfortable amount is $ 10, then this could be $ 100.
   
   
3. Critical warning: with this amount, there is already a desire to drop a nuclear bomb on your account. If the comfortable amount is $ 10, then this could be $ 500. You can attach special budget actions or pager alerts to this alert to automatically stop your EC2 instances or wake you up.

In addition to the predefined thresholds, you can also try AWS Service Cost Anomaly Detection .

Emergency Nuclear Bomb

Since you can send notifications to SNS, you can run a lambda function that starts aws-nuke , destroying all the infrastructure in your account. Never use this on a production account. Check out the GitHub repository for more information .

3. Budgetary actions

AWS recently introduced Budget Actions. This is an extension for Budget Alerts, where certain actions are triggered when the budget limit is exceeded. In addition to sending email notifications, you can now enforce custom IAM policies such as “Prevent launching EC2 instances” or let AWS close EC2 and RDS instances, as shown below.

4. Mobile application

The AWS Mobile Console gives you the ability to check all your expenses in 3-5 clicks.



Here are two screens from the mobile app:







To use the app, you must set up a dedicated user who only gets the permissions the app needs to display your expenses.



Here is the IAM policy that gives the app read access as well as cloudwatch notifications.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ce:DescribeCostCategoryDefinition",
                "ce:GetRightsizingRecommendation",
                "ce:GetCostAndUsage",
                "ce:GetSavingsPlansUtilization",
                "ce:GetReservationPurchaseRecommendation",
                "ce:ListCostCategoryDefinitions",
                "ce:GetCostForecast",
                "ce:GetReservationUtilization",
                "ce:GetSavingsPlansPurchaseRecommendation",
                "ce:GetDimensionValues",
                "ce:GetSavingsPlansUtilizationDetails",
                "ce:GetCostAndUsageWithResources",
                "ce:GetReservationCoverage",
                "ce:GetSavingsPlansCoverage",
                "ce:GetTags",
                "ce:GetUsageForecast",
                "health:DescribeEventAggregates",
                "cloudwatch:DescribeAlarms",
                "aws-portal:ViewAccount",
                "aws-portal:ViewUsage",
                "aws-portal:ViewBilling"
            ],
            "Resource": "*"
        }
    ]
}
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

Permissions are divided into three groups:

1. CloudWatch Cost Explorer (everything that starts with ce:
   
   
   
   ). Detailed information on current and projected costs.
   
   
2. CloudWatch (cloudwatch:DescribeAlarms
   
   
   
   ). , .
   
   
3. (, aws-portal:



health:
   
   
   
   ). .

5.

If you accidentally upload access keys to a public repository , attackers can launch expensive EC2 instances in your account and use them, for example, to mine bitcoins. There have been reports of cases where small instances are hiding in underutilized regions so that the owner does not notice them in the billing detail.



You can use the AWS Secrets Manager to clear passwords or other secrets from your code .



Follow this AWS guide to create your first secret . Then replace the secret from the codebase by requesting it using one of the official AWS clients ( boto3 for Python ).

import boto3

client = boto3.client('secretsmanager')

response = client.get_secret_value(SecretId='replace-me')

secret = response['SecretString']
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

Note that each secret costs $ 0.40 per month, plus $ 0.05 for 10,000 API calls.

Support service

If you receive an unexpected invoice, stop any applications that caused the situation, change access keys if necessary, and contact AWS Support. Here's a 20 second video of where to click.



Step-by-step instructions for submitting a support ticket:

• In the upper right corner, click the Support button and select Support Center
  
  
• Press the orange Create case button
  
  
• Select account and billing
  
  
• Select "Billing" as the type and "Problem with payments" as the category
  
  
• Fill in all the details and send

While some have been reimbursed for their unexpected bill, don't count on it too much.

Conclusion

The first thing to do is set up MFA and budget alerts . You can then explore more complex operations, such as budgeting actions, to freeze your account if expenses spike.



To prevent your secrets and access keys from ending up in public repositories, save them to AWS Secrets Manager .