{"id":19506,"date":"2021-05-05T10:09:04","date_gmt":"2021-05-05T16:09:04","guid":{"rendered":"https:\/\/www.fullcontact.com\/?p=19506"},"modified":"2022-04-19T07:31:29","modified_gmt":"2022-04-19T13:31:29","slug":"fullcontact-engineering-cost-savings-and-saving-our-bacon","status":"publish","type":"post","link":"https:\/\/www.fullcontact.com\/blog\/engineering\/fullcontact-engineering-cost-savings-and-saving-our-bacon\/","title":{"rendered":"FullContact Engineering: Cost Savings and Saving Our Bacon"},"content":{"rendered":"

Why Engineers Need to Consider the Cost of Cloud Computing<\/h2>\n

One of the benefits of cloud computing is the ease with which engineers can spin up infrastructure and achieve business goals rapidly. It\u2019s also one of the shortcomings, especially when cloud providers make pricing confusing to understand as you start to use more and more services. The ability to spin up resource waste in seconds with a single `terraform apply` or button press is a battle every company has, whether they acknowledge it or not. Some companies are even creating <\/span>dedicated roles and teams<\/span><\/a> to control cloud costs.<\/span><\/p>\n

I believe concepts like DevOps and FinOps are primarily cultural challenges. In the case of FinOps, this means that cloud costs are ideally everyone’s concern. While engineers shouldn\u2019t concern themselves with the total monthly costs, we can and should think about the solutions we create from a cost perspective–and it doesn\u2019t hurt to know what that monthly cost is either.\u00a0<\/span><\/p>\n

Anecdotally, it\u2019s been my observation that most engineering departments consider costs only when needed, and in some places, it\u2019s often never truly challenged by the business. While the Finance team will point out irregularities and challenge the need for exceeding our budget in a given month or two real lasting change does not usually occur as a result. Instead of being reactionary, we are now striving to engage in a way where costs are a consideration for solutions to our business challenges. When the cost of a solution becomes just another standard constraint through which we process our solutions, we adapt to hit the targets as best as possible.\u00a0\u00a0<\/span><\/p>\n

Enter Project \ud83e\udd53<\/h4>\n

As the world began to face financial uncertainty brought on by a global pandemic in early 2020, it was apparent that our cloud spend was on the rise and needed to be addressed in a more focused way than the occasional cost optimization here and there. We needed a concerted effort on cost savings that would make a meaningful and lasting impact. \u201cProject Bacon\u201d was born to save the company\u2019s bacon.\u00a0<\/span><\/p>\n

It was decided that Project Bacon would be a one-week project, similar to a company hackathon where the entire engineering team would focus on how to save and reduce costs on their team. Like a hackathon, we brainstormed a list of ideas and projects for the teams to work on and let each engineer choose the team they wanted to be part of.\u00a0<\/span><\/p>\n

The brainstorming resulted in over 30 different savings opportunities that we organized into 8 different categories (or squads for teams to work on):<\/span><\/p>\n

Object Storage and Cleanup<\/h4>\n

Object storage (S3) can easily become one of the largest cost drivers in your AWS account. As part of the project-bacon initiative, we had each team take a detailed look at its S3 usage and consider what data was necessary and what data could be cleaned up. In many of these cases, we found large data sets we could simply delete. In other cases, we applied <\/span>lifecycle policy rules<\/span><\/a> to clean up and delete data after a certain period of time. The combination of manual cleanup and lifecycle rules accounted for the majority of our cost savings during the initial project-bacon.<\/span><\/p>\n

While it is best practice to create lifecycle rules on all of your important S3 prefixes there are times you either have so many prefixes to manage or you are not sure what the rules should be. In cases like this, you can opt to use the S3 Intelligent Access Tiering (IAT). When this tier is applied to your objects AWS will track how often your objects are accessed and attempt to keep them either in Standard or Infrequent Access to give you the best overall price. AWS charges a fee in order to keep track of this data but in general once this is applied you should start to realize savings after the first 30 days. We found that when we applied IAT to a bucket with more than 500 TB of Standard objects we saw a higher cost the first month as everything was being analyzed, but after the first month when the majority of objects were transitioned to infrequent access (IAT-IA) we saw approximately an <\/span>18% cost savings<\/i><\/b> on that bucket.<\/span><\/p>\n

Rightsizing and Modernizing\u00a0<\/span><\/h4>\n

Each time AWS comes out with a new generation of EC2 instances they usually offer an incentive to upgrade to the newer ones through more competitive pricing (for example m4 to m5 instance types). While FullContact tends to stay on the latest family of instances there are times that instances are left running on the old family. Updating to the newer instance family and ensuring you are not using oversized instances (ones with more CPU or memory than your service requires) can result in large cost savings.<\/span><\/p>\n

Reserved Instances<\/h4>\n

Making use of Reserved Instances (RIs) and the newer method of Savings Plans you can save a significant amount on your compute by making an upfront commitment to AWS on how much you will use over the next one or three years. One of our weekly scorecard metrics we track is the percent of non-on-demand compute usage ((RI costs + Savings Plan Costs + Spot Costs) \/ Total Costs). Our goal is for this number<\/span> never to dip below 95%<\/i><\/b>. We keep this metric in check by making sure that we renew Reserved Instances and Savings Plans when necessary and that large dynamic workloads are all running on Spot.\u00a0<\/span><\/p>\n

EMR Reducers<\/h4>\n

We use Elastic Map Reduce (EMR) Spark to run several of our Identity Graph and Batch file export jobs. While it is convenient to be able to spin up large clusters to instantly work on a certain project this flexibility can also lead to a lot of variability in our monthly spend. As part of this initiative we focused on looking for ways for each Spark cluster to become even more efficient including:<\/span><\/p>\n