<img src="https://ws.zoominfo.com/pixel/6169bf9791429100154fc0a2" width="1" height="1" style="display: none;">

We're blowing the whistle on Legacy PAM 🏀 Join us for an Access Madness Webinar on March 28

Search
Close icon
Search bar icon

How Betterment Secures Server Access - Automate the Boring Stuff

StrongDM manages and audits access to infrastructure.
  • Role-based, attribute-based, & just-in-time access to infrastructure
  • Connect any person or service to any infrastructure, anywhere
  • Logging like you've never seen

Chris Becker, SRE, Betterment

Chris Becker is an SRE at Betterment. Previously, he did similar work on Warby Parker's Infrastructure team. At Betterment, he earned the label APT (advanced persistent threat) thanks to consistently tripping alarms with his peculiar scripts and commands.

In this talk, he discusses how Betterment's approach to server access controls evolved as the team grew exponentially. With more people and keys to manage, the SRE team needed to find ways to automate more and reduce the maintenance overhead.‍

Advantages of StrongDM for Betterment

As Betterment searched for a more scalable solution, they evaluated tools based on 3 criteria:

  • support a Beyond Corp model for remote access
  • includes an audit trail for compliance reporting
  • easy to deploy and maintain

The team initially deployed Hashicorp's Vault to manage SSH keys. While it was a very powerful product, the maintenance proved to be too much burden.

The team then deployed StrongDM.

Ease of Deployment

Because StrongDM does not require a daemon, deployment meant installing two gateways (for HA) on a bastion and thanks to VPC peering, that was it. StrongDM's gateways intelligently route all database and SSH sessions through the lowest latency path.

Automation

Because Betterment runs ASG's to run mission critical services and with that thousands of boxes spin up and down throughout the day. Any access control system needs to support that ephemeral infrastructure. StrongDM made that process easy and automated. Since everyone likes LAMBDA these days, we wrote some LAMBDA function as the glue to keep StrongDM's server inventory up-to-date. Additionally, StrongDM's gateways auto-register as they spin up using 3 tools: Packer to build gateway images, CloudInit to register (whenever an instance boots, it runs a set of scripts and can be customized per boot or start basis), and Terraform to deploy the infrastructure.

Anytime Betterment runs a Terraform plan, it goes in and looks up the StrongDM gateway instance. And it will just pull the latest one, which makes it really easy to actually deploy security updates to these gateways, or if they wanted to change parameters about them, they can do that easily.

Using AWS, occasionally there were instances that that didn't terminate cleanly, sometimes they would get shut down and then it would get stuck somewhere and it wouldn't really finish. So Betterment wrote a lambda script that will iterate over the relay, the servers and the inventory. And because of the way we named them, we can actually like infer some information about them in AWS. And then what they do is with with their code, basically say, what's all of the boxes, and then iterate over them and if the server is not in AWS, basically run the StrongDM admin command and delete them. And because the instance ID is going it is definitely going to be unique for each account, they can use that as a shortcut to delete that box from inventory.

A code snippet

The engineers at at StrongDM created a JSON output so Betterment can just load that right up into an object. And then they iterate over them and look right away and see how many instances we have in inventory. And then from there, they can loop over and for instance, delete in StrongDM if it's not running.  If there are instances that are in sort of known bad states - shutting down, if they're in stopped, or stopping, or they're terminated, they should not be in StrongDM inventory, because they won't be connectable. So essentially, all Betterment has to do is use Boto within the lambda function, which will  try and access some information about it. And then if there is an exception, they can infer that it's not there. Or if they just see that it's like shutting down or something like that, they can say, No, it's not there. So let's delete it from inventory. And then the inventory is nice and green, and happy.

Audit Trail

Because StrongDM deconstructs each protocol (ex: MySQL, MongoDB, SSH) instead of just forwarding along the request, StrongDM logs every query & SSH command, making it easy for the SRE team to answer who did what, when and where.

Automate the Boring Stuff and Be Boring

Chris also discussed the team's most important mantras:

  1. If Betterment's SRE team has to repeat the same task three times, they write a script.
  2. They're big fans of using boring technology. All the tools in this talk were stuff in bash, or Python, no need to be clever with anything. There's no need for complexity. Just get it done quickly.
Chris Becker's Conference Conclusions

StrongDM - A Really Nice Unix Citizen

According to Chris, StrongDM is "a really nice Unix citizen." What he means is it makes it really easy to pipe the output of StrongDM directly into files and things like that to work with, as well as just being like nicely deployable, because it's just one sort of go binary that can be thrown anywhere they want.

To learn more on how StrongDM helps companies with auditing, make sure to check out our Auditing Use Case.


About the Author

, Chairman of the Board, began working with startups as one of the first employees at Cross Commerce Media. Since then, he has worked at the venture capital firms DFJ Gotham and High Peaks Venture Partners. He is also the host of Founders@Fail and author of Inc.com's "Failing Forward" column, where he interviews veteran entrepreneurs about the bumps, bruises, and reality of life in the startup trenches. His leadership philosophy: be humble enough to realize you don’t know everything and curious enough to want to learn more. He holds a B.A. and M.B.A. from Columbia University. To contact Schuyler, visit him on LinkedIn.

StrongDM logo
💙 this post?
Then get all that StrongDM goodness, right in your inbox.

You May Also Like

How Fixing Access Improves Security & Productivity Webinar
How Fixing Access Improves Security & Productivity Webinar
Breaking the Cycle: How access, security, and productivity create a vicious cycle, how this manifests in the real world, and importantly, how to break it.
Devising Cloud Strategies and Solutions
Event: Devising Cloud Strategies and Solutions
Attend this MegaCast for a ton of new ideas on how to use the cloud to advance your long-range IT goals. What are the ways that software-as-a-service could replace some on-premises infrastructure? How could a switch to modern applications supercharge an area of your business? Where could an expansion of infrastructure-as-a-service usage reduce your overall costs or dramatically improve your capacity and capability?
DevSecOps: The Core Curriculum Opening Remarks
DevSecOps: The Core Curriculum Opening Remarks
Listen to CEO Liz Zalman give opening remarks at the 2019 DevSecOps conference!
Why ASICS Digital Builds 12-Factor Apps
Why ASICS Digital Builds 12-Factor Apps with a Focus on Infrastructure
John Noss is a Senior Site Reliability Engineer at ASICS Digital, formerly Run Keeper. In this talk, he shares how ASICS Digital builds 12-Factor apps with an emphasis on infrastructure.
How Hearst Eliminates DevOps Complexity
How Hearst Eliminates DevOps Complexity -- An Architecture Review
In this talk, Jim Mortko (responsible for leading all Internet-based engineering and digital production efforts) and DevOps Engineer Manuel Maldonado, they discuss how Hearst eliminated DevOps complexity through automation and tooling decisions. Listen as they walk through their services and application architecture and download the slides now.