Much has been written about establishing a multi-account strategy. From the Code Spaces incident onward, putting all your AWS eggs in a single basket has been an design choice AWS and most cloud and cloud security professionals have spoken against.
As of writing this, my current employer has 830+ AWS accounts. That’s not at the extreme end (I recall hearing at re:Invent that Fidelity has over 10,000), but it is certainly beyond the “we can do this by hand” stage of cloud maturity. I’ve blogged in the past about how we’ve built a security account, implemented Antiope for resource discovery and scorecards for reporting. These processes scaled from the 300 or so AWS accounts we had a year ago to the 800+ we’ve had now that our infosec team spans across what were three separate companies.
When we started the cloud journey back in late 2015, the infrastructure team said, “we’ll have five accounts”, they were split across four major brand categories and one that would act as a development account. The rationale for that was it was a lot of effort to point & click your way to a functioning VPC with DirectConnect and they didn’t want to do that too often. Five years later, these five SuperFund accounts still pose operational and security risk and keep gurgling up their toxic sludge.
One of the first things I pushed hard for when I took on the Cloud Security role was that every team would get their own account (or preferably two). By mid-2017 the cloud team and split from infrastructure and the fatberg that was their tagging strategy was beginning to clog up productivity. Meanwhile the collective effort to close a cloud governance audit-finding meant we were digging up accounts from all over the world that we didn’t know about. By the end of 2017 we had over 100 accounts. By the end of 2018 it was closer to 250.
Even then we had a multi-payer challenge. While 95% of the accounts were in the company’s main payer, we had acquired three other companies over the years and had not forced a migration of their child accounts into the company’s primary payer account. Paradoxically, it was deemed easier to involve ours and AWS’s legal departments to amend the EDP than it was to migrate the children. Antiope was designed to support multiple payer accounts as part of its inventory run, and our security account was trusted across all the AWS Organizations. In the process AWS Organizations was released and our payer accounts were turned into Organizational Master accounts.
By the fall of 2019, the just shy of 300 accounts I was responsible for ballooned to well over 700. And with that came a new challenge. The number of payers ballooned to about 15 give or take. My company and one of its sisters were doing the right thing will keeping the children in a single payer. The other sibling company had approximately 9 payers (they couldn’t cost-allocate otherwise). As the org-chart started to settle many of the 30 or so accounts across those 9 payers were migrated into our Organization. The other sibling had about 400 accounts, but never enabled Organizations. That was a process that took several months to complete. One group decided on their own to go out and implement Control Tower.
Meanwhile in South Lake Union - AWS was doing what they always seem to do. Add new stuff.
I always say, “Architect for the AWS you have, not the AWS you want”. I guess they’d call that Bias for Action or something. I say it’s a way to get off your fat lazy datacenter and start migrating your applications. Part of that means you need to improvise. Antiope came from limitations in AWS Config. My CloudTrail template and concept of an artifacts bucket were built before Organization CloudTrail was a thing. Our account creation process pre-dates Control Tower and landing zones. It doesn’t make sense to re-engineer at this time.
However, we’re reaching a point where our organizational maturity and AWS’s support for Organizations is beginning to collide. The ability to delegate an account in the organization as a security tooling account means the infosec team no longer needs to rely on a central cloud engineering team or hundreds of individual account owners. The team(s) that manage the payer can delegate control to an account infosec controls, and we can manage the deployments. AWS first released the capability with IAM Access Analyzer, and just last month did the same for GuardDuty.
Till now, we’ve been sending all our GD findings across organizational boundaries to our single GuardDuty master account. We were 50% of the way through deploying GuardDuty in one of our Organizations when that happened. At this point we’re able to stop reaching out individually, and we can deploy it ourselves.
There is one slight concern with infosec “just turning on GuardDuty” across the organization. We now possess the power to incur charges in their account. Perhaps thats a distinction without a difference, because we write the policies that require them to have GuardDuty. But with the new model, we’re pushing the button rather than writing the words that force them to push the button. It would be nice when the payer delegates administrative control for a service to an account in the organization it could also decide where the costs of the service are delegated.
I hope more and more of the security tooling AWS provides moves to this delegated admin account model. It would be nice to control AWS Inspector uniformly across an enterprise.
For us, we’re going to keep consolidating our payers. Sibling 1 and Sibling 2 will keep their organizations. The Control Tower team will keep theirs. At least one of sibling 1’s other Organizations will be kept separate for some legal/contractual reasons. Each of those organizations will end up with an account that is controlled by infosec or where infosec shares control with the cloud team. We will maintain our central account for all the payers where we discover and track children and where we run centralized discovery and our cloud security posture monitoring tools.