I’ve been on the CloudFormation side of the IaC Wars since 2014, when I started working in AWS. I’ve dabbled in terraform but never made it my primary IaC choice. For the Fooli Meme Factory, I needed to mess things up and then quickly revert the state to what it was at deployment time. So for SECCDC 2023, I ported Meme Factory almost entirely over to Terraform.
This led me to two problems. The first was the perennial issue I’ve had with Terraform from day one: “How do I manage state?". The second issue was how do I leverage some form of CI/CD tooling to allow me to leverage one of Terraform’s biggest strengths - the
terraform plan capability. Since Fooli is an AWS product, I figured that I should be able to use AWS native tools for this. I’ve used CodePipeline in the past to preview change-sets with aws-org-formation, so I thought it would be easy to find a well-worn pattern from AWS on doing it.
Apparently, there is no canonical way to use Terraform in CodeBuild, with CodePipeline as the method to review plans before applying them!!! Seriously, WTF?
This made me sad. So I decided to do it my damn self. And now I’m documenting it here for everyone else.
This solution will provide the following:
- CloudFormation Template (CFT) to deploy a CodePipeline, CodeBuild Projects, and an S3 Bucket for state and artifact handling.
- Buildspec files to perform the
- Makefiles because I’m old school like that.
Why a CloudFormation template for step 1? To get around the chicken-and-egg problem with state. The CFT will deploy and do the needful to get the account setup for the terraform pipeline without needing a terraform pipeline already in place
How it works.
When a push is made to a monitored GitHub repo, the CodePipeline will trigger. AWS’s CodeStar Connections are used to manage the integration between GitHub and CodePipeline. (As an aside: CodeStar connections are so under-the-radar I can’t even find a product page to link to. Just some API docs and the above blog post.) Anyway - CodeStar is a much better solution than previous methods that required overly-privileged GitHub Personal Access Tokens to be uploaded into shared AWS Accounts.
So CodeStar Connections will monitor GitHub and fire the pipeline on a push to the specified branch. After that, the pipeline will execute the following stages:
- Source Stage - where it downloads the code package from GitHub and stores it in the S3 Bucket.
- Terraform Plan Stage - where CodeBuild will execute the
terraform planand copy the
- Review Stage - sends a message to SNS, which (if configured) will email someone to review the output of the Terraform plan in CodeBuild.
- Apply Stage - If approved, this stage will fire up CodeBuild to do the
terraform applyon the preexisting
CodePipeline & CodeBuild
The CodePipeline is defined entirely in the CloudFormation Template. The CodeBuild projects are created by the CloudFormation Template, but the commands to be executed are defined in the build spec files.
The definition of the Plan stage in CodePipeline is:
And the Project is:
EnvironmentVariables in the pipeline are passed into the CodeBuild Project. CodePipeline substitutes environment variables that begin with a
# at execution, and the ones beginning with
$ are substituted by CloudFormation at deployment. The BuildSpec exports some environment variables, and they’re stored in the pipeline under the
TfPlan Namespace (Line 5). The
BuildSpec part of the CodeBuild Project (Line 9) defines the commands that CodeBuild will execute. That file looks like:
Lines 4-6 indicate that we will export two environment variables we want to pass back to CodePipeline,
BuildTag. These are needed to build the URL for reviewing the plan. The
artifacts section on line 23 defines the files created that CodeBuild/CodePipeline should store in S3 and pass between the pipeline stages.
The review stage consists of a manual step and looks like this:
Here we construct the
ExternalEntityLink from the
BuildTag from the plan stage. Again variables that begin with a
# are substituted by CodePipeline at execution and the ones beginning with
$ are substituted by CloudFormation at deployment. We send a message to the
PipelineNotificationsTopic which triggers an email to the user:
The apply stage is similar to the plan. In CodePipeline it looks like this:
In this case, we’re inputting the input artifacts from both GitHub and the plan (lines 5-7). We set the GitHubCode as the
PrimarySource on line 15, and that becomes the working directory. The other files are written to a different directory, and we have to move them in the BuildSpec file (line 9 below).
The BuildSpec for the apply looks like this:
The buildspec file installs terraform, moves the tfplan file back to where it’s expected, runs
make tf-init (because this is a new container), and then
I use Makefiles to simplify the process of deploying both in CodeBuild and when deploying the terraform locally.
The root Makefile for the repo looks like this:
And the Makefile in the
terraform directory is:
config.env file contains all the TF_VAR exports to feed variables to terraform similar to:
$(env).tfbackend contains the line to define the bucket:
There you have it, a complete solution to deploy Terraform in CodePipeline with CodeBuild and a manual review of the changes to be made. You can tweak the makefiles and buildspec files as you see fit. Here is the entire CloudFormation Template.
I’m surprised that there isn’t a CodeBuild container from Hashicorp or AWS with Terraform pre-installed. The effort of making one would be less than the expense of everyone curling the terraform binary and providers twice on every build.