Deploying Terraform Code via AWS CodeBuild and AWS CodePipeline

In a previous post i demonstrated how to use Terraform to deploy an Amazon API Gateway backed by Lambda. This was an example of how you can use Terraform as Infrastructure as Code to manage your resources in AWS.

https://dev.to/aws-builders/deploying-amazon-api-gateway-and-lambda-with-terraform-1i2o

However, the way of deploying the Terraform code to AWS was to manually run the terraform plan and apply steps locally each time we want to update the API Gateway or our Lambda code. We can improve on this and automate the deployment of the Terraform Code using AWS CodePipeline and CodeBuild.

To do this, we will still need to the Deploy the CICD Pipeline initially locally (or from a machine in AWS), which will need to be monitored to update the pipeline itself, but this Pipeline will allow any code committed to the GitHub Repository to be deployed into AWS Automatically, and should only need minimal maintenance as it should not change on the same frequency as the API Gateway / Lambda Code.

To ensure we don't just blindly deploy code that hasn't been checked, we'll split this out into stages:

CodePipeline
- Download Source Code from API Gateway Repository
- Run a Planning Step in AWS CodeBuild.
  - Download and Install Terraform
  - Initialise the Terraform Environment with an S3 Backend
  - Run the Terraform Plan, and save the output to an Artifact
- Send an Email via SNS to say the pipeline is awaiting approval
  - Await Manual Approval
- Run an Apply Step in AWS CodeBuild.
  - Download and Install Terraform
  - Initialise the Terraform Environment with an S3 Backend
  - Run the Terraform Apply using the Artifact from the Planning stage

This is what we are creating in terms of the flow:

Terraform - main.tf

For the main terraform setup, we will need some prerequisites, in terms of the backend and provider etc.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.0"
    }
  }

  # S3 backend configuration
  backend "s3" {
    bucket = "REPLACE_WITH_YOUR_TERRAFORM_STATE_BUCKET"
    key    = "REPLACE_WITH_FOLDER_NAME/terraform.tfstate"
    region = var.aws_region
    encrypt = true
  }
}

provider "aws" {
  region = var.aws_region
}

This sets up terraform to use the AWS provider and provides a location for the state file (that tracks the changes in your architecture) in S3. You should create an S3 bucket and provide it here, and insert a folder name (such as ci-cd-pipeline) for the state file to be stored in.

S3 Artifacts - main.tf

For the codepipeline, we will need to pass artifacts between the different stages (mainly the plan file), so we need somewhere to store these artifacts during the pipelines run.

resource "aws_s3_bucket" "codepipeline_bucket" {
  bucket_prefix = "tf-pipeline-artifacts-"
  force_destroy = true
}

resource "aws_s3_bucket_ownership_controls" "codepipeline_bucket" {
  bucket = aws_s3_bucket.codepipeline_bucket.id
  rule {
    object_ownership = "BucketOwnerPreferred"
  }
}

resource "aws_s3_bucket_acl" "codepipeline_bucket" {
  depends_on = [aws_s3_bucket_ownership_controls.codepipeline_bucket]
  bucket     = aws_s3_bucket.codepipeline_bucket.id
  acl        = "private"
}

resource "aws_s3_bucket_versioning" "codepipeline_bucket" {
  bucket = aws_s3_bucket.codepipeline_bucket.id
  versioning_configuration {
    status = "Enabled"
  }
}

Notifications - main.tf

When we have the approval step, we'll need to notify someone that an approval is ready, so we'll create an SNS topic to send an email.

resource "aws_sns_topic" "approval_topic" {
  name = "terraform-approval-topic"
}

resource "aws_sns_topic_subscription" "approval_email" {
  topic_arn = aws_sns_topic.approval_topic.arn
  protocol  = "email"
  endpoint  = var.notification_email
}

This will set up a topic and subscribe an email to that topic (which we'll pass in via the variables file later on).

IAM Roles - main.tf

We need to create permissions for our CodeBuild to perform its tasks, as well as the code pipeline and SNS.

# CodeBuild IAM Role
resource "aws_iam_role" "codebuild_role" {
  name = "terraform-codebuild-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "codebuild.amazonaws.com"
        }
      }
    ]
  })
}

resource "aws_iam_role_policy" "codebuild_policy" {
  name = "terraform-codebuild-policy"
  role = aws_iam_role.codebuild_role.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
          "logs:PutLogEvents"
        ]
        Resource = "*"
      },
      {
        Effect = "Allow"
        Action = [
          "s3:GetObject",
          "s3:PutObject",
          "s3:GetObjectVersion",
          "s3:ListBucket"
        ]
        Resource = [
          aws_s3_bucket.codepipeline_bucket.arn,
          "${aws_s3_bucket.codepipeline_bucket.arn}/*",
          "arn:aws:s3:::REPLACE_WITH_YOUR_TERRAFORM_STATE_BUCKET",
          "arn:aws:s3:::REPLACE_WITH_YOUR_TERRAFORM_STATE_BUCKET/*"
        ]
      }
    ]
  })
}

# CodePipeline IAM Role
resource "aws_iam_role" "codepipeline_role" {
  name = "terraform-codepipeline-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "codepipeline.amazonaws.com"
        }
      }
    ]
  })
}

resource "aws_iam_role_policy" "codepipeline_policy" {
  name = "terraform-codepipeline-policy"
  role = aws_iam_role.codepipeline_role.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "s3:GetObject",
          "s3:PutObject",
          "s3:GetObjectVersion",
          "s3:GetBucketVersioning"
        ]
        Resource = [
          aws_s3_bucket.codepipeline_bucket.arn,
          "${aws_s3_bucket.codepipeline_bucket.arn}/*"
        ]
      },
      {
        Effect = "Allow"
        Action = [
          "codebuild:BatchGetBuilds",
          "codebuild:StartBuild"
        ]
        Resource = "*"
      },
      {
        Effect = "Allow"
        Action = [
          "sns:Publish"
        ]
        Resource = aws_sns_topic.approval_topic.arn
      }
    ]
  })
}

The codebuild policy will allow our codebuild project to create logs and to set and retrieve files from our s3 bucket.

The code pipeline role is allowed to be assumed by the AWS CodePipeline service and it can also get S3 artifacts from our bucket, as well as start CodeBuild and send messages to our SNS Topic.

CodeBuild Projects - main.tf

We need 2 CodeBuild Projects, one for each of our Plan and Apply steps.

# Terraform Plan Project
resource "aws_codebuild_project" "terraform_plan" {
  name          = "terraform-plan"
  description   = "Run terraform plan"
  service_role  = aws_iam_role.codebuild_role.arn
  build_timeout = 15

  artifacts {
    type = "CODEPIPELINE"
  }

  environment {
    type                        = "LINUX_CONTAINER"
    compute_type                = "BUILD_GENERAL1_SMALL"
    image                       = "aws/codebuild/amazonlinux2-x86_64-standard:3.0"
    privileged_mode             = false
  }

  logs_config {
    cloudwatch_logs {
      group_name = "terraform-plan-logs"
    }
  }

  source {
    type      = "CODEPIPELINE"
    buildspec = <<EOF
version: 0.2

phases:
  install:
    runtime-versions:
      python: 3.8
    commands:
      - wget https://releases.hashicorp.com/terraform/1.0.11/terraform_1.0.11_linux_amd64.zip
      - unzip terraform_1.0.11_linux_amd64.zip
      - mv terraform /usr/local/bin/
  pre_build:
    commands:
      - echo "Starting Terraform plan phase..."
      - terraform --version
  build:
    commands:
      - terraform init -input=false
      - terraform plan -input=false -out=tfplan
  post_build:
    commands:
      - echo "Completed Terraform plan phase"

artifacts:
  files:
    - tfplan
    - .terraform/**/*
    - '**/*'
EOF
  }
}

# Terraform Apply Project
resource "aws_codebuild_project" "terraform_apply" {
  name          = "terraform-apply"
  description   = "Run terraform apply"
  service_role  = aws_iam_role.codebuild_role.arn
  build_timeout = 15

  artifacts {
    type = "CODEPIPELINE"
  }

  environment {
    type                        = "LINUX_CONTAINER"
    compute_type                = "BUILD_GENERAL1_SMALL"
    image                       = "aws/codebuild/amazonlinux2-x86_64-standard:3.0"
    privileged_mode             = false
  }

  logs_config {
    cloudwatch_logs {
      group_name = "terraform-apply-logs"
    }
  }

  source {
    type      = "CODEPIPELINE"
    buildspec = <<EOF
version: 0.2

phases:
  install:
    runtime-versions:
      python: 3.8
    commands:
      - wget https://releases.hashicorp.com/terraform/1.0.11/terraform_1.0.11_linux_amd64.zip
      - unzip terraform_1.0.11_linux_amd64.zip
      - mv terraform /usr/local/bin/
  pre_build:
    commands:
      - echo "Starting Terraform apply phase..."
      - terraform --version
  build:
    commands:
      - terraform init -input=false
      - terraform apply -input=false tfplan
  post_build:
    commands:
      - echo "Completed Terraform apply phase"

artifacts:
  files:
    - '**/*'
EOF
  }
}

The first project is our Plan stage. Firstly we create the codebuild project, give it a name and service role. Then we apply a 15 minute time out and set the environment variables (linux and small container running amazon linux 2 is fine for our needs).

We then set up the logs and the Source. This is where the main set up is for CodeBuild.

CodeBuild is configured with a YAML file that allows commands to be run in phases. This is the buildspec.yml file (in this case we have it inline, but you could reference it from your repository). the full specifications are here https://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html

For our purposes, the important ones are the phases. These blocks run in a specific order, and execute the commands in those blocks. The order is install, pre_build, build, and post_build. You can run commands as different users, have complex error handling, error catching etc here.

In our case we'll do the following. Both CodeBuild projects are nearly identical, just with the plan or apply specific logs, locations etc, and the only command difference is terraform plan vs terraform apply

Install phase:

set up python
set up terraform
- download terraform
- unzip terraform
- move it to the user executable directory

Pre build phase:

echo a message to screen
output the terraform version to show terraform is working (this can help catch errors for debugging)

Build phase:

initialise terraform (syncs up with our back end)
plan or apply the terraform, outputting to a tfplan file in the plan stage, then using this same file in the apply stage

Post build phase:

output a complete message

We then use the Artifacts section to pass the tfplan file from the plan stage over to the apply stage.

Code Pipeline - main.tf

Now that we have our components, we need to assemble them into a Code Pipeline.

resource "aws_codepipeline" "terraform_pipeline" {
  name     = "terraform-deployment-pipeline"
  role_arn = aws_iam_role.codepipeline_role.arn

  artifact_store {
    location = aws_s3_bucket.codepipeline_bucket.bucket
    type     = "S3"
  }

  # Source Stage - Pull from GitHub
  stage {
    name = "Source"

    action {
      name             = "Source"
      category         = "Source"
      owner            = "ThirdParty"
      provider         = "GitHub"
      version          = "1"
      output_artifacts = ["source_output"]

      configuration = {
        Owner      = var.github_repo_owner
        Repo       = var.github_repo_name
        Branch     = var.github_branch
        OAuthToken = var.github_token
      }
    }
  }

  # Plan Stage - Run Terraform Plan
  stage {
    name = "Plan"

    action {
      name             = "TerraformPlan"
      category         = "Build"
      owner            = "AWS"
      provider         = "CodeBuild"
      version          = "1"
      input_artifacts  = ["source_output"]
      output_artifacts = ["plan_output"]

      configuration = {
        ProjectName = aws_codebuild_project.terraform_plan.name
      }
    }
  }

  # Approval Stage - Manual approval before apply
  stage {
    name = "Approval"

    action {
      name     = "Approval"
      category = "Approval"
      owner    = "AWS"
      provider = "Manual"
      version  = "1"

      configuration = {
        NotificationArn = aws_sns_topic.approval_topic.arn
        CustomData      = "Please review the terraform plan and approve to apply changes"
      }
    }
  }

  # Apply Stage - Run Terraform Apply
  stage {
    name = "Apply"

    action {
      name            = "TerraformApply"
      category        = "Build"
      owner           = "AWS"
      provider        = "CodeBuild"
      version         = "1"
      input_artifacts = ["plan_output"]

      configuration = {
        ProjectName = aws_codebuild_project.terraform_apply.name
      }
    }
  }
}

The first section will set up the pipeline and its role and artifact store location.

The next few sections are stages, we'll go through these one at a time.

Source Stage

This source stage is used to link up to GitHub and pull down the source code, and output it as an artifact to be passed through to th next stage. This is currently set up to use OAuth with GitHub, but this can be changed to different providers and types of connections. (The CodeStar connection method is the current recommended way to do this, but i've stuck with OAuth here for simplicity)

https://docs.aws.amazon.com/codepipeline/latest/userguide/integrations-action-type.html#integrations-source

Plan Stage

This stage is set up to use our previously created CodeBuild for planning and links to the CodeBuild Project

Approval Stage

This stage sets up a manual approval so you can go into the console and click approve, and sends out a notification to the email address to say this is ready.

Apply Stage

Similar to Planning we link up to our Codebuild for the apply step.

Outputs - main.tf

If required, you can output some information such as the codepipeline url and the sns topic arn for information:

output "codepipeline_url" {
  description = "URL to the CodePipeline console for the created pipeline"
  value       = "https://${var.aws_region}.console.aws.amazon.com/codesuite/codepipeline/pipelines/${aws_codepipeline.terraform_pipeline.name}/view?region=${var.aws_region}"
}

output "approval_topic_arn" {
  description = "ARN of the SNS topic used for pipeline approval notifications"
  value       = aws_sns_topic.approval_topic.arn
}

Variables - variables.tf

We've set up some variables in the main file to control deployments, covering the region and github information, these should go in a variables.tf file so you can pass the required parameters into the terraform.

`variable "aws_region" {
description = "The AWS region to deploy resources into"
type = string
default = "eu-west-1"
}

variable "github_repo_owner" {
description = "GitHub repository owner"
type = string
default = "REPO_OWNER_NAME"
}

variable "github_repo_name" {
description = "GitHub repository name"
type = string
default = "REPO_NAME"
}

variable "github_branch" {
description = "GitHub repository branch"
type = string
default = "main"
}

variable "github_token" {
description = "GitHub OAuth token"
type = string
sensitive = true
}

variable "notification_email" {
description = "Email address to receive approval notifications"
type = string
}`

Conclusion

This method of planning and applying terraform via CodeBuild can be useful when deploying terraform in an automated way whenever the repository containing the terraform is updated, while still keeping a manual approval step in place, and allowing the plan to be reviewed in the CodeBuild plan step logs.

John Walker @uzusan