Terraform BAU — SOPs for AWS Infra Engineers

01 Files deep-dive — what each one does, line by line

New engineers see seven files with confusing names. Here is what each is for, why it exists, and what goes inside it. Terraform reads every .tf file in the current folder, alphabetically, and stitches them together as one big config. The filenames are pure convention — but follow the convention because that is what every reviewer expects.

1.1 — versions.tf · the contract

Pins the Terraform binary version and provider versions. First file the senior writes; last one to change.

Copyterraform {
  required_version = ">= 1.6.0, < 2.0.0"     # your binary must be in this range

  required_providers {
    aws = {
      source  = "hashicorp/aws"             # registry namespace
      version = "~> 5.74"                   # 5.74.x ok, 6.x not ok
    }
    random = { source = "hashicorp/random", version = "~> 3.6" }
    tls    = { source = "hashicorp/tls",    version = "~> 4.0" }
  }
}

1.2 — providers.tf · how Terraform talks to AWS

Copyprovider "aws" {                          # the default (un-aliased) provider
  region = var.aws_region

  assume_role {                            # engineer SSO -> deploy role
    role_arn     = var.deploy_role_arn
    session_name = "tf-${var.environment}"
  }

  default_tags {                           # tag EVERY resource automatically
    tags = {
      Environment = var.environment
      ManagedBy   = "terraform"
    }
  }
}

provider "aws" {                          # aliased provider, e.g. another region
  alias  = "us_west"
  region = "us-west-2"
}
# Inside a resource use:  provider = aws.us_west

1.3 — backend.tf · where state lives

Copyterraform {
  backend "s3" {
    bucket         = "lf-tfstate-nonprod-222"     # the bucket per account
    key            = "uat/network.tfstate"          # <-- the per-env knob
    region         = "us-east-1"
    dynamodb_table = "lf-tfstate-locks"
    encrypt        = true
  }
}

1.4 — variables.tf · declarations only, no values

This file declares what the config accepts as input. It never holds values. Values come from *.tfvars, -var, or TF_VAR_* env vars.

Copy# variables.tf - declarations

variable "environment" {
  type        = string                       # required type
  description = "prod | prod-support | uat | test"

  validation {                              # enforce shape at plan time
    condition     = contains(["prod","prod-support","uat","test"], var.environment)
    error_message = "environment must be prod, prod-support, uat, or test."
  }
}

variable "vpc_cidr" {
  type        = string
  description = "VPC CIDR block, /16"
  default     = "10.0.0.0/16"             # default = optional input
}

variable "db_password" {
  type        = string
  sensitive   = true                         # hides value in plan output
  description = "Master DB password (typically supplied by Secrets Manager, not tfvars)"
}

variable "app_servers" {                     # complex types are first-class
  type = list(object({
    name          = string
    instance_type = string
    public        = bool
  }))
  default = []
}

1.5 — data.tf · read-only lookups

Data sources query AWS without managing the resource. Data sources re-evaluate every run; good for AMIs (you want the newest) but means you can get unexpected diffs — pin AMIs in production.

Copy# data.tf

data "aws_caller_identity" "current" {}        # who am I?
data "aws_region" "current" {}                  # the region

data "aws_availability_zones" "available" {
  state = "available"
}

data "aws_ami" "al2023" {
  most_recent = true
  owners      = ["amazon"]
  filter {
    name   = "name"
    values = ["al2023-ami-*-x86_64"]
  }
}

# Use anywhere as data.<type>.<name>.<attr>
# e.g. data.aws_caller_identity.current.account_id

1.6 — main.tf · the table of contents

Despite the name, main.tf rarely contains the bulk of code in a real repo. Resources live in modules. The env-level main.tf is just "this env composes these modules".

Copy# envs/uat/main.tf

# 1. local values - computed once, used in many places
locals {
  name_prefix = "lf-${var.environment}"
  account_id  = data.aws_caller_identity.current.account_id
  tags = merge(var.common_tags, {
    Environment = var.environment
    Account     = local.account_id
  })
}

# 2. module composition - the actual stack for this env
module "network" {
  source     = "../../modules/network"
  name       = "${local.name_prefix}-vpc"
  cidr_block = var.vpc_cidr
  tags       = local.tags
}

module "security" {
  source      = "../../modules/security"
  name_prefix = local.name_prefix
  vpc_id      = module.network.vpc_id          # cross-module reference
  tags        = local.tags
}

1.7 — locals.tf · computed values used internally

Variables are inputs. Outputs are exports. Locals are computed values used inside. Private to the folder.

Copylocals {
  name_prefix = "lf-${var.environment}"

  base_tags = {
    Environment = var.environment
    ManagedBy   = "terraform"
    CostCenter  = lookup(var.common_tags, "CostCenter", "unallocated")
  }
  tags = merge(local.base_tags, var.common_tags)

  is_prod_like = contains(["prod", "prod-support"], var.environment)
  azs          = slice(data.aws_availability_zones.available.names, 0, 2)

  public_subnets  = [for i, az in local.azs : cidrsubnet(var.vpc_cidr, 8, i + 1)]
  private_subnets = [for i, az in local.azs : cidrsubnet(var.vpc_cidr, 8, i + 11)]
}

1.8 — outputs.tf · the public surface

Copyoutput "vpc_id" {
  value       = module.network.vpc_id
  description = "VPC id of this environment"
}

output "private_subnet_ids" {
  value = module.network.private_subnet_ids
}

output "db_secret_arn" {
  value     = module.db.secret_arn
  sensitive = true
}

1.9 — *.tfvars files · the values

Variables declare; tfvars supply. Per-env tfvars is the single most important pattern in this guide.

Copy# envs/uat/uat.tfvars
environment      = "uat"
aws_region       = "us-east-1"
deploy_role_arn  = "arn:aws:iam::222222222222:role/TerraformDeploy"
vpc_cidr         = "10.20.0.0/16"
instance_type    = "t3.medium"
common_tags = {
  CostCenter = "CC-1042"
  Owner      = "infra-platform"
  DataClass  = "internal"
}

Load it explicitly: terraform plan -var-file=uat.tfvars. Anything named terraform.tfvars or *.auto.tfvars auto-loads — avoid those in multi-env work.

1.10 — load order recap

1. Reads backend.tf only — needed before anything else.
2. Loads every .tf in the folder (alphabetical order is irrelevant; references resolve automatically).
3. Resolves variable values: defaults → terraform.tfvars → *.auto.tfvars → -var-file → -var → TF_VAR_* (last wins).
4. Resolves data sources (queries AWS).
5. Builds the resource graph and plans diffs against state.

02 Environment variables & credentials — how Terraform finds AWS

"It worked on my laptop but failed in CI." Almost always an env-var or credential issue.

2.1 — Terraform's own environment variables

Copy# Examples - shell or CI config
export TF_VAR_db_password="$(aws secretsmanager get-secret-value --secret-id db/master --query SecretString --output text)"
export TF_LOG=DEBUG
export TF_LOG_PATH=/tmp/tf-$(date +%s).log
export TF_PLUGIN_CACHE_DIR="$HOME/.terraform.d/plugin-cache"
export TF_IN_AUTOMATION=1
export TF_INPUT=0

2.2 — AWS provider credential chain (the real source of bugs)

The AWS provider tries these in order and uses the first one it finds. Knowing this order saves hours.

1. Static credentials in the provider block (don't do this).
2. AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY + AWS_SESSION_TOKEN env vars.
3. AWS_PROFILE → reads ~/.aws/credentials and ~/.aws/config.
4. EC2 instance metadata (IMDS) when running on EC2 with an IAM role.
5. ECS / EKS task role.

2.3 — Recommended setup · AWS IAM Identity Center (SSO)

Copy# ~/.aws/config
[profile sso-base]
sso_session       = lf
sso_account_id    = 333333333333
sso_role_name     = DeveloperAccess
region            = us-east-1

[profile tf-nonprod]
source_profile    = sso-base
role_arn          = arn:aws:iam::222222222222:role/TerraformDeploy
region            = us-east-1
role_session_name = pponnam-tf-nonprod

[profile tf-prod]
source_profile    = sso-base
role_arn          = arn:aws:iam::111111111111:role/TerraformDeploy
region            = us-east-1
role_session_name = pponnam-tf-prod

[sso-session lf]
sso_start_url  = https://lf.awsapps.com/start
sso_region     = us-east-1

Copy# Daily flow
aws sso login --sso-session lf
export AWS_PROFILE=tf-nonprod
aws sts get-caller-identity                   # verify
cd envs/uat
terraform plan -var-file=uat.tfvars

2.4 — Per-folder env vars with direnv

Copy# envs/uat/.envrc - committed (no secrets)
export AWS_PROFILE=tf-nonprod
export AWS_REGION=us-east-1
export AWS_SDK_LOAD_CONFIG=1
export TF_PLUGIN_CACHE_DIR="$HOME/.terraform.d/plugin-cache"

Copycd envs/uat
direnv allow      # env loads on every cd

2.5 — Variable precedence (memorise)

From lowest to highest priority — later wins:

1. default in variables.tf.
2. terraform.tfvars (auto-loaded).
3. *.auto.tfvars in alphabetical order.
4. TF_VAR_<name> environment variables.
5. -var-file=foo.tfvars on the CLI (in order given).
6. -var name=value on the CLI.

2.6 — Quick credential debug recipe

Copyaws sts get-caller-identity                            # whoami
echo "AWS_PROFILE=$AWS_PROFILE  AWS_REGION=$AWS_REGION"    # active profile
aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::222222222222:role/TerraformDeploy \
  --action-names ec2:CreateVpc                          # can role do this?
TF_LOG=DEBUG TF_LOG_PATH=/tmp/tf.log terraform plan -var-file=uat.tfvars
tail -f /tmp/tf.log                                     # debug output

03 SOP — create a new environment from scratch

Use case: the team needs a new env called preprod in the non-prod account for a regulated workload before it gets promoted to prod. Senior gives you the ticket. This is what you do, top to bottom. Do not skip steps; do not reorder them.

Copy# What needs to exist before step 4:
arn:aws:iam::222222222222:role/TerraformDeploy            # reused
s3://lf-tfstate-nonprod-222/preprod/network.tfstate       # key implicit on first put
DynamoDB table lf-tfstate-locks                           # shared

Copygit checkout main && git pull --ff-only
git checkout -b infra-2104-add-preprod-env

Copycp -r envs/uat envs/preprod
cd envs/preprod
ls
# backend.tf  main.tf  outputs.tf  providers.tf  uat.tfvars  variables.tf  versions.tf  .envrc

Copygit mv uat.tfvars preprod.tfvars

Copyterraform {
  backend "s3" {
    bucket         = "lf-tfstate-nonprod-222"
    key            = "preprod/network.tfstate"     # <-- changed
    region         = "us-east-1"
    dynamodb_table = "lf-tfstate-locks"
    encrypt        = true
  }
}

Copyprovider "aws" {
  region = var.aws_region
  assume_role {
    role_arn     = var.deploy_role_arn
    session_name = "tf-preprod-${terraform.workspace}"
  }
  default_tags {
    tags = {
      Environment = "preprod"
      ManagedBy   = "terraform"
    }
  }
}

Copyenvironment     = "preprod"
aws_region      = "us-east-1"
deploy_role_arn = "arn:aws:iam::222222222222:role/TerraformDeploy"
vpc_cidr        = "10.40.0.0/16"                     # non-overlapping
instance_type   = "m6i.large"                       # prod-shaped
cluster_size    = 2
backup_retention_days = 14
common_tags = {
  CostCenter = "CC-2104"
  Owner      = "infra-platform"
  DataClass  = "confidential"
}

Copyvariable "environment" {
  type = string
  validation {
    condition     = contains(["prod","prod-support","uat","test","preprod"], var.environment)
    error_message = "environment must be one of the supported values."
  }
}

Copy/envs/preprod/  @lf/infra-platform @lf/security

Copycd envs/preprod
terraform fmt -recursive ../../
terraform init
# If it offers to copy state from an old key - say NO. Fresh env, fresh state.

Copyterraform validate
terraform plan -var-file=preprod.tfvars -out=tfplan

Copygit add -A
git commit -m "INFRA-2104: add preprod environment in non-prod account"
git push -u origin HEAD
gh pr create --fill

Copycd envs/preprod
terraform plan -var-file=preprod.tfvars
# EXPECT: "No changes. Your infrastructure matches the configuration."

terraform output
aws ec2 describe-vpcs --vpc-ids $(terraform output -raw vpc_id)

04 SOP — daily BAU: edit existing infra and apply

Use case: a ticket lands — "INFRA-2210: increase UAT app tier from m6i.large to m6i.xlarge for load testing next Tuesday." Internalise this routine.

Copygit checkout main && git pull --ff-only
cd envs/uat
direnv allow
terraform init
terraform plan -var-file=uat.tfvars
# EXPECT: "No changes. Your infrastructure matches the configuration."

Copygit checkout -b infra-2210-uat-app-tier-xlarge

Copygrep -n instance_type envs/uat/uat.tfvars
# envs/uat/uat.tfvars:5: instance_type = "m6i.large"

# If the value isn't in tfvars, walk up: env main.tf -> module variables.tf
grep -rn instance_type modules/compute/

Copy# envs/uat/uat.tfvars (diff)
-instance_type = "m6i.large"
+instance_type = "m6i.xlarge"

Copyterraform fmt -recursive ../../
terraform validate

Copyterraform plan -var-file=uat.tfvars -out=tfplan
# module.app.aws_launch_template.app will be updated in-place
#   ~ instance_type = "m6i.large" -> "m6i.xlarge"
# Plan: 0 to add, 1 to change, 0 to destroy.

Copygit diff
git add -A
git commit -m "INFRA-2210: bump UAT app tier to m6i.xlarge for load test"
git push -u origin HEAD
gh pr create --fill

Copygh pr merge --squash --delete-branch

Copygh run watch                              # GitHub Actions live
# Drop a Slack note in #infra-changes when apply starts and finishes.

Copyterraform plan -var-file=uat.tfvars
# EXPECT: "No changes."

aws autoscaling describe-auto-scaling-groups \
  --auto-scaling-group-names lf-uat-app-asg \
  --query 'AutoScalingGroups[].LaunchTemplate.Version'

aws ec2 describe-instances \
  --filters "Name=tag:Environment,Values=uat" "Name=instance-state-name,Values=running" \
  --query 'Reservations[].Instances[].[InstanceId,InstanceType]'

4.x — The "always-on" minimum-viable cadence

4.y — "I broke something" recovery flow

1. Stop. Don't make it worse with another apply.
2. Re-read the latest plan/apply log. Capture the error.
3. Decide: roll forward (small fix PR) or roll back (git revert <sha> on main, merge, CD applies the previous state).
4. If state and reality disagree, see Part 1 section 9 troubleshooting. Use terraform import / state rm rather than fighting plan.
5. Document what happened in the ticket. If customer-impacting, file a postmortem.

05 New-engineer onboarding — Day 1, Week 1, Month 1

Hand this to a new engineer on day one. Senior validates each box as it gets checked. By the end of month one, they should be running BAU tickets unsupervised on test/uat and shadowing on prod.

06 Senior's insights — the unwritten rules

Things you'd learn the hard way over five years. Read them now.

Copyresource "aws_rds_cluster" "prod" {
  lifecycle {
    prevent_destroy = true
    ignore_changes  = [master_password]
  }
}