The business

• ~6,000 employees, financial services + 2 recent subsidiaries (FinSub, RetailSub)
• Hybrid: 4 datacenters (NY, LDN, SG, FRA), Direct Connect to AWS in us-east-1, eu-west-1
• Production: EC2 + ECS + RDS + FSx Windows, microservices in private subnets behind ALB/NLB
• Identity: Okta for MFA & SSO, federated to per-account IAM via SAML (no IDC — legacy)
• EC2 fleet domain-joined to on-prem AD (corp.globalcorp.local). DNS via R53 Resolver outbound endpoint
• Each AWS account has its own Git repo + Terraform state in S3 + DynamoDB lock

Why this matters for troubleshooting

• Cross-account routing via Transit Gateway with route domains per environment
• SCP enforces tagging, region restriction, IMDSv2-only
• RAM-shared resources: PHZ associations, prefix lists, TGW

Account inventory (used throughout)

What the user sees

• Terraform apply is green; instance i-0abc123 reaches running.
• App team RDPs in via SSM port-forward, opens Event Viewer:

System > NETLOGON > Event 5719:
This computer was not able to set up a secure session with a domain
controller in domain corp.globalcorp.local because of the following:
The remote procedure call was cancelled.

cloud-init-output.log:
Add-Computer : An Active Directory domain controller (AD DC) for the
domain corp.globalcorp.local could not be contacted.

Business impact

• Orders microservice deployment paused (needs domain-joined service account)
• SLO at risk: 30 min budget left before stakeholder Slack at 4·9s
• Recurring — happened to 3 instances this week, hand-fixed each time

Constraints (read these first)

Hypothesis tree (5-step)

Bisection plan

1. DNS first. If DNS fails everything else looks broken. Resolve-DnsName -Type SRV _ldap._tcp.dc._msdcs.corp.globalcorp.local
2. L4 reach to a DC. Test-NetConnection 10.10.0.10 -Port 389 & -Port 445.
3. RPC dynamic. Most cheeky — FW rule passes well-known but drops 49152–65535. Pick a random port, test.
4. Identity. Pull the secret from instance to confirm role + KMS pass.
5. Auth. Time + OU rights are the last 5%.

From your laptop (cross-account)

# confirm identity + assume into prod-app
aws sts get-caller-identity
aws sts assume-role --role-arn arn:aws:iam::666666666666:role/FedAppDev \
    --role-session-name s001 --query Credentials

# Reachability Analyzer (config-time check)
aws ec2 create-network-insights-path \
    --source i-0abc123 --destination 10.10.0.10 \
    --protocol TCP --destination-port 445
aws ec2 start-network-insights-analysis --network-insights-path-id nip-...

Inside the instance via SSM Run Command (cheeky)

aws ssm send-command --instance-ids i-0abc123 \
  --document-name AWS-RunPowerShellScript \
  --parameters 'commands=[
    "Resolve-DnsName -Type SRV _ldap._tcp.dc._msdcs.corp.globalcorp.local",
    "Test-NetConnection 10.10.0.10 -Port 389",
    "Test-NetConnection 10.10.0.10 -Port 445",
    "Test-NetConnection 10.10.0.10 -Port 50000",
    "w32tm /query /status",
    "klist"
  ]'

What “good” looks like

Name                                    Type   TTL  Section
----                                    ----   ---  -------
_ldap._tcp.dc._msdcs.corp.globalcorp... SRV    600  Answer
  Priority : 0  Port : 389  Target : dc1-ny.corp...

ComputerName     : 10.10.0.10
RemoteAddress    : 10.10.0.10
RemotePort       : 389
TcpTestSucceeded : True

ComputerName     : 10.10.0.10
RemotePort       : 50000
TcpTestSucceeded : False   # <-- this is our smoking gun

Cross-check on the firewall side

# Network Firewall log query in CloudWatch Logs Insights
fields @timestamp, src, dst, dst_port, action
| filter dst = "10.10.0.10"
| filter src like /^10\.20\.10\./
| filter action = "DROP"
| stats count() by dst_port

Story

• The Network Firewall stateful rule group gc-corp-ad-allow permits the well-known AD ports: 53/88/389/636/3268/3269/445.
• It does not permit the RPC dynamic range 49152–65535 (Win Server 2008+ default; AD uses these for SAM/LSAD/NetLogon RPCs).
• The first phase of Add-Computer succeeds (DNS & LDAP work), but the secure channel setup needs a dynamic RPC port → FW drops → client retries → eventually surfaces as RPC was cancelled.
• This was masked because the FW was deployed by netsec team using an upstream “AD allow” rule template that pre-dates dynamic RPC.

Why this took a week to find

Immediate (PR + console — gc-network)

# 1. Patch the Network Firewall rule group to permit
#    the AD RPC dynamic range OR a constrained sub-range.
aws network-firewall describe-rule-group \
    --rule-group-name gc-corp-ad-allow --type STATEFUL --query 'RuleGroup' > rg.json

# 2. Append rule (Suricata syntax) and update.
# pass tcp $HOME_NET any -> $CORP_DC any (msg:"AD RPC dyn"; \
#   flow:to_server,established; sid:1000201; rev:1; \
#   dst_port:[49152:65535];)
aws network-firewall update-rule-group \
    --rule-group-name gc-corp-ad-allow --type STATEFUL \
    --rule-group file://rg.json --update-token <token>

Validate

aws ssm send-command --instance-ids i-0abc123 \
  --document-name AWS-RunPowerShellScript \
  --parameters 'commands=[
    "Test-NetConnection 10.10.0.10 -Port 50000",
    "Add-Computer -DomainName corp.globalcorp.local -Credential $cred -Restart"
  ]'

Correct fix — constrain the range, codify it

1. Ask the AD team to fix the dynamic port range to 50000–50099 on all DCs (registry) and publish it as a documented contract.
2. Update Network Firewall rule to permit only that constrained range.
3. Update the SG egress on Windows EC2 to allow the same subset to pl-corp-onprem.
4. Document in the AD SRE runbook + tag the FW rule group with Owner=netsec, References=AD-DC-Ports.

Branch + plan

# in gc-network repo
git checkout -b fix/nf-ad-rpc-dynamic
# edit modules/inspection-fw/rules/ad-allow.suricata
git diff --stat
terraform fmt -recursive
terraform validate
terraform plan -var-file=envs/us-east-1.tfvars \
   -target=module.inspection_fw.aws_networkfirewall_rule_group.ad_allow

Snippet of the change (HCL + Suricata)

resource "aws_networkfirewall_rule_group" "ad_allow" {
  capacity = 200
  name     = "gc-corp-ad-allow"
  type     = "STATEFUL"
  rule_group {
    rules_source {
      rules_string = file("${path.module}/rules/ad-allow.suricata")
    }
    rule_variables {
      ip_sets {
        key = "CORP_DC"
        ip_set { definition = ["10.10.0.10/32", "10.10.0.11/32"] }
      }
    }
  }
  tags = local.tags
}

ad-allow.suricata (added rule)

# existing well-known AD ports...
pass tcp $HOME_NET any -> $CORP_DC [53,88,389,636,3268,3269,445] \
  (msg:"AD well-known"; sid:1000101; rev:2;)

# NEW: AD RPC dynamic range (constrained to 50000-50099)
pass tcp $HOME_NET any -> $CORP_DC 50000:50099 \
  (msg:"AD RPC dyn constrained"; flow:to_server,established;
   sid:1000201; rev:1;)

PR + apply

1. Open PR; checkov + tflint green; CI posts plan to PR.
2. Reviewers: @netsec-leads, @ad-leads.
3. Merge → apply.yml assumes FedTerraformApply via OIDC, runs terraform apply.
4. Drift-check job nightly; if FW console-edited again, drift opens an issue.

Detective controls (CloudWatch & Config)

• CW alarm on Network Firewall metric DroppedPackets with dimension StatefulRuleGroup=gc-corp-ad-allow — non-zero in 5 min → PagerDuty.
• CW Logs Insights saved query: “AD RPC drops to corp DCs” bookmarked in dashboard gc-ad-health.
• Config rule: required-network-firewall-rule-group-tags — rule groups must carry References=AD-DC-Ports tag (…so they show up in this audit).
• EventBridge on SSMRunCommand success/failure for GC-JoinOnPremAD; failures trigger a Lambda that posts diagnostics to Slack #ad-domain-join.

Preventive controls

• Synthetic canary: a small Windows EC2 in shared-svcs runs a Test-NetConnection matrix to all corp DCs every 5 min, emits CW custom metric.
• Pull request CI: any change to aws_networkfirewall_rule_group requires @ad-leads review (CODEOWNERS).
• SCP exception watcher: if anyone tries to disable IMDSv2 enforcement, audit acct alerts.
• Runbook link on every NETLOGON 5719 alert that points to this scenario's slide deck.

What happened

• Terraform apply succeeds. aws ec2 run-instances returns an InstanceId.
• Instance shows pending for ~8 min then transitions to shutting-down → terminated.
• StateReason: Server.InternalError: Internal error on launch.

Business impact

• Blue/green deploy can't expand green fleet. Stuck behind quota of 1 deploy in flight.
• Repeats for 4/10 instances, randomly — not deterministic by AZ or instance type.

Constraints

CloudTrail look-up

aws cloudtrail lookup-events \
   --lookup-attributes AttributeKey=ResourceName,AttributeValue=i-0xx \
   --max-results 5 --query 'Events[].CloudTrailEvent' \
   | jq -r '.' | jq 'select(.errorCode!=null) | {errorCode,errorMessage}'

# 1. Pull the StateReason directly
aws ec2 describe-instances --instance-ids i-0xx \
  --query 'Reservations[].Instances[].{S:State.Name,R:StateReason}'

# 2. Pull instance-status (more granular)
aws ec2 describe-instance-status --instance-ids i-0xx \
  --include-all-instances

# 3. Inspect the snapshot encryption + KMS key
aws ec2 describe-snapshots --snapshot-ids snap-0xx \
  --query 'Snapshots[].{Enc:Encrypted,KMS:KmsKeyId,Owner:OwnerId}'

# 4. Check key policy in the source account
aws --profile gc-tools kms get-key-policy \
   --key-id alias/tools-ami --policy-name default | jq

# 5. List grants on the key (look for our role)
aws --profile gc-tools kms list-grants \
   --key-id alias/tools-ami \
   --query 'Grants[?contains(GranteePrincipal,`666666666666`)]'

# 6. Try the decrypt directly with an exec-role on a test instance
aws ssm send-command --instance-ids i-test \
  --document-name AWS-RunShellScript \
  --parameters 'commands=[
    "aws kms describe-key --key-id arn:aws:kms:us-east-1:121212121212:key/aaa..."
  ]'

Key policy patch (in gc-tools-cicd repo)

data "aws_iam_policy_document" "tools_ami_key" {
  statement {
    sid = "AllowSpokeAccountsToUseKey"
    actions = ["kms:Decrypt","kms:DescribeKey",
               "kms:ReEncrypt*","kms:GenerateDataKey*",
               "kms:CreateGrant"]
    principals { type="AWS"; identifiers=["arn:aws:iam::666666666666:root"] }
    resources = ["*"]
    condition {
      test     = "StringEquals"
      variable = "kms:ViaService"
      values   = ["ec2.us-east-1.amazonaws.com"]
    }
  }
}

Spoke side — ASG service role + KMS

resource "aws_iam_role_policy" "asg_kms" {
  role = "AWSServiceRoleForAutoScaling"
  policy = jsonencode({
    Version="2012-10-17",
    Statement=[{
      Effect="Allow",
      Action=["kms:CreateGrant","kms:Decrypt",
              "kms:ReEncrypt*","kms:GenerateDataKey*",
              "kms:DescribeKey"],
      Resource="arn:aws:kms:us-east-1:121212121212:key/aaa-tools-ami"
    }]
  })
}

Observed

• Instance boots; SSM Session is fine; but the orders-api service isn't installed.
• cloud-init-output.log is empty on Linux; EC2Launch log shows “UserData persist disabled” on Windows.
• Manually re-running the script works.

Impact

• Auto-built fleet has 30% no-op instances; canary fails to flip green.

Constraints

Quick path

1. Pull the rendered user-data from IMDS — if it's empty, the LT is the bug.
2. If it's present, check cloud-init status --long and journalctl -u cloud-final.
3. Windows: Get-Content C:\ProgramData\Amazon\EC2Launch\log\agent.log -Tail 200

Linux

# IMDSv2 token first
TOKEN=$(curl -s -X PUT http://169.254.169.254/latest/api/token \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 60")

# Rendered user-data
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
   http://169.254.169.254/latest/user-data | head -40

# cloud-init status + log
cloud-init status --long
sudo journalctl -u cloud-final --no-pager | tail -100

# Look for stale semaphores baked into AMI
ls -la /var/lib/cloud/sem/
ls -la /var/lib/cloud/instance/

Windows

# EC2Launch v2 task state
Get-Service AmazonSSMAgent
Get-Content "C:\ProgramData\Amazon\EC2Launch\log\agent.log" `
   -Tail 200

# Has UserData been marked “run-once”?
Test-Path "C:\ProgramData\Amazon\EC2Launch\state\.run-once"

# Re-arm UserData for next boot
& "C:\Program Files\Amazon\EC2Launch\EC2Launch.exe" reset --schedule

# Check the rendered user-data
Invoke-WebRequest -Headers @{"X-aws-ec2-metadata-token"=$tk} `
   -Uri "http://169.254.169.254/latest/user-data"

Packer provisioner (Linux)

# last provisioner before snapshot
provisioner "shell" {
  inline = [
    "sudo cloud-init clean --logs",
    "sudo rm -rf /var/lib/cloud/sem/* /var/lib/cloud/instance",
    "sudo rm -f /etc/machine-id && sudo touch /etc/machine-id",
    "sudo truncate -s 0 /etc/hostname",
    "sudo rm -rf /root/.ssh /home/ec2-user/.ssh"
  ]
}

Packer provisioner (Windows)

provisioner "powershell" {
  inline = [
    "& 'C:\\Program Files\\Amazon\\EC2Launch\\EC2Launch.exe' reset",
    "& 'C:\\Program Files\\Amazon\\EC2Launch\\EC2Launch.exe' sysprep --shutdown"
  ]
}

Terraform AMI promotion gate

# in tools-cicd: promotion job
resource "aws_ssm_parameter" "prod_ami_id" {
  name  = "/gc/prod/ami/orders-api"
  type  = "String"
  value = data.aws_ami.candidate.id
  lifecycle { precondition {
    condition     = data.aws_ami.candidate.tags["PackerCleanupRun"] == "true"
    error_message = "AMI must be tagged PackerCleanupRun=true."
  } }
}

Observed

• App calls aws sts get-caller-identity from inside the instance and gets old role FedAdmin — not the expected orders-api-task-role.
• Result: writes to S3 fail with AccessDenied; reads from a different bucket succeed.
• Started after the team replaced the instance profile via Terraform.

Constraints

# 1. What does the API say is associated?
aws ec2 describe-iam-instance-profile-associations \
  --filters Name=instance-id,Values=i-0xx \
  --query 'IamInstanceProfileAssociations[].{S:State,Arn:IamInstanceProfile.Arn}'

# 2. What does IMDS say?
TK=$(curl -s -X PUT http://169.254.169.254/latest/api/token \
   -H "X-aws-ec2-metadata-token-ttl-seconds: 60")
curl -s -H "X-aws-ec2-metadata-token: $TK" \
  http://169.254.169.254/latest/meta-data/iam/security-credentials/

# 3. SDK says what?
aws sts get-caller-identity

# 4. Force-rotate by re-associating profile
ASSOC=$(aws ec2 describe-iam-instance-profile-associations \
  --filters Name=instance-id,Values=i-0xx \
  --query 'IamInstanceProfileAssociations[0].AssociationId' --output text)

aws ec2 replace-iam-instance-profile-association \
  --association-id $ASSOC \
  --iam-instance-profile Name=ip-orders-api-v2

# 5. Restart the app or SSM agent
sudo systemctl restart orders-api
sudo systemctl restart amazon-ssm-agent

# 6. Confirm
sleep 30 && aws sts get-caller-identity

resource "aws_launch_template" "orders_api" {
  name_prefix = "orders-api-"
  iam_instance_profile { name = aws_iam_instance_profile.orders_v2.name }
  user_data = base64encode(templatefile("ud.sh.tftpl",{}))
  metadata_options {
    http_tokens                 = "required"
    http_put_response_hop_limit = 2
  }
  tag_specifications { resource_type="instance"; tags=local.tags }
}

resource "aws_autoscaling_group" "orders_api" {
  ...
  launch_template { id=aws_launch_template.orders_api.id; version="$Latest" }
  instance_refresh {
    strategy = "Rolling"
    preferences { min_healthy_percentage = 90 }
    triggers = ["launch_template"]   # <-- key
  }
}

Why triggers = ["launch_template"] matters

• By default ASG instance-refresh only triggers on AMI changes.
• Adding "launch_template" means any LT version bump (incl. instance profile) auto-rolls the fleet.
• Combined with min_healthy_percentage = 90, the rollout is safe.

Observed

• ASG can't scale: An error occurred (InvalidAMIID.NotFound) when calling the RunInstances operation.
• Same AMI ID worked yesterday. Console UI shows AMI in source acct but not in spoke.
• Recent change: tools-cicd team rotated AMI bake pipeline; old AMIs deregistered.

Constraints

# 1. Does the AMI exist for the source owner?
aws ec2 describe-images --owners 121212121212 \
  --image-ids ami-0abc123def456 \
  --query 'Images[].{ID:ImageId,State:State,Name:Name}' \
  --output table

# 2. Was it deregistered?
aws --profile gc-tools cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=ami-0abc123def456 \
  --max-results 5 --query 'Events[].{T:EventTime,N:EventName,U:Username}'

# 3. Resolve via SSM parameter (what should happen)
aws ssm get-parameter --name /gc/prod/ami/orders-api \
  --query Parameter.Value --output text

# 4. Check launch permission
aws --profile gc-tools ec2 describe-image-attribute \
  --image-id ami-0abc123def456 --attribute launchPermission

# 5. Check snapshot share for encrypted AMI
aws --profile gc-tools ec2 describe-snapshot-attribute \
  --snapshot-id snap-0xx --attribute createVolumePermission

data "aws_ssm_parameter" "orders_ami" {
  name = "/gc/prod/ami/orders-api"
}

resource "aws_launch_template" "orders_api" {
  image_id = data.aws_ssm_parameter.orders_ami.value
  ...
}

# in tools-cicd: write parameter on every promotion
resource "aws_ssm_parameter" "prod_ami" {
  name      = "/gc/prod/ami/orders-api"
  type      = "String"
  data_type = "aws:ec2:image"
  value     = aws_ami_copy.candidate.id
  overwrite = true
}

Cross-account read of the parameter

• Use RAM to share the parameter to spoke accounts (or cross-account permissions on the SSM parameter for SSM Advanced — not available for Standard).
• Spokes reference via shared parameter; Terraform provider supports this with a cross-account aws alias.

Observed

• Bastion in DMZ subnet launches but only has private IP. No public IP assigned.
• Vendor partner can't SSH from internet via the EIP that was supposed to be auto-attached.

Constraints

# 1. Subnet flag
aws ec2 describe-subnets --subnet-ids subnet-dmz-use1a \
  --query 'Subnets[].{Auto:MapPublicIpOnLaunch,IPs:AvailableIpAddressCount}'

# 2. Instance state
aws ec2 describe-instances --instance-ids i-0xx \
  --query 'Reservations[].Instances[].{Pub:PublicIpAddress,Priv:PrivateIpAddress}'

# 3. EIP available?
aws ec2 describe-addresses \
  --query 'Addresses[?AssociationId==`null`].PublicIp'

# 4. Did user-data fail silently?
sudo cat /var/log/cloud-init-output.log | grep -i associate

# 5. IAM perm proof
aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::666...:role/bastion-role \
  --action-names ec2:AssociateAddress \
  --resource-arns arn:aws:ec2:us-east-1:666...:elastic-ip/eipalloc-0xx

# 6. Manually associate (fix attempt)
aws ec2 associate-address \
  --instance-id i-0xx --allocation-id eipalloc-0xx

resource "aws_iam_role_policy" "bastion_eip" {
  role = aws_iam_role.bastion.id
  policy = jsonencode({
    Version="2012-10-17",
    Statement=[{
      Effect="Allow",
      Action=["ec2:AssociateAddress","ec2:DisassociateAddress",
              "ec2:DescribeAddresses"],
      Resource="*",
      Condition={
        StringEquals={"aws:ResourceTag/Role"="bastion"}
      }
    }]
  })
}

# user-data hardening
provisioner "file" {
  content = <<-EOT
    #!/usr/bin/env bash
    set -euo pipefail
    TOKEN=$(curl -s -X PUT \
      http://169.254.169.254/latest/api/token \
      -H "X-aws-ec2-metadata-token-ttl-seconds: 60")
    INST=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
      http://169.254.169.254/latest/meta-data/instance-id)
    aws ec2 associate-address \
      --instance-id $INST --allocation-id ${var.eip_alloc_id}
  EOT
}

Observed

• App was patched: stop → start. Came back with new private IP (10.20.10.99 → 10.20.10.121).
• Microservice orders-api.gcaws.internal still resolves to old IP for ~10 minutes.
• Downstream services 5xx with connection refused.

Constraints

# 1. Current PHZ record
aws route53 list-resource-record-sets \
  --hosted-zone-id Z0XXX \
  --query "ResourceRecordSets[?Name=='orders-api.gcaws.internal.']"

# 2. Current instance IP
aws ec2 describe-instances --instance-ids i-0xx \
  --query 'Reservations[].Instances[].PrivateIpAddress'

# 3. From inside the host
getent hosts orders-api.gcaws.internal
dig +short orders-api.gcaws.internal

# 4. Update the record (immediate fix)
aws route53 change-resource-record-sets \
  --hosted-zone-id Z0XXX --change-batch file://upsert.json

# 5. Force JVM to re-resolve (cheeky)
sudo systemctl restart orders-api    # cleanest
# or via JMX:
jcmd <pid> VM.system_properties | grep -i ttl

# 6. Confirm
ss -tnp | grep orders-api    # new sockets to right IP

Path A: front with ALB / NLB

resource "aws_lb" "orders" { internal=true; load_balancer_type="application"; ... }
resource "aws_lb_target_group" "orders" { ... }
resource "aws_route53_record" "orders" {
  zone_id = data.aws_route53_zone.gcaws.zone_id
  name    = "orders-api"
  type    = "A"
  alias { name = aws_lb.orders.dns_name; zone_id = aws_lb.orders.zone_id; evaluate_target_health = true }
}

Path B: auto-update via EventBridge + Lambda

resource "aws_cloudwatch_event_rule" "ec2_state" {
  event_pattern = jsonencode({
    source      = ["aws.ec2"],
    detail-type = ["EC2 Instance State-change Notification"],
    detail      = { state=["running"] }
  })
}
resource "aws_lambda_function" "phz_updater" { ... }
# Lambda reads instance tag DnsName, upserts PHZ record

Observed

• ASG can't scale: We currently do not have sufficient c6i.4xlarge capacity in the AZ you requested (us-east-1a).
• ASG configured with single AZ + single instance type (legacy).
• Happens at 9am traffic peak on weekdays.

Constraints

# 1. ASG scaling activity
aws autoscaling describe-scaling-activities \
  --auto-scaling-group-name orders-asg --max-records 10 \
  --query 'Activities[].{T:StartTime,S:StatusCode,M:StatusMessage}'

# 2. Quota
aws service-quotas get-service-quota \
  --service-code ec2 --quota-code L-1216C47A

# 3. Subnet IPs
aws ec2 describe-subnets --subnet-ids subnet-priv-use1a \
  --query 'Subnets[].AvailableIpAddressCount'

# 4. Probe other AZs (dry-run)
for az in us-east-1a us-east-1b us-east-1c; do
  echo $az
  aws ec2 run-instances --dry-run --instance-type c6i.4xlarge \
    --image-id ami-0xx --subnet-id $(subnet_for $az) \
    --query Errors --output text 2>&1 | head -2
done

# 5. ODCR check
aws ec2 describe-capacity-reservations \
  --filters Name=state,Values=active \
  --query 'CapacityReservations[].{T:InstanceType,AZ:AvailabilityZone,Avail:AvailableInstanceCount}'

resource "aws_autoscaling_group" "orders" {
  vpc_zone_identifier = local.private_subnets_3az
  min_size=4; desired=8; max_size=40
  mixed_instances_policy {
    launch_template { launch_template_specification {
      launch_template_id = aws_launch_template.orders.id; version="$Latest"
    } }
    instances_distribution {
      on_demand_base_capacity                  = 4
      on_demand_percentage_above_base_capacity = 50
      spot_allocation_strategy                 = "capacity-optimized"
    }
    override { instance_type = "c6i.4xlarge" }
    override { instance_type = "c6a.4xlarge" }
    override { instance_type = "c5.4xlarge" }
    override { instance_type = "m6i.4xlarge" }
  }
  capacity_reservation_specification {
    capacity_reservation_preference = "open"
  }
}

resource "aws_ec2_capacity_reservation" "orders_floor" {
  instance_type = "c6i.4xlarge"
  instance_platform = "Linux/UNIX"
  availability_zone = "us-east-1a"
  instance_count    = 4
  end_date_type     = "unlimited"
  instance_match_criteria = "open"
  tags = local.tags
}

Observed

• Terraform apply fails with UnauthorizedOperation: with an explicit deny in a service control policy.
• Console launch by FedAdmin also fails — same SCP applies.
• Other instances in the account work fine — only new launches fail.

Constraints

# 1. Show the failing API call from CloudTrail
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=RunInstances \
  --max-results 1 --query 'Events[].CloudTrailEvent' \
  | jq '.[0] | fromjson | {errorCode, errorMessage}'

# 2. Pull SCPs in scope
aws --profile gc-mgmt organizations list-policies-for-target \
  --target-id ou-xxx --filter SERVICE_CONTROL_POLICY

# 3. Validate Terraform tag plan
terraform show -json tfplan | jq '.. | objects | select(.tag_specifications) | .tag_specifications'

# 4. Test directly
aws ec2 run-instances --image-id ami-0xx --instance-type t3.micro \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=CostCenter,Value=ENG-100},{Key=Owner,Value=alice},{Key=Env,Value=dev}]' \
  'ResourceType=volume,Tags=[{Key=CostCenter,Value=ENG-100}]' \
  --dry-run

resource "aws_launch_template" "orders" {
  ...
  tag_specifications { resource_type="instance"; tags=local.tags }
  tag_specifications { resource_type="volume";   tags=local.tags }
  tag_specifications { resource_type="network-interface"; tags=local.tags }
}

provider "aws" {
  region = "us-east-1"
  default_tags { tags = local.tags }
}

locals {
  tags = {
    CostCenter = "ENG-100"
    Owner      = "orders-team"
    Env        = "prod"
  }
}

Lint as a guardrail

# tflint plugin: aws-ruleset
plugin "aws" { enabled=true; version="0.30.0"; source="github.com/terraform-linters/tflint-ruleset-aws" }
rule "aws_resource_missing_tags" {
  enabled = true
  tags    = ["CostCenter", "Owner", "Env"]
}

Observed

• Legacy launch template still has HttpTokens=optional (allows IMDSv1).
• SCP denies RunInstances when ec2:MetadataHttpTokens != required.
• Old AMI also has SDK code that uses IMDSv1 only.

Constraints

• SCP applied to all Workloads OU accounts.
• Org-wide policy: IMDSv2 required from 2024-09.
• App written in 2017 with old AWS SDK.

# 1. LT current setting
aws ec2 describe-launch-template-versions \
  --launch-template-id lt-0xx --versions '$Latest' \
  --query 'LaunchTemplateVersions[].LaunchTemplateData.MetadataOptions'

# 2. Per-instance audit
aws ec2 describe-instances --instance-ids i-0xx \
  --query 'Reservations[].Instances[].MetadataOptions'

# 3. Find IMDSv1 callers across fleet
aws cloudwatch get-metric-statistics --namespace AWS/EC2 \
  --metric-name MetadataNoToken --dimensions Name=InstanceId,Value=i-0xx \
  --start-time -1h --end-time now --period 60 --statistics Sum

# 4. Update LT (new version)
aws ec2 create-launch-template-version \
  --launch-template-id lt-0xx \
  --source-version '$Latest' \
  --launch-template-data '{
    "MetadataOptions":{
      "HttpTokens":"required",
      "HttpEndpoint":"enabled",
      "HttpPutResponseHopLimit":2
    }
  }'

# 5. Live-modify existing instances
aws ec2 modify-instance-metadata-options --instance-id i-0xx \
  --http-tokens required --http-put-response-hop-limit 2

resource "aws_launch_template" "orders" {
  metadata_options {
    http_endpoint               = "enabled"
    http_tokens                 = "required"   # IMDSv2
    http_put_response_hop_limit = 2
    instance_metadata_tags      = "enabled"
  }
}

Org-wide checkov rule

# .checkov.yml
check:
  - CKV_AWS_79  # EC2 should require IMDSv2
  - CKV_AWS_341 # LT hop_limit <= 2

Observed

• DR drill: restore RDS not feasible, instead restore an EBS snapshot to attach to a new instance.
• aws ec2 attach-volume fails: InvalidVolume.ZoneMismatch or just times out.
• Sometimes AccessDenied on KMS for the volume created in another region.

Constraints

• Snapshot owned by gc-prod-data (us-east-1).
• DR region: eu-west-1.
• Volume must attach to instance in eu-west-1c.
• Encrypted with regional KMS CMK (us-east-1 only).

# 1. Volume + instance AZ
aws --region eu-west-1 ec2 describe-volumes \
  --volume-ids vol-0xx --query 'Volumes[].{AZ:AvailabilityZone,KMS:KmsKeyId}'
aws --region eu-west-1 ec2 describe-instances \
  --instance-ids i-0xx --query 'Reservations[].Instances[].Placement.AvailabilityZone'

# 2. Re-create volume in correct AZ
aws --region eu-west-1 ec2 create-volume \
  --snapshot-id snap-dst --availability-zone eu-west-1c \
  --volume-type gp3 --encrypted --kms-key-id alias/eu-data

# 3. Snapshot progress
aws --region eu-west-1 ec2 describe-snapshots \
  --snapshot-ids snap-dst --query 'Snapshots[].{P:Progress,S:State,K:KmsKeyId}'

# 4. Cross-account share check
aws --profile gc-prod-data ec2 describe-snapshot-attribute \
  --snapshot-id snap-src --attribute createVolumePermission

# 5. Attach
aws --region eu-west-1 ec2 attach-volume \
  --volume-id vol-new --instance-id i-0xx --device /dev/sdf

# SSM Automation document (Terraform-managed)
resource "aws_ssm_document" "dr_restore_volume" {
  name          = "GC-DR-RestoreEBS"
  document_type = "Automation"
  content       = file("docs/dr-restore-ebs.yaml")
}

# dr-restore-ebs.yaml (excerpt)
parameters:
  SnapshotId: { type: String }
  TargetAz:   { type: String, default: "eu-west-1c" }
  KmsKeyId:   { type: String, default: "alias/eu-data" }
mainSteps:
  - name: copy
    action: aws:executeAwsApi
    inputs: { Service: ec2, Api: CopySnapshot, ... }
  - name: wait
    action: aws:waitForAwsResourceProperty
  - name: create_volume
    action: aws:executeAwsApi
    inputs: { Api: CreateVolume,
      AvailabilityZone: "{{ TargetAz }}" }

# DLM lifecycle policy
resource "aws_dlm_lifecycle_policy" "orders_data" {
  description = "orders-data daily snap + DR copy"
  state       = "ENABLED"
  policy_details {
    schedule {
      cross_region_copy_rule {
        target = "eu-west-1"
        encrypted = true
        cmk_arn   = aws_kms_alias.eu_data.target_key_arn
        retain_rule { interval = 30; interval_unit = "DAYS" }
      }
    }
  }
}

Observed

• Java app crashes with OOM 30s after instance running.
• EC2 type c6i.xlarge (4 vCPU, 8 GiB).
• Root volume 8 GiB — full after agent installs.
• No swap configured.

Constraints

• Java -Xmx not set; defaults to 25% of memory.
• SCP forbids ebs-optimized=false.
• Org default ebs-encryption-by-default=true.

# 1. Disk + memory
df -h /
free -m
swapon --show

# 2. JVM flags
sudo -u app jps -l
sudo -u app jcmd <pid> VM.flags | grep -E 'MaxHeap|MinHeap|UseG1'
sudo -u app jcmd <pid> VM.system_properties | grep mx

# 3. EBS burst (gp2)
aws cloudwatch get-metric-statistics --namespace AWS/EBS \
  --metric-name BurstBalance --dimensions Name=VolumeId,Value=vol-0xx \
  --start-time -1h --end-time now --period 60 --statistics Minimum

# 4. Linux OOM-kill
sudo dmesg | grep -i 'killed process'
sudo journalctl -k --since "1 hour ago" | grep -i oom

resource "aws_launch_template" "orders" {
  ebs_optimized = true
  block_device_mappings {
    device_name = "/dev/xvda"
    ebs { volume_size=30; volume_type="gp3"; iops=3000; throughput=125; encrypted=true }
  }
  block_device_mappings {
    device_name = "/dev/sdb"
    ebs { volume_size=20; volume_type="gp3"; encrypted=true }   # /var/log
  }
  user_data = base64encode(file("ud.sh"))   # mounts + JVM tuning
}

# ud.sh excerpt
mkfs.xfs /dev/nvme1n1
mount /dev/nvme1n1 /var/log
echo "/dev/nvme1n1 /var/log xfs defaults,nofail 0 2" >> /etc/fstab

# swap
fallocate -l 2G /swapfile && chmod 600 /swapfile
mkswap /swapfile && swapon /swapfile

# JVM
echo 'JAVA_OPTS="-Xms4g -Xmx5g -XX:+UseG1GC -XX:+ExitOnOutOfMemoryError"' \
  >> /etc/orders-api.env

Observed

• ALB target group sees ~30s of 5xx during spot interruption.
• Instance vanishes with no graceful drain.
• Customer error rate breached SLO.

Constraints

• Spot fleet 60% of ASG.
• 2-min interruption notice via IMDS.
• ALB deregistration delay default 300s.

# 1. Lifecycle hooks on the ASG
aws autoscaling describe-lifecycle-hooks --auto-scaling-group-name orders-asg

# 2. ALB TG drain
aws elbv2 describe-target-group-attributes --target-group-arn arn:...:targetgroup/orders/...
# expect deregistration_delay.timeout_seconds <= 60 for most apps

# 3. Spot interruption history
aws ec2 describe-spot-instance-requests --filters Name=state,Values=closed

# 4. Listen for interruption from inside instance
TOKEN=$(curl -s -X PUT http://169.254.169.254/latest/api/token \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 60")
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/spot/instance-action

# 5. Run NTH locally
sudo systemctl status aws-node-termination-handler

resource "aws_autoscaling_lifecycle_hook" "terminate" {
  name                   = "orders-terminate"
  autoscaling_group_name = aws_autoscaling_group.orders.name
  lifecycle_transition   = "autoscaling:EC2_INSTANCE_TERMINATING"
  default_result         = "CONTINUE"
  heartbeat_timeout      = 90
  notification_target_arn = aws_sns_topic.lifecycle.arn
  role_arn               = aws_iam_role.lifecycle.arn
}
resource "aws_lb_target_group" "orders" {
  ...
  deregistration_delay = 30
}

# NTH on each instance (DaemonSet for k8s, systemd for plain EC2)
provisioner "file" {
  destination = "/etc/systemd/system/aws-node-termination-handler.service"
  content     = file("nth.service")
}

Observed

• Old AMI behavior in fresh instances even after promotion.
• ASG $Default points to v3; v5 is latest.
• Console shows v5 marked Default but actual launches use v3.

Constraints

• ASG launch_template.version = "$Default" in Terraform.
• Manual set-default-version happened in console.
• Drift between code and live state.

# 1. LT versions
aws ec2 describe-launch-template-versions \
  --launch-template-id lt-0xx \
  --query 'LaunchTemplateVersions[].{V:VersionNumber,D:DefaultVersion,I:LaunchTemplateData.ImageId}'

# 2. ASG launch config
aws autoscaling describe-auto-scaling-groups \
  --auto-scaling-group-names orders-asg \
  --query 'AutoScalingGroups[].LaunchTemplate'

# 3. Force ASG to v5 explicit
aws autoscaling update-auto-scaling-group \
  --auto-scaling-group-name orders-asg \
  --launch-template LaunchTemplateId=lt-0xx,Version=5

# 4. Trigger refresh
aws autoscaling start-instance-refresh \
  --auto-scaling-group-name orders-asg \
  --preferences MinHealthyPercentage=90,InstanceWarmup=120

resource "aws_autoscaling_group" "orders" {
  launch_template {
    id      = aws_launch_template.orders.id
    version = aws_launch_template.orders.latest_version  # pin explicit
  }
  instance_refresh {
    strategy = "Rolling"
    triggers = ["launch_template"]
    preferences { min_healthy_percentage = 90; instance_warmup = 120 }
  }
}

CI drift check

# nightly cron in CI
terraform plan -refresh-only -detailed-exitcode
# exit 2 = drift; raise issue

Observed

• User-data calls aws secretsmanager get-secret-value.
• Random failures: ~10% of launches return UnrecognizedClientException: The security token included in the request is invalid.
• Retrying the same instance 30s later succeeds.

Constraints

• SCP requires IMDSv2.
• Role attached at LT.
• Instance profile propagation eventually consistent (1–2 sec usually).

# 1. Confirm the race
sudo grep -E 'UnrecognizedClient|InvalidSignatureException' /var/log/cloud-init-output.log

# 2. Test from instance after boot
for i in 1 2 3; do
  aws sts get-caller-identity || echo retry; sleep 1
done

# 3. Wait pattern in user-data
until aws sts get-caller-identity >/dev/null 2>&1; do sleep 2; done
SECRET=$(aws secretsmanager get-secret-value --secret-id orders-prod \
  --query SecretString --output text)

# ud.sh template
#!/usr/bin/env bash
set -euo pipefail
TOK=$(curl -s -X PUT http://169.254.169.254/latest/api/token \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 60")
for i in {1..30}; do
  if aws sts get-caller-identity >/dev/null 2>&1; then break; fi
  sleep 2
done
SECRET=$(aws secretsmanager get-secret-value --secret-id ${secret_id} \
  --query SecretString --output text)

Or move secret pull to SSM Run Command

# EventBridge on EC2 running → SSM doc → pull secret

Observed

• Some instances appear in console as “non-compliant” under Tag Policy view.
• Tag value is Production, policy wants prod.
• Resources still launch — tag policies don't deny by default.

Constraints

• Org tag policy enforces Env values: [prod, stg, dev].
• Old IaC writes Production.
• Drift surfaces in Tag Policy compliance, not in CloudTrail errorCode.

# 1. Effective tag policy at account level
aws --profile gc-mgmt organizations describe-effective-policy \
  --policy-type TAG_POLICY --target-id 666666666666

# 2. Find non-compliant resources
aws resourcegroupstaggingapi get-resources \
  --tag-filters Key=Env,Values=Production
# compare to allowed: prod / stg / dev

# 3. Bulk re-tag
aws resourcegroupstaggingapi tag-resources \
  --resource-arn-list arn:aws:ec2:...:instance/i-0xx \
  --tags Env=prod

# 4. Tag policy compliance summary
aws --profile gc-audit config describe-compliance-by-config-rule \
  --config-rule-names required-tags

locals {
  tags = merge({
    Env        = "prod"
    CostCenter = "ENG-100"
    Owner      = "orders-team"
  }, var.extra_tags)
}

provider "aws" {
  default_tags { tags = local.tags }
}

# tag policy strict mode
{
  "tags": {
    "Env": {
      "tag_key": { "@@assign": "Env" },
      "tag_value": { "@@assign": ["prod", "stg", "dev"] },
      "enforced_for": { "@@assign": ["ec2:instance", "rds:db"] }
    }
  }
}

Observed

• ASG can't scale: InsufficientFreeAddressesInSubnet.
• Subnet is /27 with 3 IPs free out of 32.
• Many ENIs are detached but not released.

Constraints

• Subnet originally sized for 4 instances.
• EKS pods consume secondary IPs aggressively.
• Lambda VPC ENIs hold onto IPs.

# 1. IP availability
aws ec2 describe-subnets --filters Name=vpc-id,Values=vpc-0xx \
  --query 'Subnets[].{ID:SubnetId,AZ:AvailabilityZone,Free:AvailableIpAddressCount,CIDR:CidrBlock}' \
  --output table

# 2. Who holds the ENIs?
aws ec2 describe-network-interfaces \
  --filters Name=subnet-id,Values=subnet-priv-use1a \
  --query 'NetworkInterfaces[].{S:Status,D:Description,O:Attachment.InstanceOwnerId}' \
  --output table

# 3. Free orphaned ENIs
aws ec2 delete-network-interface --network-interface-id eni-0xx

# 4. Add secondary CIDR + new subnet
aws ec2 associate-vpc-cidr-block --vpc-id vpc-0xx \
  --cidr-block 100.64.0.0/16
aws ec2 create-subnet --vpc-id vpc-0xx \
  --cidr-block 100.64.10.0/22 --availability-zone us-east-1a

# 5. Switch EKS to prefix delegation (more IPs/instance)
kubectl set env -n kube-system ds aws-node ENABLE_PREFIX_DELEGATION=true

resource "aws_vpc_ipv4_cidr_block_association" "secondary" {
  vpc_id     = aws_vpc.main.id
  cidr_block = "100.64.0.0/16"
}
resource "aws_subnet" "private_carrier" {
  count             = 3
  vpc_id            = aws_vpc.main.id
  cidr_block        = cidrsubnet("100.64.0.0/16", 6, count.index)
  availability_zone = local.azs[count.index]
  tags = merge(local.tags, { Tier="private-carrier", KubernetesCarrier="true" })
}

# enable prefix delegation in EKS
resource "aws_eks_addon" "vpc_cni" {
  cluster_name = aws_eks_cluster.main.name
  addon_name   = "vpc-cni"
  configuration_values = jsonencode({
    env = { ENABLE_PREFIX_DELEGATION = "true" }
  })
}

Observed

• Instance refresh stops at 30%; one instance won't terminate.
• API: OperationNotPermitted: The instance has termination protection.
• Someone manually enabled it during an incident, never removed.

Constraints

• ASG auto-scaling protections also apply.
• Two flags: instance-level DisableApiTermination + ASG protected_from_scale_in.

# 1. Check both flags
aws ec2 describe-instance-attribute --instance-id i-0xx \
  --attribute disableApiTermination

aws autoscaling describe-auto-scaling-instances \
  --instance-ids i-0xx \
  --query 'AutoScalingInstances[].ProtectedFromScaleIn'

# 2. Disable EC2 termination protection
aws ec2 modify-instance-attribute --instance-id i-0xx \
  --no-disable-api-termination

# 3. Disable ASG scale-in protect
aws autoscaling set-instance-protection \
  --instance-ids i-0xx \
  --auto-scaling-group-name orders-asg \
  --no-protected-from-scale-in

# 4. Resume refresh
aws autoscaling resume-processes \
  --auto-scaling-group-name orders-asg

# Tag-driven cleanup
resource "aws_lambda_function" "ops_hold_cleanup" {
  function_name = "gc-ops-hold-cleanup"
  ...
}
resource "aws_cloudwatch_event_rule" "daily" {
  schedule_expression = "cron(0 8 * * ? *)"
}

# Lambda body (excerpt)
for inst in ec2.describe_instances(Filters=[{Name='tag:OpsHold',Values=['true']}]):
    age = now - inst.tags.get('OpsHoldSet')
    if age > 24h:
        ec2.modify_instance_attribute(InstanceId=inst.id, DisableApiTermination=False)
        ec2.modify_instance_attribute(InstanceId=inst.id, Tags={'OpsHold':'cleared'})
        slack('cleared OpsHold on '+inst.id)

Observed

• Instance went unhealthy due to underlying host issue. CW alarm StatusCheckFailed_System never fired the recovery action.
• Alarm in INSUFFICIENT_DATA for the last 6h.

Constraints

• Alarm action: arn:aws:automate:us-east-1:ec2:recover.
• EC2 type c6i.4xlarge supports recovery.
• Metric data missing: instance not reporting.

# 1. Alarm definition
aws cloudwatch describe-alarms \
  --alarm-names orders-recover-i-0xx \
  --query 'MetricAlarms[].{T:TreatMissingData,A:AlarmActions,P:DatapointsToAlarm,E:EvaluationPeriods}'

# 2. Last 6h metric data
aws cloudwatch get-metric-statistics --namespace AWS/EC2 \
  --metric-name StatusCheckFailed_System \
  --dimensions Name=InstanceId,Value=i-0xx \
  --start-time -6h --end-time now --period 60 --statistics Maximum

# 3. Fix the alarm
aws cloudwatch put-metric-alarm \
  --alarm-name orders-recover-i-0xx \
  --metric-name StatusCheckFailed_System --namespace AWS/EC2 \
  --dimensions Name=InstanceId,Value=i-0xx \
  --statistic Maximum --period 60 --threshold 0 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 5 --datapoints-to-alarm 3 \
  --treat-missing-data breaching \
  --alarm-actions arn:aws:automate:us-east-1:ec2:recover

resource "aws_cloudwatch_metric_alarm" "recover" {
  for_each = toset(var.instance_ids)
  alarm_name          = "recover-${each.value}"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 5
  datapoints_to_alarm = 3
  metric_name         = "StatusCheckFailed_System"
  namespace           = "AWS/EC2"
  period              = 60
  statistic           = "Maximum"
  threshold           = 0
  treat_missing_data  = "breaching"
  alarm_actions       = ["arn:aws:automate:us-east-1:ec2:recover"]
  dimensions = { InstanceId = each.value }
}

Even simpler

resource "aws_instance" "x" {
  maintenance_options { auto_recovery = "default" }
}

Observed

• SSM Console shows 0 managed instances for the new fleet.
• aws ssm describe-instance-information returns empty.
• SSM Session Manager fails with Target not found.

Constraints

• SSM agent installed on AMI — should auto-start.
• Instance has IAM role with AmazonSSMManagedInstanceCore.
• VPC has VPC endpoints for ssm/ssmmessages/ec2messages.

# 1. From instance (via console direct connect or get-system-log)
sudo systemctl status amazon-ssm-agent
sudo journalctl -u amazon-ssm-agent --no-pager | tail -50
sudo cat /var/log/amazon/ssm/amazon-ssm-agent.log | tail -100

# 2. Reach the endpoint
getent hosts ssm.us-east-1.amazonaws.com
curl -v https://ssm.us-east-1.amazonaws.com 2>&1 | head -10

# 3. From console
aws ssm describe-instance-information \
  --filters Key=InstanceIds,Values=i-0xx
# empty -> agent never registered

# 4. Inspect VPCe SG
aws ec2 describe-vpc-endpoints --vpc-endpoint-ids vpce-0xx \
  --query 'VpcEndpoints[].{S:Groups,P:PrivateDnsEnabled,SG:Groups}'

resource "aws_vpc_endpoint" "ssm" {
  vpc_id              = aws_vpc.main.id
  service_name        = "com.amazonaws.us-east-1.ssm"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = local.private_subnets
  security_group_ids  = [aws_security_group.vpce.id]
  private_dns_enabled = true   # <-- must be true
  tags                = local.tags
}
# repeat for ssmmessages, ec2messages

# SG for vpce
resource "aws_security_group_rule" "vpce_ingress" {
  type              = "ingress"
  from_port         = 443
  to_port           = 443
  protocol          = "tcp"
  source_security_group_id = aws_security_group.workload.id
  security_group_id = aws_security_group.vpce.id
}

Observed

• App stops responding; CW alarm fires reboot action.
• ~30s later, ASG ELB health check marks instance unhealthy and terminates.
• Both happen — instance reboots and gets killed.
• Net effect: 60s outage instead of 10s reboot.

Constraints

• ASG health_check_type=ELB, grace 60s.
• CW alarm reboot action on app metric.
• No coordination between actions.