# Terraform rebuilds a console-built AWS account from prod — read-only spec, parity diff, GitHub OIDC

> Codifying a console-built AWS data layer in Terraform from a read-only capture of prod, verified by a config diff and run through GitHub OIDC.

- Source: https://oharu121.com/blog/terraform-rebuild-console-aws-account-prod-spec-parity-oidc/
- Published: 2026-10-11T10:38:54+09:00
- Tags: Terraform, AWS, GitHub Actions

---
## Introduction

Every resource in a production AWS data layer I work on had been built by hand in the console: S3 buckets, Glue tables, an OpenSearch Serverless collection, two Bedrock knowledge bases, a Lambda, and an AgentCore gateway and code interpreter. Nothing described any of it, so its configuration was hard to copy into another environment.

Then a test account was added, for testing new features before they reach production. It already ran the main application's stack, and its data layer was empty.

I used it to write the data layer as **a Terraform module based on a read-only capture of prod's configuration**, applied the module there, and diffed the two accounts until **no unexplained difference was left**.

The test account now rebuilds from the repository, which is the only place the data layer is defined. Every pull request gets a `terraform plan` from GitHub Actions through OIDC, and merging to `main` applies it to the test account.

## Why Terraform and not CDK

**Adopting existing resources decided it.** The repository is Python, and CDK can be written in Python too, so the language would have matched. Terraform was ahead on import and on blast radius:

1. **Import.** Both tools can bring a Bedrock knowledge base under management, and the difference is who writes the definition. `terraform plan -generate-config-out` writes the HCL from the live resource, so the next `plan` shows what changed in the configuration. `cdk import` needs a hand-written construct that exactly matches the live resource, and `cdk import` does not check that its properties are correct and complete.

*Figure — ImportPaths: Both paths end with the knowledge base under management. Terraform writes the definition from the live resource and the next plan checks it; with CDK the construct is written by hand, and no step checks it.*

> **HEADS UP**
>
> **How HCL and `plan` compare.** HCL (HashiCorp Configuration Language) is Terraform's configuration language. `plan` reads the real resources, compares them with the HCL, and lists what it would create, change or delete; `apply` carries out that plan and writes the result to state. A difference between the generated HCL and the live resource shows up in the next `plan`.

2. **Blast radius.** A CloudFormation stack that adopts resources ties them together, so one failed update rolls the whole stack back. Terraform applies resource by resource, so whatever succeeded stays and is recorded in state.

*Figure — BlastRadius: One update changes A, B and C, and B fails. CloudFormation undoes the whole update; Terraform keeps A in state and leaves only what depends on B undone.*

CDK had one more advantage besides the language: state. CloudFormation keeps each stack's state inside the service, and AWS handles its locking, history and rollback. The bucket CDK does create, through the `CDKToolkit` bootstrap stack both accounts already had, only stages assets.

Terraform's state is a file in an S3 bucket in the test account, and that bucket needs a manual bootstrap. The bucket is itself created by Terraform, so until it exists the state has nowhere to go. The first `apply` keeps state on local disk, and once the bucket exists, `terraform init -migrate-state` copies that local state file into it.

The bootstrap happens once; the ownership does not. The state file can hold resource attributes in plain text, so the bucket needs the access control and versioning of any sensitive bucket, and a mismatch between state and the real resources is fixed with `terraform state` commands.

*Figure — StateOwnership: CDK's state lives inside CloudFormation and AWS looks after it. Terraform's state is a file in a bucket that needs a manual bootstrap, and managing it never stops.*

In the end the test account imported only two resources, security group rules I had added by hand, so import barely mattered there. The OpenSearch index mattered more: every way of letting Terraform own it has a cost, covered in its own section below, and CDK would not have handled it better.

The test account's data layer started empty, so Terraform created almost everything new. Prod is different: every resource there already exists, built in the console and holding real data, so none of them can be deleted and recreated, and each one has to be imported into Terraform. That is when adopting existing resources really counts.

## Prod as a read-only spec, test as the proof

**Terraform never touches prod.** I read prod once, with a script that calls only AWS APIs such as `List`, `Get` and `Describe`, and saved what it returned into a `spec/` directory next to the module: the Glue column definitions, the vector index mapping, and the tool schema the AgentCore gateway exposes.

The Terraform module reads those files, so its configuration traces back to a resource that exists in prod.

*Figure — ProdAsSpec: Prod is only ever read. The module applies to the test account, a separate script copies the data, and the snapshot diff compares both accounts. The data lane and the snapshot diff are explained further down.*

The Terraform module itself holds **no account IDs**. Every external resource it needs is looked up rather than hardcoded: the landing zone gives the VPC and subnets the same `Name` tags in every account, so they are found by tag, and the application stack's security group and task role are found by name.

Because those names are the same in every account, **running the module against prod later needs no change to the module, only one more tfvars file.**

*Figure — ModuleReuse: One module, two accounts. The module finds the network by Name tag and the application stack's resources by name, so only the tfvars file differs.*

S3 bucket names must be unique across all AWS accounts. Two of prod's document buckets have names without an account ID, so the test account cannot create buckets with the same names. The safer pattern is to append the account ID (`kb-docs` → `kb-docs-222222222222`) or the environment name (`kb-docs` → `kb-docs-stg`).

With the AWS resources in place, the data stored in prod still has to be copied: the files in S3 buckets, including the data behind Athena, and the application's records in DynamoDB.

I kept that data out of Terraform state and copy it with a seed script, in two steps: `export` reads prod's data into a local cache, and `import` writes the cache to the matching places in the test account.

Knowledge base IDs, the gateway URL, the code interpreter ID and bucket names all have to be replaced: copied unchanged, they would send the test application to prod's knowledge bases and gateway.

So `import` replaces every prod value with its test counterpart as it writes. Bucket names follow a naming rule, but values AWS generates, such as the knowledge base ID, exist only once Terraform has created the resource.

```python title="scripts/seed_test_env.py"
# One pass, longest match first. The test bucket name starts with the prod
# bucket name ("docs" -> "docs-222222222222"), so sequential str.replace
# calls would rewrite the already-rewritten value a second time.
pattern = re.compile("|".join(re.escape(k) for k in sorted(mapping, key=len, reverse=True)))
text = pattern.sub(lambda m: mapping[m.group(0)], json.dumps(item))
if leftover.search(text):
    raise SystemExit(f"prod value survived in {label}")
```

## What Terraform owns and what it only looks up

Anything another team created, Terraform looks up through a data source, a read-only query that manages nothing. Only what the data layer needs on top of it is written as a resource, which Terraform creates and manages.

The VPC and subnets belong to the landing zone, and the ECS service, its security group and its task role belong to the CloudFormation template the application team manages. **Terraform reads all of them and creates none.**

*Figure — OwnershipBoundary: Terraform looks up the landing zone network and the application stack, and owns the data layer plus the rules that used to be added by hand.*

Comparing prod with the freshly built application stack in test turned up **changes I had made by hand in prod**, on resources the template owns:

1. A port 3000 ingress rule on the service security group. The template's rule allows traffic from the load balancer's security group, which only matches packets that arrive straight from the load balancer. Here, traffic between the public and protected subnets is routed through the landing zone's Gateway Load Balancer (GWLB) endpoint into the provider's inspection VPC, so on arrival it no longer counts as coming from that group. I had added a rule by IP range instead, allowing the load balancer subnets' CIDRs, which matches whichever path the packet takes.

*Figure — InspectionPath: Through the landing zone's inspection VPC, packets no longer count as coming from the load balancer's security group, so only the CIDR rule matches.*

2. A 443 inbound rule whose source is the security group itself, meaning members of the same security group may connect to each other. The ECS tasks, the Lambda and the `bedrock-runtime` VPC endpoint all sit in this security group, so this rule is what lets the tasks and the Lambda reach the endpoint.
3. Two inline policies on the task role, one allowing Athena and Glue access, the other allowing reads of source PDFs.

Terraform now owns all three as separate resources attached to the template's objects. A stack update that removes one gets it back on the next `apply`.

The endpoint also waits for the 443 rule. It has private DNS enabled, so once it exists every `bedrock-runtime` call in the VPC goes through it, including the application's own; without the rule those calls would all be blocked, so `depends_on` enforces the order:

```hcl title="terraform/modules/data-layer/network.tf"
resource "aws_vpc_endpoint" "bedrock_runtime" {
  vpc_id              = data.aws_vpc.lz.id
  service_name        = "com.amazonaws.${local.region}.bedrock-runtime"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = local.protected_subnet_ids
  security_group_ids  = [local.service_sg_id]
  private_dns_enabled = true

  depends_on = [aws_vpc_security_group_ingress_rule.self_https]
}
```

## Creating the OpenSearch Serverless index with `terraform_data`

**The vector index is created by a script through the native API, only if it is absent, and Terraform never updates or deletes it.** Every route to letting Terraform own the index has a cost:

1. The AWS provider has no index resource. `internal/service/opensearchserverless/` contains collections, policies and VPC endpoints, and no `index.go`.
2. The `opensearch` provider can own one, but it plans a replacement when Bedrock adds dynamic fields to the mapping ([#175](https://github.com/opensearch-project/terraform-provider-opensearch/issues/175), still open in v2.6.0), and replacing the index deletes every vector in it. `lifecycle { ignore_changes = [mappings] }` with `prevent_destroy` closes that risk. It does not fix configuration: the provider's `url` is the collection endpoint, which exists only after the collection is created, so **a fresh account needs two applies**, the first with `-target` on the collection, and CI's auto-apply runs one. That limitation is reported in the provider's issues and was not reproduced here.
3. `AWS::OpenSearchServerless::Index` exists in CloudFormation, so `aws_cloudcontrolapi_resource` looked viable. But although Cloud Control lists the prod indexes with `ListResources`, `GetResource` fails on every one of them.

```bash
aws cloudcontrol get-resource --type-name AWS::OpenSearchServerless::Index --identifier "docs-index|https://<collection-id>.ap-northeast-1.aoss.amazonaws.com"
# aws: [ERROR]: An error occurred (GeneralServiceException) when calling the GetResource operation: AWS::OpenSearchServerless::Index Handler returned status FAILED: Error occurred during operation 'GetIndex'.
```

Every `plan` first refreshes each resource, so as long as `GetResource` fails, every `plan` fails with it. The native `create-index` and `get-index` calls work, so I switched to a small script that does create-if-absent and nothing more:

```bash title="terraform/modules/data-layer/scripts/ensure_index.sh"
for i in $(seq 1 12); do
  if aws opensearchserverless get-index --id "$COLLECTION_ID" --index-name "$INDEX_NAME" >/dev/null 2>&1; then
    echo "index ${INDEX_NAME}: exists"; exit 0
  fi
  # A new data access policy can take a while to become ready, so retry.
  if aws opensearchserverless create-index --id "$COLLECTION_ID" --index-name "$INDEX_NAME" \
      --index-schema "file://${SCHEMA_FILE}" >/dev/null 2>&1; then
    echo "index ${INDEX_NAME}: created"; exit 0
  fi
  sleep 15
done
exit 1
```

The script also means one provider fewer: no third-party provider, no second SigV4 setup, no extra version to track. **The cost is that the index is in no plan.** Terraform cannot see mapping drift, only the snapshot diff later in this article catches it, and mapping changes are applied by hand.

I created a `terraform_data` resource to run it, with the collection ID, the index name and a hash of the mapping file as triggers.

I also put a 60-second `time_sleep` between the index and the knowledge base: right after creation the index is still propagating, and creating the knowledge base then can fail because the index is not found ("no such index").

I put only the three fields the knowledge base names into the mapping file: the vector, text and metadata fields. **Bedrock adds the rest dynamically during ingestion.**

## Proving parity: a normalized config diff and identical retrieval

I wrote a snapshot script that **looks up the resources in prod and test by name, compares their settings, and fails on any difference not listed in `EXPECTED_DIFFS`**. Every resource, from the knowledge base to the gateway, is found by name and compared.

Before comparing, the script replaces account-specific values with placeholders such as `<ACCT>`, `<bucket:rag>`, `<kb:docs>` and `<subnet:LZPrivateSubnet1a>`, so values that are supposed to differ, like account IDs, do not show up as differences. Lists are sorted after the placeholders go in, so the JSON from prod and test comes back in the same order.

```python title="scripts/infra_snapshot.py"
EXPECTED_DIFFS = {
    "lambda.code_sha256": "test runs the repository handler; prod runs older console code",
    "lambda.handler": "same reason",
    "kb_data_sources.*.name": "prod names were generated by the console quick-create",
    "*.tags": "test carries default_tags (ManagedBy, Repository)",
}
```

The first run printed three differences, all of them settings the Terraform module had wrong:

1. S3 Bucket Keys were enabled in prod and disabled in test.
2. The module was missing the marketplace statement the console adds to every knowledge base role in prod.
3. A gateway role statement was scoped differently from prod.

After I fixed those three, **every difference outside `EXPECTED_DIFFS` was gone**. The index mappings matched too, including the fields Bedrock added during ingestion.

Even when configuration matches, behaviour may not, so I ran the same retrieval test in both accounts: five queries against each of the two knowledge bases, top five results. **All ten returned the same documents in the same order**, and every source link in test pointed at a test bucket.

I also counted rows in five tables with Athena, and both sides matched.

## GitHub OIDC: plan on pull requests, apply on merge to main

I set up a GitHub Actions workflow: every pull request that touches `terraform/` gets a plan, posted as a comment on that pull request, and once the pull request merges to `main` the workflow applies to the test account. AWS credentials come from GitHub's OIDC token, so the repository stores no keys.

### Two roles with two subjects

The OIDC token GitHub issues carries a subject (the `sub` field) recording which repository the token came from and which event triggered it, and an AWS trust policy compares that value to decide who may assume which role. So I set up two roles, each trusting one subject only:

1. **Plan role**: the plan job runs code from the pull request, so the plan role trusts only the `pull_request` subject, and it is read-only.
2. **Apply role**: the apply job runs in the GitHub environment `test`, so the apply role trusts only the `environment:test` subject, and it has administrator access.

*Figure — OidcRoles: The plan role trusts pull requests and cannot read data. The apply role trusts only the test environment, and the environment only deploys from main.*

I put the two jobs in separate concurrency groups. GitHub keeps one pending run per group, so **a shared group would let a newer pull request's plan replace an apply that was waiting**.

Before a plan or an apply, Terraform creates a `.tflock` lock file next to the state file in S3, so the two jobs never operate on state at the same time.

### Fixing the `sub` prefix

The first pull request failed before Terraform started:

```text
Retry AssumeRole: attempt 1 of 12 failed: Could not assume role with OIDC: Not authorized to perform sts:AssumeRoleWithWebIdentity.
```

The trust policy expected `repo:<owner>/<repo>:pull_request`. **The repository sends `repo:<owner>@<owner-id>/<repo>@<repo-id>:pull_request`**, which embeds the numeric owner and repository IDs. A trust policy compares the subject exactly, so AWS refused `AssumeRoleWithWebIdentity`. The prefix GitHub actually uses can be checked with this API:

```bash
gh api repos/<owner>/<repo>/actions/oidc/customization/sub
# {"use_default":true,"use_immutable_subject":true,"sub_claim_prefix":"repo:<owner>@<owner-id>/<repo>@<repo-id>"}
```

> **HEADS UP**
>
> **Why GitHub made the subject immutable.** If a repository is deleted or its organization renamed, and someone registers the freed name, GitHub mints the new repository tokens with the old subject, so the new owner inherits the old trust. The new format appends immutable IDs, as in `repo:octo-org@123/octo-repo@456`, binding the subject to the original repository.

I changed both trust policies to match that prefix:

```hcl title="terraform/bootstrap/test/ci.tf" del={2} ins={3}
locals {
  github_subject = "repo:<owner>/<repo>"
  github_subject = "repo:<owner>@<owner-id>/<repo>@<repo-id>"
}
# plan:  "${local.github_subject}:pull_request"
# apply: "${local.github_subject}:environment:test"
```

### Adding Bedrock reads to ReadOnlyAccess, and denying data reads

After I fixed the trust policy, the plan got further, but then failed with this error:

```text
Error: listing tags for Bedrock Agents Knowledge Base (arn:aws:bedrock:ap-northeast-1:<account>:knowledge-base/<kb-id>)
AccessDeniedException: User: arn:aws:sts::<account>:assumed-role/github-terraform-plan/GitHubActions is not authorized to perform: bedrock:ListTagsForResource
```

`ReadOnlyAccess` does not include `bedrock:ListTagsForResource`, so I added `bedrock:Get*` and `bedrock:List*` to the plan role, beside the `bedrock-agentcore` and `aoss` reads it already had. Then I reran the plan with `gh run rerun <run-id> --failed`, which picks up the updated role without a new commit.

`ReadOnlyAccess` also reads far more than a plan needs: S3 objects, DynamoDB items and CloudWatch log events. The test account holds a copy of production data and the application's chat logs, and the plan role runs pull request code, so anyone who can open a pull request could read that data through the plan.

So I **added a rule denying every data read outside the state bucket**:

```hcl title="terraform/bootstrap/test/ci.tf"
statement {
  sid    = "DenyDataReads"
  effect = "Deny"
  actions = [
    "s3:GetObject", "s3:GetObjectVersion",
    "dynamodb:GetItem", "dynamodb:BatchGetItem", "dynamodb:Query", "dynamodb:Scan",
    "logs:GetLogEvents", "logs:FilterLogEvents", "logs:StartQuery",
    "athena:GetQueryResults", "bedrock:Retrieve", "bedrock:RetrieveAndGenerate",
  ]
  not_resources = ["${aws_s3_bucket.state.arn}/*"]
}
```

> **CAREFUL**
>
> **`simulate-principal-policy` reported `explicitDeny` for every action.** Given one role and a list of actions, it reports a decision per action, but it folds in the landing zone's service control policies and returned `AllowedByOrganizations: false` across the board. So I switched to `simulate-custom-policy`, passing only the role's own policy plus an allow-all, which tested the Deny on its own.

### Locking apply to main with a deployment branch policy

**The AWS trust policy cannot tell which branch an environment job came from**: an environment job's subject (`sub`) names only the environment, as in `…:environment:test`, and never the branch. Without a branch rule on the GitHub side, a `workflow_dispatch` from any branch, or a pull request that adds `environment: test` to a job, assumes the administrator role.

If the environment a workflow names does not exist yet, it is created automatically, with no branch protection by default. So I created the `test` environment before the workflow first ran and limited it to deploying from `main`:

```bash
gh api -X PUT repos/<owner>/<repo>/environments/test --input - <<'JSON'
{"deployment_branch_policy": {"protected_branches": false, "custom_branch_policies": true}}
JSON
gh api -X POST repos/<owner>/<repo>/environments/test/deployment-branch-policies -f name=main -f type=branch
```

> **NICE**
>
> After the merge, the apply from `main` assumed the role and reported `Apply complete! Resources: 0 added, 0 changed, 0 destroyed.` The Lambda package is rebuilt on the runner, and had its hash differed from the local build, Terraform would have updated the Lambda. **0 changed proves the two builds produced the same package.**

## AgentCore network interfaces hold the security group for up to 8 hours

Finally, I tried rebuilding the test account from zero, but `terraform destroy` stopped on the code interpreter's security group after the 45-minute delete timeout:

```text
Error: deleting Security Group (sg-<id>): operation error EC2: DeleteSecurityGroup, https response error StatusCode: 400, api error DependencyViolation: resource sg-<id> has a dependent object
```

The code interpreter itself was already gone, but two network interfaces of type `agentic_ai` were still attached to the security group. They were requested by an AgentCore service account, an account AWS manages, not by the test account. AWS documents that these network interfaces **may persist for up to 8 hours** after deletion, and detaching them is not permitted.

That did not block the rebuild. Everything else had been destroyed and only that security group remained in state, so I ran `terraform apply`, and **it reused the security group and rebuilt the rest of the account**.

After I re-ran the seed script, the snapshot diff, the retrieval test and the Athena row counts all came back the same as before.

## Summary

Exporting prod's configuration into a `spec/` directory gave the Terraform module something to be checked against.

I chose Terraform over CDK because it writes the configuration itself when it imports existing resources. The vector index went to a script calling the native API: in a fresh account the `opensearch` provider cannot be configured until the collection exists, and CI's single apply cannot do that.

The CI side hit three problems that a role created by hand never meets: the immutable subject format, the Bedrock reads `ReadOnlyAccess` lacks and the data reads it should not have, and the deployment branch policy that the environment trusted by the apply role needs.

## References

- [Terraform: import blocks and generating configuration](https://developer.hashicorp.com/terraform/language/import)
- [Terraform S3 backend, including use_lockfile](https://developer.hashicorp.com/terraform/language/backend/s3)
- [CloudFormation: resource types that support resource import](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/resource-import-supported-resources.html)
- [AWS CDK v2: cdk import](https://docs.aws.amazon.com/cdk/v2/guide/ref-cli-cmd-import.html)
- [GitHub changelog: immutable subject claims for GitHub Actions OIDC tokens](https://github.blog/changelog/2026-04-23-immutable-subject-claims-for-github-actions-oidc-tokens/)
- [GitHub Docs: OpenID Connect reference](https://docs.github.com/actions/reference/openid-connect-reference)
- [Amazon Bedrock AgentCore: deleting a code interpreter, and the 8-hour network interface retention](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/code-interpreter-delete.html)
- [terraform-provider-opensearch #175: index replacement with dynamic properties](https://github.com/opensearch-project/terraform-provider-opensearch/issues/175)