# _override.tf runs committed Terraform in a sandbox — and apply caught what validate missed

> Gitignored Terraform override files retarget a committed config at your own AWS sandbox: a local backend, overridden locals, and what only apply found.

- Source: https://oharu121.com/blog/terraform-override-tf-sandbox-testing-real-apply/
- Published: 2026-08-27T21:54:04+09:00
- Tags: Terraform, AWS, GitHub Actions

---
**Key takeaways**

- `*_override.tf` is already in the standard Terraform `.gitignore`, so sandbox-only settings can sit beside a committed config without ever being committed.
- An override file can replace a `backend` block and individual `locals`, which is enough to retarget a whole configuration at a different account.
- `terraform validate` checks schema, not acceptance. Four defects in this config survived a clean validate and only appeared on a real apply.
- The most valuable of those: a CLI had silently discarded a setting it echoed back and wrote to its own config file. Terraform preserved it.
- Run `plan` a second time after apply. "No changes" is the check that the config actually describes what exists.

## Introduction

I needed to answer whether a set of AWS resources that had been created by hand could be reproduced through CI/CD, and that answer needed evidence rather than an opinion. The awkward part was where it would run: the configuration belongs in a shared repository targeting AWS accounts I have no access to, and the only account I could actually deploy to was a sandbox.

The agent's approach was to write the Terraform against the shared repository's conventions, then **retarget it at the sandbox using two gitignored override files** so the committed configuration never contained anything sandbox-specific. That worked, and it turned a design discussion into a measured result.

More usefully, the apply surfaced **four defects that a clean `terraform validate` had not**, including a setting a CLI had silently thrown away. This article covers the override technique, what each defect cost, and the pipeline the result argues for.

## The situation: a config that cannot be applied where it is written

The shape of this is common enough to be worth naming. A configuration is written to match a shared repository's conventions, referencing a state backend, a permissions boundary, and account-specific identifiers that only exist in an environment you cannot reach. Everything about it is correct and none of it can be run.

**The tempting workaround is to edit the values locally, apply, and remember to revert.** That works exactly until you forget, and what gets committed is a config pointing at your sandbox.

## Two override files

Terraform merges any file named `*_override.tf` over the configuration it sits beside. **The standard Terraform `.gitignore` already excludes that pattern**, which is what makes this safe by default rather than by discipline:

```text title=".gitignore"
# Ignore override files as they are usually used to override resources locally and so
# are not checked in
override.tf
override.tf.json
*_override.tf
*_override.tf.json
```

The first override replaces the backend, so state stays on disk instead of in a bucket that does not exist yet:

```hcl title="backend_override.tf"
terraform {
  backend "local" {}
}
```

The second replaces individual `locals`. Only the named keys change; everything else in the original `locals` block still applies:

```hcl title="sandbox_override.tf"
locals {
  # The sandbox has no permissions boundary; the committed value names a
  # policy in an account this run cannot see.
  existing_permission_boundary_arn = null
  runtime_name                     = "example_sandbox"
  artifact_bucket_name             = "sandbox-artifacts-<account-id>"
}
```

Module arguments can be overridden the same way, which matters when the sandbox needs a different artifact than the one CI would build:

```hcl title="sandbox_override.tf (continued)"
module "compute" {
  entry_point  = ["app/server.py"]
  runtime_type = "PYTHON_3_12"
}
```

A module override merges rather than replaces, so `source` and every other argument continue to come from the committed `main.tf`. Two short files, nothing committed, and `git status` stays clean.

*Figure — OverrideMechanism: The sandbox run adds two gitignored files. The CI run adds nothing.*

## What plan proved, and what it did not

`plan` came back with **12 to add, 0 to change, 0 to destroy**, and reading it caught a naming bug that neither `validate` nor `fmt` would ever flag. An output rendered as `https://mcp-mcp-dv-...` because a template interpolated a project name into a string that already contained it. Obvious once visible, invisible until something rendered it.

But a successful plan proves the provider *accepts* the configuration. **It does not prove AWS does.** That distinction is where the rest of this article lives.

## What apply caught that validate did not

| What surfaced | Why only apply could show it |
| --- | --- |
| Provider returned `null` for an empty `environment_variables` map | The inconsistency is between what was sent and what came back |
| Omitting `max_lifetime` made AWS supply `28800`, producing a permanent diff | The default is applied server-side |
| `destroy` failed with `409 BucketNotEmpty` on a versioned bucket | Only a real teardown exercises deletion |
| A CLI had silently discarded a setting it claimed to apply | Requires comparing two really-deployed resources |

The first two arrived as the same error class:

```text
Error: Provider produced inconsistent result after apply
… produced an unexpected new value: .environment_variables: was
cty.MapValEmpty(cty.String), but now null.
```

Both are provider bugs, and both are trivially avoidable once seen. Passing `null` rather than `{}` fixes the first. Setting the value AWS would default to fixes the second, and the comment explaining why is worth more than the line itself:

```hcl
# Omitting max_lifetime makes AWS set 28800 and return it, so the provider
# reports "was null, now 28800" on every plan.
lifecycle_configuration = [{
  idle_runtime_session_timeout = var.idle_timeout
  max_lifetime                 = var.max_lifetime
}]
```

The third only appears at teardown. **A bucket with versioning enabled is not empty after you delete its contents.** The delete markers and noncurrent versions remain, and `destroy` stops with `409 BucketNotEmpty`. The fix is a `force_destroy` flag, and finding it during a deliberate teardown costs nothing. Finding it when you urgently need an environment gone costs considerably more.

## The setting that was silently discarded

This is the one that justified the whole exercise.

The resources had originally been created by a vendor CLI. That CLI accepts a flag setting an idle session timeout, prints the value back in its own summary output, and writes it to its own configuration file. Every local artifact agreed the setting was 300 seconds.

Querying the two really-deployed resources told a different story:

| Resource | Created by | Deployed idle timeout |
| --- | --- | --- |
| original | vendor CLI, flag set to 300 | **900** |
| sandbox | this Terraform | **300** |

Same account, same region, same intended configuration. **The API accepts the value; the CLI never sent it.** Reading the tool's source found the cause: the code path for container deployments passes the lifecycle configuration through to the API call, and the code path for zip deployments does not. The flag is parsed, validated, echoed, persisted, and dropped.

The billing model made this more than cosmetic, because memory is charged for as long as a session is alive. But the general form is what matters: **the only artifact that disagreed with the mistake was the deployed resource itself.** Nothing local could have revealed it, and no amount of re-reading the config would have helped.

## Confirming the config describes reality

After apply, run `plan` again:

```text
No changes. Your infrastructure matches the configuration.
```

That sentence is the acceptance test. It proves the configuration describes what exists, rather than merely having been accepted once. Both provider inconsistencies above would have shown up here as a permanent diff if they had been left unfixed.

**Then destroy, and check that unrelated resources in the account survived.** A teardown that takes something else with it is a defect you want to discover in a sandbox.

## The pipeline this is for

**This part is a design, not something running yet.** It is blocked on OIDC trust being established between the repository and the target accounts, which is somebody else's task and has its own lead time. I am including it because the sandbox result is what argues for it, and because the shape generalises.

*Figure — Pipeline: Plan on the pull request, apply on merge. The artifact step is the only part that differs from a container-based deployment.*

**The intended flow is the conventional one:** a pull request runs `fmt`, `validate`, lint and `plan`, publishing the diff for review without applying anything. A merge builds the deployment artifact, uploads it, and applies to a staging environment. Production is a manual trigger.

### Two details that are easy to get wrong

**Authentication is OIDC, not stored keys.** The workflow requests `id-token: write` and assumes a role per environment, so nothing long-lived is stored in the repository. This is standard, and it is the part that has lead time, because the trust relationship has to be created by someone with administrative access to the target account.

**The artifact step is the only genuinely new piece.** For a container-based service, CI builds an image, pushes it to a registry, and the deployment references it. For a zip-based one, CI builds a zip, uploads it to object storage, and the resource references the key. Everything else is unchanged: the plan-on-PR job, the apply-on-merge job, and the environment gating.

The apply also exposed an ordering problem in this design, which the agent deliberately left unsolved. The bucket and the resource that reads from it are created in the same apply, so on a first-ever run the artifact does not exist when the resource is created. Uploading from Terraform with an object resource fixes the ordering but makes `plan` fail whenever the artifact is absent locally. **It was left unimplemented rather than committed as an untested answer**, which is more honest than shipping a guess into a branch whose entire value is that it was verified.

## Summary

`*_override.tf` is a small feature that solves a specific and recurring problem: running a configuration in a place it was not written for, without editing it. The pattern is already gitignored, so the safety is structural rather than remembered.

The larger point is about what counts as evidence. `validate` and a green `plan` mean the configuration parses and the provider accepts it. **The four defects here all sat on the far side of that line**, and one of them had been silently wrong for days while every local artifact insisted it was fine. A sandbox account costs almost nothing and is the only place that distinction can be tested safely.

## References

- [Terraform override files, including the merge semantics for locals and module blocks](https://developer.hashicorp.com/terraform/language/files/override)
- [Terraform backend configuration, on overriding the backend for a local run](https://developer.hashicorp.com/terraform/language/backend)
- [The gitignore.io Terraform template that already excludes `*_override.tf`](https://www.toptal.com/developers/gitignore/api/terraform)
- [Configuring OpenID Connect in AWS for GitHub Actions, which removes long-lived credentials from the repository](https://docs.github.com/en/actions/how-tos/security-for-github-actions/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services)