git rm --cached left 26 MB in .git — the working tree was clean and history was not

Introduction

I was asked for the source of a project as a tgz, excluding .venv and similar. That is a two-minute request, and packaging it is what surfaced the problem: the repository was 26 MB, and almost all of it was a build artifact that had been committed on the first commit without anyone noticing.

The agent removed it with git rm --cached, added the directory to .gitignore, and reported that the repository had gone from 26 MB to 292 KB. That claim was wrong, and it is the ordinary way this mistake survives: the tracked files really had shrunk to 292 KB, and nobody looked at .git, which was still 26 MB. The blob was reachable from five of the six commits and would have travelled with the repository forever.

The real fix was git filter-repo, and it took under a second. This article covers how the artifact got in, why the obvious removal command did not remove it, how to tell the difference by measuring, and why the window for fixing it cheaply closes at the first push.

How 26 MB got in: .gitignore matched the file, not the directory

The deployment tool for this project writes two things into the repository root. A config file, .bedrock_agentcore.yaml, and a build directory, .bedrock_agentcore/, holding a zip of vendored dependencies.

The .gitignore the agent wrote listed one of them:

.venv/
.cognito.env
.bedrock_agentcore.yaml
__pycache__/

A gitignore pattern is matched against paths, not against name stems. .bedrock_agentcore.yaml matches a file with exactly that name. It says nothing about .bedrock_agentcore/, which is a different path, so the directory and its 26 MB zip were never ignored and went in with the first git add -A.

Nothing complained. The commit succeeded, the tests passed, and the repository worked perfectly for the next several hours.

git rm –cached, and the number that looked like success

When packaging the archive, the agent found the artifact and removed it the obvious way:

git rm -r --cached .bedrock_agentcore
printf '.bedrock_agentcore/\n' >> .gitignore
git commit -m "Stop tracking the AgentCore build artifact directory"

Then measured, and reported the result as a fix:

git ls-files -z | xargs -0 du -ch | tail -1
#   292K  total

The archive itself was built with git archive, which walks the tree rather than the object store, so it came out at 83 KB and was genuinely clean. Everything visible agreed that the problem was gone.

The measurement was of the wrong thing. git ls-files lists what is tracked now. It has no opinion about what previous commits contain, and neither does git status, and neither does the size of your working directory.

What was actually still there

git rm --cached does one thing: it removes a path from the index, so the next commit will not contain it. It is not a history operation and it does not claim to be. Commits are immutable, so the five commits that already referenced the blob still referenced it, and an object reachable from any commit is an object git keeps.

Two commands settle it. One measures the object store, the other asks whether the blob is still reachable:

du -sh .git
#   26M  .git

git rev-list --objects --all | grep dependencies.zip
#   de67db72dca595f3e22bd20b3ac5c77449734dd7 .bedrock_agentcore/.../dependencies.zip

So the repository was 292 KB of files inside a 26 MB repository. The gap between those two numbers is the whole bug, and it is invisible to every command that reports on the working tree.

The window: nothing had been pushed yet

Two facts decided that this was worth fixing immediately rather than living with.

The first is that this repository had no remote at all. It was created locally and had never been pushed, so no other clone existed and no commit ID was referenced by anything outside the directory.

The second was worth checking rather than assuming, because the same code had been contributed to a shared repository as a feature branch. If the artifact had travelled there, this would have been a much more awkward conversation:

git rev-list --objects --all | grep -iE "dependencies\.zip|bedrock_agentcore"
#   (no output)

Clean. The branch added 183 KB in total, because it had been assembled by copying source files rather than by copying the repository.

That combination is the cheap case. A history rewrite changes every commit ID after the point it touches, which is free when nothing references those IDs and expensive when a colleague has them checked out, a pull request is open against them, or CI has recorded them. I have skipped a rewrite before on exactly this reasoning, when the recoverable weight was 0.8 MB and the commit IDs were already carried by 25 tags and 34 merged pull requests. Here it was 26 MB and nothing had been pushed, which is the opposite answer to the same question.

Rewriting with filter-repo

git filter-repo replaced filter-branch and is what the git documentation now points at. The operation is a path removal across all commits:

# Back up first: with no remote, this directory is the only copy.
git bundle create ../project-backup.bundle --all

git filter-repo --path .bedrock_agentcore --invert-paths --force

--path selects a path, and --invert-paths inverts the selection so that everything except it is kept. --force is needed because filter-repo expects a fresh clone and this was the working repository.

It is fast, and it says what it did:

Parsed 6 commits
New history written in 0.07 seconds; now repacking/cleaning...
Completely finished after 0.20 seconds.

What changed, and what did not

Every commit survived with its message intact, and the working tree was untouched, which a spot check confirmed by importing the package. What changed is every commit ID, exactly as promised.

One cosmetic consequence is worth expecting. The commit whose entire purpose was removing the artifact now removes nothing, because the artifact no longer exists at any point in the rewritten history. All that is left in it is the .gitignore line:

6d88cf7 Stop tracking the AgentCore build artifact directory
 .gitignore | 7 ++++---

The message still describes the intent accurately, so it stayed. Fixing it would mean a second rewrite for no functional gain.

There is one last trap, and it is a funny one. The backup bundle is 25 MB, because it faithfully preserves the history containing the blob. Leaving it next to the repository means keeping precisely the bytes you set out to remove. Verify the rewrite, then delete it:

git bundle verify ../project-backup.bundle
#   The bundle records a complete history.

Summary

git rm --cached is not a history command and does not pretend to be. It removes a path from the index so that future commits omit it, which is exactly right for the case it is designed for and completely insufficient for a file that has already been committed.

The reason the mistake sticks is that everything convenient to measure agrees with you afterwards. git status is clean, the working tree is small, git archive produces a tidy tarball. The one number that disagrees is du -sh .git, and it is the only one that describes what a clone will actually download.

If a large file has been committed, check whether anything has been pushed before doing anything else. That single fact decides whether the fix is a sub-second local command or a coordination problem.

References

• git rm documentation, including what –cached does to the index
• git filter-repo, the tool the git project recommends in place of filter-branch
• gitignore pattern format, on how patterns match paths and how a trailing slash restricts a match to directories
• git bundle documentation, for creating and verifying an offline backup of a repository