← All posts

Terraform state: lessons from three disasters

Terraform state is the most dangerous file in your infrastructure, and most teams treat it more casually than their application code.

Disaster one: two engineers ran apply concurrently against local state. The fix is table stakes now — remote state with locking from day one — but the deeper lesson was that convenience defaults are production risks.

Disaster two: a CI misconfiguration pointed a destroy at the wrong workspace. Lesson: production applies go through a plan that a human reads, environments live in separate state files with separate credentials, and destroy requires a flag nobody sets by accident.

Disaster three: a stuck lock, a hasty force-unlock, and a state file that no longer matched reality. Lesson: state surgery is an incident. Declare it, snapshot the state first, and do imports and moves one resource at a time with a second pair of eyes.

Boring process is what makes infrastructure as code safe. The code is the easy part.

// more like this

Blameless postmortems that actually change things

Set SLOs before you build dashboards

Comments

Comments are moderated before appearing. Leave your email to be notified when yours is approved or replied to.