The only thing worse than causing a major product failure is causing it a second time in the exact same way. That's why it's critical to run a post mortem after you've had any kind of major failure or outage. 

In this podcast, Micah and Laura discuss what people get wrong about post mortems and give some suggestions about making them better.