I was asked recently: “What mistake of your subordinates did you remember the most?”
In one of my past companies, we planned to move to a new data center. The system administrator set up database restoration from a copy of the last day, but for some reason made it a daily task, obviously planning to cancel the task after moving, and.. forgot about it.
The day in the new data center we lived in peace, and the apocalypse happened a day later – the database was overwritten and we lost data for a full business day in our information system, which, I must say, was responsible for everything: data on the warehouse, the movement of goods through it, purchases, deliveries and shipments, orders of all users, order payments, new users data and much more.
We had to restore the database from various sources, while the company continued the operational day and the recovery should have taken place taking into account all the new changes.
This hell lasted about 60 hours without sleep breaks or trips home. It was miracle that we managed to do it, but this experience was firmly imprinted in the memory of everyone it touched.
