Experiencing the Aftermath Will Make You Tougher, Wiser, and Ready For Anything

This article has been indexed from DZone Security Zone

I was in charge of managing a dedicated server running Debian 7. The server hosts multiple websites with email services, the server also hosts multiple instances of a critical web application for a client who is running a business across different regions.

That day was a very important day, as the client was expecting his own client to turn up. The client wanted to make a demo of the application and show how they manage some business processes.

During that event, I got a phone call claiming client users not able to access the web application. I took the request as usual and started checking the filed issue, and a few seconds later, I got another call about other users not able to access their mailbox. It is then I realized that a very nasty thing is happening and I’m in serious trouble.  

I quickly figured out that I had made the worst mistake ever!

That day I was performing usual maintenance tasks on the server, freeing some disk space here and there. However, at some moment, I deleted critical files that belong to different services like Postgres, Mysql, mail server, etc. I didn’t notice anything until I started receiving reported issues from the clients. 

It was catastrophic in all corners.

We lost three months of data as backups resided in a single place where I launched the deletion operation, there were no other copies of backups. Many services were surviving with what was left in RAM (I guess) and any respawned process was lethal for the corresponding service.

In the field, the client was badly embarrassed in front of his client, as he was cut off at the beginning of the demo.