Over the years working as a programmer I’ve caused several downtimes, broken features and critical bugs. I’ve tried to think which were the worst ones, or at least the most interesting ones. I only came up with three which surprises me a bit because I thought there would be more.

  1. The over-update - The classic “SQL update where statement that updates too much”. I’m sure this one is very common since it’s easy to make. The query I wrote ran after a user made an action and updated the data for all other users. A lot of angry emails from users were received complaining that their data changes on it’s own without them doing anything. A previous database snapshot had to be restored and the affected users had temporarily bad data until the issue was fixed.

  2. The email over-sending - This one is similar to the over eager SQL update. The feature I was working on was supposed to send an email to a group of recipients. It worked fine, but it also sent the email to more recipients than it should have. Besides the weird customer experience it was also a security risk. The CEO of a major company who were the client of the company I worked for, got notified about the issue while on a ski vacation and called my boss for an explanation. If you are wondering, I did have an extensive testing for this feature that even tested that the emails were not sent to the wrong recipients but I didn’t take a specific edge case into consideration. The irony was that this whole issue was fixed by a tiny change to the code.

  3. The disappearing data - For a new project that I had to build fast and that took off quickly I used a new ORM that I found and which looked promising. After a few days I’ve realized that some of the data in my database tables disappeared. I spent hours researching the issue thinking that it is some sort of malfunction of the database. Eventually I realized that I was running the ORM in development mode and it had a “special feature” that loaded the entire database into memory each time a query ran on it. Once the size of the database became too big to be held in memory, the process crashed and the data would disappear. The end result was loss of data, but even worse, it meant that some users saw content of other users as a result of a few tables being reset and others not. It was my fault for running the ORM in development mode in production and not reading the documentation properly. However I also didn’t use that ORM ever again.