Two of my most memorable moments as a programmer

[[235144]]

Last week, my colleagues and I had a quick chat about the times we screwed up at work. Now that we no longer make those mistakes, it's funny to think about them. But even though it's funny, these mistakes have actually taught us a lot.

Sharing your mistakes is important so others can learn from them and potentially get better at their jobs. Here are a few of my recent mistakes.

Why are so many production databases deleted by mistake?

[[235145]]

A few months ago, there was a post on Reddit about an entry-level developer who accidentally deleted the production database on his first day. We cringe when we see posts like this about people making huge, indelible mistakes. We realize that it’s not impossible for us to make those kinds of mistakes ourselves — most of the time we pull back from the brink.

There are so many examples of this happening. When I was at my first job, a senior DBA accidentally deleted the production database on his first day. The team helped him recover from the mistake with a week-old backup of the database, and he was able to keep his job. Now, 10 years later, he is still being teased about it.

One morning earlier this year, I was called in to investigate a problem that had occurred in production for a client. They were about to conduct a beta test of the product with a small number of users, but their website homepage suddenly stopped displaying anything. I guessed that there might be a bug or a vulnerability in the system.

I logged into the production machine, called up the database, and found that the articles table was empty. OK, this confirms that the web page is blank.

There were still users in the user table, which was strange, so we lost all the articles, but at least their test users still had their accounts, and we could explain that this was a beta version and this kind of thing happens from time to time.

I was confused for a while. I couldn't remember what I did. I didn't think I was stupid enough to enter the command to delete the users in the table in the console window, but there it was - there was no articles table and no users table. I sat there, feeling a little shocked.

Then my brain went into overdrive, trying to figure out how to fix the problem. Did I really delete the user table? Yes. Did we run a backup of the database? No. How would we explain this to the client? I had no idea.

I remember going to the project manager, sitting down with her and explaining what had happened, and how there was no data in the articles table, so the site looked empty. Oh, and I had accidentally deleted the users table. Now they needed to re-invite all the users - if they could even figure out who they were. Oops.

I returned to my seat feeling deeply defeated.

But I felt something was wrong. How could we have lost all the articles tables in the first place? So I continued to dig deeper, partly because I couldn't accept the result, and partly because I wanted to save face. After a while, I noticed the key problem.

There are 5 other databases on the server, one of which has a very similar name to the one I'm looking at.

I checked and the articles were there, and the users table was intact. It turns out that a configuration change had inadvertently made it a production database, causing the site to point to a brand new database. Where were the users I saw there? Seed data.

What a relief! I was feeling nervous and sick all morning, but luckily we “repaired” all the data and found the real cause of the problem, without having to announce the bad news of accidentally deleting the database in advance.

We benefited a lot from this little episode, the simplest one is: now we always back up our database... This is probably the most effective stomach medicine for us developers.

Always rushing to make progress, but never catching up

[[235146]]

Another prominent mistake I made recently was less dramatic, but was actually a series of small mistakes that eventually accumulated into a big problem.

One of the challenges of our project development is the tight timeline (but not all?)

At the first meeting, we all agreed that the project would take twice as long as we could afford. From the beginning, the deadline was tight, so we quickly passed the certification phase to move on to the features that the customer really cared about.

I had only implemented authentication once before in a single page app, but I still didn't fully understand how the various parts of the app fit together.

Trying to get the app out as fast as I could was a big mistake. I missed something very important:

1. After the user logs in, the page is loaded through cookies, but my app page does not provide a waiting time for loading, but decides the order based on the order of events, so the server will reply that you do not have permission. This error is rare and difficult to occur again, because in most cases the events are completed in the correct order.
2. Moreover, the authentication process never checks whether the user token is invalid. If you do not visit the website frequently, when you find that you cannot log in to the website, you need to log out and log in again.
3. The token should be renewed on every request, but I never had the time to understand the rules. So here comes the timing problem again. If we make several requests at once, and the replies we receive depend on the order they came, then future requests will use the wrong token.

We pushed as hard as we could, but ended up taking twice as long as we were given. The difference was that we built apps with more bugs, and then spent even more time tracking them down and fixing them.

I was embarrassed by the mistakes I made at work and felt ashamed in front of everyone because I had messed up everything.

I will say this: since then, I’ve spent time learning about authentication mechanisms and now understand OAuth, JWT, refresh tokens, and invalidation. I’ve carefully read authentication code written by others in many libraries, and built authentication flows in several different languages ​​and frameworks.

Failure is the mother of success

This is what I learn from every failure. As long as you are willing to learn, almost every such experience will benefit you.

If you learn from your mistakes, you'll improve. If a player makes a mistake for the first time, I try not to be upset with them, because they usually know they messed up.

But I also try not to blame those who always make mistakes and refuse to change despite repeated warnings. They also need sympathy.

If you can do these four things when dealing with mistakes, you will continue to improve:

• Laugh at your mistakes
• Learning from experience
• Afterwards, I tried to clear my name
• Share it with others so that they can benefit from it too.

I leave you with a famous anecdote about the value of making mistakes: In the early 20th century, Thomas J. Watson, president of IBM, encountered an employee who had caused the company to suffer heavy losses due to repeated mistakes. When asked whether he should fire the employee, Watson replied:

"No, I just spent $600,000 training him, why would I let someone else hire him for his experience?"

What interesting mistakes have you made in the past? Share them with us!