Debugging is all about Challenging Assumptions

I’ve been awash in debugging a new code deployment most of the week. The gist of it, if you’re interested, is that we’re deploying a group of machines running several Redis instances (one per CPU core).  Collectively they’ll serve as an alternative to an existing MySQL-based system for tracking a variety of short and medium term data points.

A bit of data (just an identifier) will be fed into the system by some of our front-end web servers and then picked up by another process that will fetch more data, crunch it lightly, and then store summary information back in various Redis buckets.  (Sorted Sets are cool, BTW.)

The trouble came, as it often does, shortly after the deployment when I started seeing errors and slow responses from some of the web servers using the new code.  To troubleshoot,  we rolled back the code on all but one server and I spent the next several days working out what was wrong.

Zip Line Fun in Mexico

I’ve found a bug in the high-level module I wrote, a few bugs in the lower-level module I’ve been using (but did not write), and a bug in the wrapper module that I wrote to sit between the high-level and lower-level module. Some of the bugs were findable by a simple reading of the code and thinking through the symptoms I’ve been seeing. Others required a lot more “debugging with print statements” and making guesses.  Those generally take a lot longer.

The common thread in all of the harder to find bugs, both in this situation and in others I’ve faced, is that my assumptions often get in the way. The more I can step back and challenge the “obvious” things I believe about what should be happening, the faster I can figure out what the heck went wrong.

It gets easier and faster with time and practice, but it’s still frustrating. I think it’s human nature not to question your assumptions about things that should be “obvious”, but this is one of the situations in which that is essential.  Because what you think is happening and what is really happening are often just different enough to make life interesting.

About Jeremy Zawodny

I'm a software engineer and pilot. I work at craigslist by day, hacking on various bits of back-end software and data systems. As a pilot, I fly Glastar N97BM and high performance gliders in the northern California and Nevada area. I'm also the original author of "High Performance MySQL" published by O'Reilly Media. I still speak at conferences and user groups on occasion.
This entry was posted in programming. Bookmark the permalink.

One Response to Debugging is all about Challenging Assumptions

  1. Joe says:

    It is not just your assumptions, but also those of the author of the code. I’m debugging some software for image processing (a portion of AviStack). My problem is that I know the process flow too well and have written the code to follow my work habits. When I share this code, they quickly find some obvious “bugs” or unexpected behavior because they go about their work with a completely different flow. My app is a GUI on top of objects. I learned a long time ago to make the event handlers behind the GUI very atomic so that they only manipulated the state of the object and kept their required knowledge of the rest of the code to an absolute minimum. Nevertheless, complicated requirements often result in complicated code especially as the requirements and design evolve (creep). Assumptions at the time of basic design are the hardest to overcome.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s