Congrats to RethinkDB

Congratulations to the RethinkDB team on their fundraising round!

2010 will be an incredibly exciting year for database technology. The amount of stored information has long been increasing exponentially, presenting unprecedented challenges for modern technology companies. Dropping RAM prices and affordable multicore CPUs have drastically changed the hardware profile of a typical server machine from what it was only five years ago. The emergence of solid-state drives is about to revolutionize database systems, enabling new kinds of applications we can barely envision today.

Indeed. We’re seeing some pretty amazing results at work where we’re in the process of replacing a large number of traditional RAID-10 arrays of power hungry and [comparatively] slow spinning disks with solid-state storage. The performance gains are hard to imagine if you haven’t seen them for yourself first-hand.

I was excited and honored to become an advisor to RethinkDB last year. The team has a very real chance to build something revolutionary at a time when a major shift of our computing infrastructure is working its way around the industry.

These are very interesting times for anyone building data intensive back-end services on-line.

Posted in programming | Leave a comment

Student Pilots: Beware of Sacramento Executive Airport

About three weeks ago, my wife Kathleen flew into Sacramento Executive Aiport (KSAC) as part of her solo cross-country flight. (That’s one of the final stages a student pilot goes through on the way to getting a Private Pilot Single Engine Land (PP-SEL) rating.)  Her flight went from Reid Hillview Airport (KRHV) in San Jose to Sacramento and then on to Los Banos (KLSN) before heading back home to San Jose.

She called me after each leg of the flight to let me know how things were progressing. And when she called from Los Banos she mentioned that the ground controller in Sacramento wasn’t particularly helpful. Even though he knew she was a student pilot and not terribly familiar with the airport, he never really offered any taxi help. (Contrast that with our recent landing in Stockton where the controller asked us if we were familiar with the airport AND offered progressive taxi instructions even before we finished turning from the runway onto the taxiway.) Instead of being helpful, he sarcastically made a comment intended to keep her from taxiing onto the active runway–something she wasn’t in danger of doing in the first place.

I should back up and mention that when she was in the transient parking area and preparing to taxi to the runway for her departure, there was another airplane behind her. Being less than familiar with the runway and taxiway layout (see the airport diagram), she offered to let the aircraft behind her go ahead so the should follow them out (they were locals). But the ground controller wasn’t interested in facilitating that.

In the grand scheme of things, this would be no big deal. You encounter “less than helpful” people now and then in aviation. She flew on from Sacrament to Los Banos, stopped there for a bit, and continued on back to San Jose. About an hour after landing, she was back home and we were headed out to dinner to celebrate a successful solo cross-country flight.  Well, after seeing the fancy “Congratulations” balloon, cupcakes, and card that were waiting on the kitchen table–then we headed out to Mama Mia’s.

Fast forward a few weeks… to last night.  She received an email from her instructor saying that he’d been contacted by the Sacramento FSDO (Flight Standards District Office) who had received some negative feedback about her visit there a few weeks ago. Puzzled about what that could be, she talked to her instructor and the Sacramento FSDO supervisor today to discuss things. (I also spoke with the FSDO today.)

To make a long story short, here’s what happened. Sitting in the right seat of the airplane taxiing behind her at Sacramento was a Designated Pilot Examiner (DPE), one of the anointed pilots that the FAA has deemed worthy of judging whether or not a student pilot is ready to receive their new rating. After taxiing behind her, he became concerned that she might not have received all training she needed to operate at a tower controlled airport. So he took it upon himself to file a complaint, though we’re told he originally classified it as an “incident” (which has special meaning in the FAA lexicon) with the local officials: the Sacramento FSDO.

Now here’s where it gets really good. What apparently alarmed him is that when she had to taxi across a runway on her way to the active departure runway, she slowed down to look both ways (much like we’re taught to do when crossing the street as children). Why might she do such a thing? Well, let’s see. She was in an airplane accident not to long ago. So safety is nothing to take lightly. (Would you?!) And it’s not unheard of for ground controllers to occasionally, you know, make a mistake. They are human after all. So why not proceed with a bit of caution? Besides, most instructors rightfully teach their students to make one last look before entering a runway, active or otherwise.

So in return for her caution, she had the pleasure of staying up half the night last night worried about what she could have possibly done so wrong that the FAA wanted to talk to her. And then she got to listen to an FAA representative make her feel like a bad pilot and suggest that she had an attitude problem with the ground controller on the radio. The best part of all is that she’s not even the subject of the investigation!

All this hassle, concern, and drama because she was being cautious at an airport she’d only been to once before. You’d think with the FAA’s focus on preventing runway incursions, they’d appreciate seeing pilots taking a few extra seconds before charing across a runway.

So, student pilots out there: beware of Sacramento Executive Airport. There’s clearly at least one less than helpful controller with an attitude, and if you’re really unlucky, you might also encounter an examiner who enjoys telling his friends at the local FSDO when a visiting pilot is behaving in a way that’re more cautious than he’s used to seeing.

Oh, we do plan to make a Freedom of Information Act Request to get a copy of the audio tapes (all controller radio communication is recorded in case it’s needed in a later investigation). Given that all this “helpful” activity is funded by our tax money in the first place, it’d be good to have a copy. Maybe we’ll even put it online. Believe it or not, they can’t allow her to hear the tapes–even though she’s on them–while there’s an investigation in progress.

Yay for government “process” protecting us all.

Posted in flying | 56 Comments

NoSQL is Software Darwinism

In I Can’t Wait for NoSQL to Die, Ted Dziuba (who’s always good for a fun rant that includes a dose of reality) tries to make the point that real businesses use relational databases and that your startup isn’t Google. Well done, Ted. I laughed to myself a few times reading your post. And I agree–to a point.

I see this “NoSQL movement” (as it’s called) as a bit of Software Darwinism that’s playing out before our eyes. There are a lot of Key/Value stores and distributed schema-free Document Oriented Databases out there. They’re springing up like weeds in a spring garden. And folks love to blog about them and/or talk about how their favorite is better than the others (or MySQL).

That’s all well and good. It’s how this stuff works. But if you look a couple years down the road, there’s a world where Oracle, MySQL, and PostgreSQL are still around and serve as the source of truth for a lot of really important data–especially in OLTP systems. But part of that reality also includes newer systems like Redis, Cassandra, CouchDBVoldemort, and Hadoop.

And don’t think that the MySQL universe won’t have evolved by then.  Look no further than XtraDB or InnoDB plug-in, PBXT, and Drizzle. Meanwhile, if you’re not into all the hype and/or spending your time finding all the bugs, that’s okay. Grab some popcorn and watch the show. The end result will be better software and more choices for solving problems.

See Also: Eric Day’s Thoughts on NoSQL

Posted in programming | 25 Comments

Best Oatmeal Recipe

Last year I discovered the joys of slow cooked steel cut oats. Ever since trying them, I’ve never been able to buy pre-packaged, processed, and flavored oatmeal that you see in boxes of single serving packs at the grocery store.

I’ve been making small variations on a basic recipe a couple times a month for about half a year now. And yesterday, with some nice fresh peaches from the store, I made the best batch so far.  I tweeted about it this morning and quickly got a request for the recipe, so here it is.

You’ll need a small crock pot (or slow cooker). We use the Proctor Silex 33015 1.5-Quart Round Slow Cooker bought on Amazon just for this purpose.

  • 2 fresh peaches, cut into 1″ pieces
  • 1/2 – 1 cup of dried cherries
  • 3/4 – 1 cup of steel cut oats
  • 3 1/2 cups water
  • 1/2 cup milk
  • 2 spoons of light brown sugar

Combine all ingredients in the small crock pot and cook overnight (8-9 hours). Enjoy in the morning, optionally topped with freshly chopped banana and/or strawberries and a little bit of brown sugar.

Serves 2-3 adults.

All I can say is YUM! You have no idea how good oatmeal can taste until you’ve tried this. And there are so many variations you can use–different fruits, syrup, and so on.

Posted in cooking | Tagged , | 2 Comments

This is how Open Source works

In Redis weekly update #1 – Hashes and… many more! Salvatore Sanfilippo (the author of Redis) describes the last week’s worth of changes to Redis.  The last item is about a bug I found when setting up cross-datacenter Redis replication at Craigslist.  He describes the bug and the fix quite well, concluding with:

I’m so happy because of this stories, involving five people willing to share their work for free, to push their knowledge forward. This are a few of the best traits of humanity itself. There is really something good about open source, and I’m honored to collaborate with such great guys.

I have to say, this is exactly how Open Source is supposed to work.  He makes the source code available (and easily forkable on github) and actively participates on the #redis channel of the Freenode IRC network.  So even though I’m in California and he’s in Italy, we were able to communicate via email, the project’s issue tracking system (via Google Code), IRC, and a github pull request.

As he notes, it took a bit of time to find this bug, but the discussions we had on-line helped both of us to focus our search.  It was ultimately a single error message in one of the non-failed servers that caught my attention and focused in on the problem he describes in his post.

I can’t imagine what it’d have been like to track this down without source code and communicating with the customer service organization of a larger software vendor.

This morning I patched our code, deployed the fix, and tested it.  So far, so good.  I expect that when Redis 1.2.6 is released in a day or two, we can be assure that nobody else will hit this bug.

Excellent.

Posted in programming | Tagged | 15 Comments

First San Francisco Redis Meetup: March 25th

Thanks to Ted Nyman for organizing The first San Francisco Redis Meetup which will be on March 25th at the Engine Yard offices in SOMA.

Last week I posted an idea to the idea board and it turned into the first meeting’s format:

Based on member feedback, our first Meetup will be discussion-based, centered around a series of 3-5 minute Lightning Talks about Redis. Since a good percentage of our members are currently using Redis, I hope members will volunteer to talk a little about their experiences with Redis.

Sounds like fun.  I’m planning to be there to talk about our recently setup Redis clusters at Craigslist.

Thanks again to Ted and Engine Yard.

Posted in programming | Tagged | Leave a comment

New Airplane: 2005 Flight Design CTSW (N722VJ)

After losing our Citabria (see: Citabria N5156X Landing Gear Accident), we set about looking for a new (well a replacement) airplane.  The main things we were after in the search were the following:

  • 2 seats (more is better, but budget is an issue)
  • efficient fuel burn
  • inexpensive maintenance
  • newer design and materials

The overall cost of ownership was probably the biggest consideration.  And the more we talked about it, the more we gravitated toward Light Sport Airplanes.  They’re newer, fairly inexpensive to buy, and definitely less expensive than traditionally certified airplanes.

Budget dictated that we find a fairly inexpensive used model, so I did a fair amount of hunting around.  In doing so, I learned that most of the cheaper used LSAs are either fairly high-time, used by flight schools, or both.  Those that remained were tens of thousands of dollars above what we were able to spend.

So imagine my surprise when, on my second visit to the Flight Design Forum, I discovered that the price had just been reduced on N722VJ: a 2005 Flight Design CTSW.

Flight Design CTSW Solo

After a bit of phone and email contact with the owner(s), I arranged a time to visit back in January (after our cruise to Mexico) and see the airplane in person.  And, to make a long story short, we took delivery of our new airplane a few weeks ago.

Flight Design CTSW Solo

So far I have about 5.5 hours of flying time in it and am really enjoying it.  It’s fairly quiet, comfortable, easy to fly, and has great visibility.  The most notable difference between the CTSW and other airplanes I’ve flown is the landing technique.  I find that treating it like a high performance glider makes the landings a lot easier.

More pictures of N722VJ are in my Flickr account.

Posted in flying | 12 Comments

Debugging is all about Challenging Assumptions

I’ve been awash in debugging a new code deployment most of the week. The gist of it, if you’re interested, is that we’re deploying a group of machines running several Redis instances (one per CPU core).  Collectively they’ll serve as an alternative to an existing MySQL-based system for tracking a variety of short and medium term data points.

A bit of data (just an identifier) will be fed into the system by some of our front-end web servers and then picked up by another process that will fetch more data, crunch it lightly, and then store summary information back in various Redis buckets.  (Sorted Sets are cool, BTW.)

The trouble came, as it often does, shortly after the deployment when I started seeing errors and slow responses from some of the web servers using the new code.  To troubleshoot,  we rolled back the code on all but one server and I spent the next several days working out what was wrong.

Zip Line Fun in Mexico

I’ve found a bug in the high-level module I wrote, a few bugs in the lower-level module I’ve been using (but did not write), and a bug in the wrapper module that I wrote to sit between the high-level and lower-level module. Some of the bugs were findable by a simple reading of the code and thinking through the symptoms I’ve been seeing. Others required a lot more “debugging with print statements” and making guesses.  Those generally take a lot longer.

The common thread in all of the harder to find bugs, both in this situation and in others I’ve faced, is that my assumptions often get in the way. The more I can step back and challenge the “obvious” things I believe about what should be happening, the faster I can figure out what the heck went wrong.

It gets easier and faster with time and practice, but it’s still frustrating. I think it’s human nature not to question your assumptions about things that should be “obvious”, but this is one of the situations in which that is essential.  Because what you think is happening and what is really happening are often just different enough to make life interesting.

Posted in programming | 1 Comment

Citabria N5156X Landing Gear Accident

Back on December 26th, 2009 my wife Kathleen was flying with an instructor in the Citabria that we owned half of as part of her ongoing flight training toward her Single Engine Land pilot rating.  (She’s already a rated glider pilot.) They were flying out of Reid Hillview (KRHV) in San Jose, our home airport.

Kathleen and N5156X at Lee Vining

They were practicing short-field landings. That means they were coming in slower and more steeply than normal–but not dangerously so. On their second (and soon to be final) landing of the day, the right landing gear snapped off upon touching down.

Citabria N5156X Landing Gear Accident

The picture above was shot the next day after it had been out in the rain. That’s why you see rust.

The result is that the landing gear, still attached by the brake line, swung around and punched a hole in the right rear fuselage before coming off completely.

IMG_6910

The missing landing gear also resulted in a multiple prop strikes (meaning that the propeller hit the runway and bent). That resulted in internal engine damage.

Citabria N5156X Landing Gear Accident

Eventually, the right wing hit the ground and they lost directional control. The airplane ended up in the grass between runway 13L and 13R but they were uninjured–thankfully!

IMG_6894

Our mechanic, who has been working on these airplanes about as long as I’ve been alive, had a look at it and told us he’d never seen one break like that before. The steel itself broke–not the fuselage attach bolts which are the typical “weak link” on a Citabria.

Thankfully the airplane was insured. After seeing the pictures and description of the damage and ballpark estimated repair work, they decided to buy the airplane from us and payout fully on the claim. They called it a total loss.

Citabria N5156X Landing Gear Accident

A few days later, N5156X was hauled away to an aircraft scrap yard where undamaged parts of it will presumably be sold off.

I’m going to miss N5156X. It was the airplane that I did the bulk of my private training in, including my solo cross-country flights.

Flying Citabria N5156X on my First Solo Cross Country Flight

We also flew it out to Jackson Hole and Yellowstone last summer.

Yellowstone Trip Day 1

You can see more of the photos in my Citabria N5156X Landing Gear Accident set on Flickr.

Posted in flying | 20 Comments

Roma for Redis Clusters

The Roma project looks pretty interesting.

According to the README file:
Roma is a set of utilities that help query and maintain a replicatedpartitioned (shared) Redis cluster. It handles start, stop, status,backup, restore, arbitrary queries, and includes a monitoring daemon(romad) which handles fail-over, periodic backups, and establishes the cluster status.
Sounds great.  We’re just starting to deploy a different type of Redis cluster at craigslist and I’m interesting in seeing where Roma and other Redis Proxy ideas go.
Posted in programming | Tagged | 1 Comment