Handling Database Failover at Craigslist

There has been some interesting discussion on-line recently about how to handle database (meaning MySQL, but really it applies to other systems too) failover. The discussion that I’ve followed so far, in order, is:

As Rick James (from Yahoo) notes in the comments on Baron’s posting, they take the same approach that I still advocate and which we use at Craigslist: no automated failover. Get a human involved. But try to make it as easy for that human to do two very important things:

  1. Get a clear picture of the state of things
  2. Put things in motion once a choice has been made

It’s that simple.

Peter’s posting gets at the heart of the matter for me. While it’d be fun (and scary) to try and build a great automated system to detect failures and Do The Right Thing, it’s also a really hard problem to solve. There are lots of little gotchas and if you get it wrong, the amount of pain you can bring is potentially enormous.

At Craigslist, we share some similarities with Yahoo. We own our own hardware and it is installed in space that we manage. We try to select good hardware and take good care of it. And things still fail (of course). But the failures are not so frequent that we’re constantly worried about the next MySQL master that’s going to die in the middle of the night.

Rick pointed at MHA in his comment. I need to have a look at it and/or point some of my coworkers at it. I didn’t realize it existed and spent a couple weeks creating a custom tool to help with #1 above. In the event of a master failure, it looks at all available slaves, finds the most suitable candidates, presents a list, and allows the operator to choose a new master. Once selected, the script then tries to automate as much of the switching as possible.

Though I’ve stared at the code quite a bit and tried to reason about the ways it might fail, and I feel pretty good about it, we’ve never actually used it. And that’s OK, really. We have a nicely documented playbook of what to do in that sort of situation already. It has served us well. And, as I said, it doesn’t happen that often. All the script does it try to automate existing practice so that we can turn 10-20 minutes of “read-only” time into less than 5 minutes.

There’s a point at which you start to wonder if that savings is worth the risk of a tricky to spot bug finding its way in and turning 20 minutes into many hours of late night pain. I’m not sure where I stand on that in this particular case. Something like Galera Cluster for MySQL is interesting too, but I kinda feel like it pays not to be an early adopter here too. If we had a lot of problems with master failures, I’d surely feel differently.

Posted in craigslist, mysql, tech, yahoo | 12 Comments

Coursera on-line classes and the future of learning…

Ten years from now a “college education” is going to look radically different from when I went to school. And I think that’s a good thing, especially when you consider the skyrocketing costs of “higher education” and the miserable job market that recent graduates have faced.

This all started for me when I first saw MIT’s Open Courseware and then when Standford offered a few Computer Science courses on-line. I had actually signed up for Andrew Ng’s Machine Learning class but never made time in my schedule to participate. Since then, Andrew and Daphne Koller have kicked things up a notch by starting Coursera. They’ve built a platform that allows instructors to distribute their courses to many, many people on-line at a very low cost.

If you haven’t seen it, take a minute and browse the list of courses. There are 124 at the time of this writing, and that’s up from just a few weeks ago. I’ve already signed up for several (check my Coursera Profile), one of which starts tomorrow.

By figuring out how to make great instruction available to literally millions of people worldwide every year, and solving some of the harder problems associated with class sizes that are a factor of 100 more than what most instructors are using to handling (even with Teaching Assistants), Coursera is on to something–something potentially quite big.

I think the institution of college is about to undergo some very interesting changes. Few of us are able to predict the final outcome, but it’s going to be very interesting to watch–and maybe even more interesting to actually participate! Both Kathleen and I have signed up for some classes. I’m really looking forward to expanding my Computer Science and Programming horizons a bit and trying out a new style of learning and participation.

It’s worth watching Daphne’s TED Talk: what we’re learning from on-line education. I found some very surprising (and inspiring) ideas in there. Coursera is a very promising experiment in education.

Posted in Uncategorized | 6 Comments

iPad Annoyance: File Uploads in Safari

A few weeks ago I got a new iPad and have generally been quite happy with it. When paired with a wireless keyboard, I can even do some basic (lightweight) “work” on it as a remote terminal. But there’s a rather surprising issue that both Kathleen and I have encountered: file uploads from the browser (Safari) are simply not supported.

So if you shoot a picture with the camera, you can’t hop on to your WordPress dashboard, upload the image, and blog about it. At first I thought it was just me (or us) but a bit of searching reveals that this is, in fact, a “feature” of sorts.

One suggestion I’ve seen is to use a third-party browser, such as iCab. While I’m not opposed to paying a few bucks for something that actually works, the comments on that app make me think it’s not quite the solution either.

I guess you could argue that I “just” need an app that uploads to whatever site it is that I’m using at the time, but that’s pretty unrealistic. I’m a bit puzzled why Apple doesn’t let you browse your media libraries (photos and video, at least) to upload to web sites from mobile Safari. It stikes me as a very, very common need.

No every web site is “big” enough to be able to afford developing their own app for the iPhone and iPad.

So this makes me wonder if I’m just missing something. Is there a reasonably well known workflow that accomplishes what I want? If you’re using an iPhone or iPad, how do you handle this?

Posted in other, wtf | 28 Comments

iPad and Wireless Keyboard

In the last year or so, I’ve been working to reduce my computer inventory so that I spend less time administering things and more time actually using them. As part of that, Kathleen and I recently got new iPads (the retina display is amazing).

After a bit of tinkering around, it occurred to me to try pairing it with the Apple Wireless Keyboard. It’s clearly not the same as a laptop but it’s definitely a replacement for a netbook or low-end notebook. Kathleen took to it right away, so now we each have one.

It’s going to be interesting to see how much I’m able to do with an iPad, wireless keyboard, and decent Internet access.

Posted in Uncategorized | 19 Comments

Speaking at Percona MySQL Conference and Sphinx Search Day

I’m happy to see that the vacuum created by O’Reilly Media dropping the annual MySQL Conference has been filled by the good people at Percona in the form of the Percona MySQL Conference, which is in Santa Clara from April 10-12th.

I’ll be speaking at 3:00pm on the 12th. My talk is titled Living with SQL and NoSQL at craigslist: a Pragmatic Approach. The abstract for the talk is:

craigslist uses a variety of data storage systems in its backend systems: in-memory, SQL, and NoSQL. This talk is an overview of how craigslist works with a focus on the data storage and management choices that were made in each of its major subsystems. These include MySQL, memcached, Redis, MongoDB, Sphinx, and the filesystem. Special attention will be paid to the benefits and tradeoffs associated with choosing from the various popular data storage systems, including long-term viability, support, and ease of integration.

If you’re interested in learning more about our data systems at craigslist, drop by.

Then, on the following day (April 13th), the folks behind Sphinx, Drizzle, and SkySQL are putting on a one-day mini-conference that features speakers on a variety of interesting topics. I’ll be speaking as part of Sphinx Search Day 2012. I’m looking forward to meeting Ivinco there as well.

I don’t yet have an abstract for my Sphinx talk, but it’ll probably be similar to a brown bag talk I recently gave at craigslist. It was an overview of how we’re using a mix “traditional” batch-updating Sphinx and Real-Time Sphinx to power some new features.

Hope to see you there!

Posted in Uncategorized | 16 Comments

Money vs. Interesting Work

From the comments of Never Make Counter Offers on Bram Cohen’s weblog,

…Money is really like a chronic disease, my financial situation is something that intrudes on what I want to do, and prevents me from doing it unless I keep it treated. Either the work is interesting, or it isn’t what I want to do with my life. (You turn 40 at the same time regardless of what you’ve been doing, the question “was it worth it” is not a dollar value question.)

I suspect all real geeks are like this. Being paid very well to sit in a cubicle with nothing to do (been there, quit that) is low-grade torture.

Rob Landley

Aside | Posted on by | 4 Comments

magit makes git awesome in emacs

I recently came across the Meet Magit video which is an excellent introduction to the magit emacs mode for git. I use both Emacs and Git a lot at work and at home, so having something a bit more comprehensive and useful than just the standard version control integration that you get with Emacs by default sounded good.

But watching that video, I realized that no only it magit completely awesome, I actually learned a fair amount about git in the process. I’d almost say it’s worth watching even if you have no interest in magit but do feel like there’s more you could know about git.

Having seen the video, I find that I most often refer to the Magit Cheatsheet and the Magit User Manual. If you’re an emacs and git user, you probably owe it to yourself to spend 15-20 minutes watching the video and trying out magit. I think you’ll quickly realize how useful it is.

Posted in craigslist, programming | Leave a comment

craigslist Charitable Fund Donates to the Perl Foundation

I’m very happy to see the announcement on the Perl Foundations’s home page this morning.

According to craigslist CEO and former Perl Hacker Jim Buckmaster, “craigslist has gloried in and relied upon Perl for most of its software development for more than a decade. craigslist Charitable Fund is honored to recognize the wizardry and generosity of the Perl community, help ensure the ongoing maintenance of Perl5, and contribute to the future evolution of Perl.” He added, “It was unclear at first how best to give something back to Perl. Fortunately there was more than one way to do it.”

And I’m very happy that craigslist (my employer) recognizes the value that Perl has provided over the years in all the work we do at craigslist. I’m sure the $100,000 will go a lon way toward helping with ongoing Perl development and maintenance this year.

Here’s to more great stuff from Perl in 2012!

Discussion also on Hacker News.

Oh, by the way. We’re hiring great Perl hackers at craigslist. Let me know if you’re interested. 🙂

Posted in craigslist, perl | 1 Comment

Visiting Frozen Tenaya Lake

Yesterday Kathleen and I visited Tenaya Lake in the Tioga Pass area of Yosemite National Park. The weather has been odd this season. While it’s been quite cold at the higher elevations, there’s been virtually no precipitation yet, so the lake has frozen over but it is still easily accessible.

Tenaya Lake (frozen)

The ice is easily 8 inches or more think in most places, so lots of folks were out wandering around. She wrote up a posting with more pictures on HowToEatAndLive: Walking on Tenaya Lake.

I had a blast sliding around on the lake! At times like this it really pays to live nearby.

Given the rarity of these conditions, we may not be able to do this again for another 10 years or more. And it’s good that we went when we did. The weather is starting to change–headed for a more seasonal mix of cold and rain/snow.

Lots of pictures, including some really cool ice shots she took, are in my Tenaya Lake album on Picasa.

Posted in fun | 1 Comment

Renewed Weight Loss, Health, and a new blog…

Back in mid-2006, I documented my fairly dramatic weight loss in a series of blog postings. Some of you may even remember that. It was on boingboing and all that.

I went from a high of 224 pounds to a low of 164, which made for a total loss of 60 pounds (I’m about 6 feet tall). It’s interesting (and a little disturbing) to go back and re-read what I wrote back then:

Fast forward to January of 2011, and I had gained back about half the weight. I was hovering just below 200 pounds and had needed to purchase larger jeans a few times during the five years in between. Considering the success rate of most diets, that’s not bad but it’s also hardly ideal. Other issues had crept up along the way too.

The story of how I re-lost most of that weight and generally got a lot healthier is the subject of Learning to Eat Well and Stay Healthy, a blog post I just wrote on a new site called HowToEatAndLive that my wife Kathleen and I have started for 2012. It’s going to document a lot of what we’ve learned about health, nutrition, and living in the last year (and interesting things we continue to discover in the new year). We’ve both learned a lot and have tons to write.

If you’re curious to learn about how I lost the weight I gained back and have kept it off, that’s the place to go. It’s a little rough around the edges yet, but we’re working to add some finishing touches as well as a lot more information.

Posted in health | 53 Comments