Seeking Sucks: Spoiled by SSDs

Posted on April 4, 2013 by

I’m in the process of rebuilding full-text indexes for a good sized document collection that lives in a sharded MongoDB cluster. And the funny thing about this is that I don’t really use MongoDB that much. I mean we put data into it day after day, but I don’t personally have to interact with it that often. For this particular use case it “just works” the vast majority of the time I don’t have to think about it.

I like that.

But this particular task involves slurping ALL the data out of that cluster and onto a cluster of sharded Sphinx servers so I can re-index the roughly 3 billion documents. That’s all well and good, but since our MongoDB cluster isn’t terribly performance sensitive, it is built on old-fashioned (am I allowed to use that phrase?) spinning disks. And you know what that means, right?

Yeah, seek time matters. A lot.

If this was hitting our production MySQL clusters, I wouldn’t care nearly as much. Those all use one flavor or another of flash stoarge. In fact, we’ve been using SSDs long enough and in enough places that I’m spoiled at this point. I sort of cringe every time I have to deal with disk seeks. That’s so five years ago.

Anyway, I knew this would be an issue so I tried to be clever. I dumped all the document IDs from Mongo in advance, doing so in a way that give them to me in “disk order” so that when I later had to fetch them for indexing, I’d be able to minimize the seeking and hopefully maximize the throughput.

Well, that plan kind of half worked. You see, I had made the assumption that “disk order” on one member of a replica set would be the same as “disk order” on another member of the set. That appears not to be the case. So I had to work around this by telling the indexer processes not to use the mongos routing server, instead talking directly to the mongod on the specific server(s) that I fetched the ids from originally.

I look forward to a few more years from now, when we really do view spinning disks as “the new tape” and use them mainly for archival tasks.

Posted in craigslist, nosql, tech, mongodb, sphinx | 5 Comments

When Google Voice Transcription Goes Wrong…

Posted on October 9, 2012 by

Hilarity ensues…

Hey there, I decided to try to catch up with you and I just got back from him. So I got it yet. My My Sweet diarrhea certainly help. I can tell you a little more about it later and got a couple of things to touch base with you on. I know you’re kind of busy concert in the bay area today, but if you get. If we could chat. So anyways. Things are fine here. Just leave me. Cats and what they Hi Sweetie. So, yeah. On the process coming along so I guess. Later on out. I’ll straining. But anyways, I will hopefully catch up with you sometime soon. Okay bye.

I guess the lesson here is that adding “My Sweet diarrhea” to just about anything would make it funny.

(This was all prompted by Facebook posting about a friend’s “best voicemail transcript ever”, though hers was from Vonage.)

Posted in fun, wtf | Leave a comment

Handling Database Failover at Craigslist

Posted on September 18, 2012 by

There has been some interesting discussion on-line recently about how to handle database (meaning MySQL, but really it applies to other systems too) failover. The discussion that I’ve followed so far, in order, is:

As Rick James (from Yahoo) notes in the comments on Baron’s posting, they take the same approach that I still advocate and which we use at Craigslist: no automated failover. Get a human involved. But try to make it as easy for that human to do two very important things:

  1. Get a clear picture of the state of things
  2. Put things in motion once a choice has been made

It’s that simple.

Peter’s posting gets at the heart of the matter for me. While it’d be fun (and scary) to try and build a great automated system to detect failures and Do The Right Thing, it’s also a really hard problem to solve. There are lots of little gotchas and if you get it wrong, the amount of pain you can bring is potentially enormous.

At Craigslist, we share some similarities with Yahoo. We own our own hardware and it is installed in space that we manage. We try to select good hardware and take good care of it. And things still fail (of course). But the failures are not so frequent that we’re constantly worried about the next MySQL master that’s going to die in the middle of the night.

Rick pointed at MHA in his comment. I need to have a look at it and/or point some of my coworkers at it. I didn’t realize it existed and spent a couple weeks creating a custom tool to help with #1 above. In the event of a master failure, it looks at all available slaves, finds the most suitable candidates, presents a list, and allows the operator to choose a new master. Once selected, the script then tries to automate as much of the switching as possible.

Though I’ve stared at the code quite a bit and tried to reason about the ways it might fail, and I feel pretty good about it, we’ve never actually used it. And that’s OK, really. We have a nicely documented playbook of what to do in that sort of situation already. It has served us well. And, as I said, it doesn’t happen that often. All the script does it try to automate existing practice so that we can turn 10-20 minutes of “read-only” time into less than 5 minutes.

There’s a point at which you start to wonder if that savings is worth the risk of a tricky to spot bug finding its way in and turning 20 minutes into many hours of late night pain. I’m not sure where I stand on that in this particular case. Something like Galera Cluster for MySQL is interesting too, but I kinda feel like it pays not to be an early adopter here too. If we had a lot of problems with master failures, I’d surely feel differently.

Posted in craigslist, mysql, tech, yahoo | 9 Comments

Coursera on-line classes and the future of learning…

Posted on September 17, 2012 by

Ten years from now a “college education” is going to look radically different from when I went to school. And I think that’s a good thing, especially when you consider the skyrocketing costs of “higher education” and the miserable job market that recent graduates have faced.

This all started for me when I first saw MIT’s Open Courseware and then when Standford offered a few Computer Science courses on-line. I had actually signed up for Andrew Ng’s Machine Learning class but never made time in my schedule to participate. Since then, Andrew and Daphne Koller have kicked things up a notch by starting Coursera. They’ve built a platform that allows instructors to distribute their courses to many, many people on-line at a very low cost.

If you haven’t seen it, take a minute and browse the list of courses. There are 124 at the time of this writing, and that’s up from just a few weeks ago. I’ve already signed up for several (check my Coursera Profile), one of which starts tomorrow.

By figuring out how to make great instruction available to literally millions of people worldwide every year, and solving some of the harder problems associated with class sizes that are a factor of 100 more than what most instructors are using to handling (even with Teaching Assistants), Coursera is on to something–something potentially quite big.

I think the institution of college is about to undergo some very interesting changes. Few of us are able to predict the final outcome, but it’s going to be very interesting to watch–and maybe even more interesting to actually participate! Both Kathleen and I have signed up for some classes. I’m really looking forward to expanding my Computer Science and Programming horizons a bit and trying out a new style of learning and participation.

It’s worth watching Daphne’s TED Talk: what we’re learning from on-line education. I found some very surprising (and inspiring) ideas in there. Coursera is a very promising experiment in education.

Posted in Uncategorized | 6 Comments

iPad Annoyance: File Uploads in Safari

Posted on May 22, 2012 by

A few weeks ago I got a new iPad and have generally been quite happy with it. When paired with a wireless keyboard, I can even do some basic (lightweight) “work” on it as a remote terminal. But there’s a rather surprising issue that both Kathleen and I have encountered: file uploads from the browser (Safari) are simply not supported.

So if you shoot a picture with the camera, you can’t hop on to your WordPress dashboard, upload the image, and blog about it. At first I thought it was just me (or us) but a bit of searching reveals that this is, in fact, a “feature” of sorts.

One suggestion I’ve seen is to use a third-party browser, such as iCab. While I’m not opposed to paying a few bucks for something that actually works, the comments on that app make me think it’s not quite the solution either.

I guess you could argue that I “just” need an app that uploads to whatever site it is that I’m using at the time, but that’s pretty unrealistic. I’m a bit puzzled why Apple doesn’t let you browse your media libraries (photos and video, at least) to upload to web sites from mobile Safari. It stikes me as a very, very common need.

No every web site is “big” enough to be able to afford developing their own app for the iPhone and iPad.

So this makes me wonder if I’m just missing something. Is there a reasonably well known workflow that accomplishes what I want? If you’re using an iPhone or iPad, how do you handle this?

Posted in other, wtf | 20 Comments

iPad and Wireless Keyboard

Posted on May 11, 2012 by

In the last year or so, I’ve been working to reduce my computer inventory so that I spend less time administering things and more time actually using them. As part of that, Kathleen and I recently got new iPads (the retina display is amazing).

After a bit of tinkering around, it occurred to me to try pairing it with the Apple Wireless Keyboard. It’s clearly not the same as a laptop but it’s definitely a replacement for a netbook or low-end notebook. Kathleen took to it right away, so now we each have one.

It’s going to be interesting to see how much I’m able to do with an iPad, wireless keyboard, and decent Internet access.

Posted in Uncategorized | 17 Comments

Speaking at Percona MySQL Conference and Sphinx Search Day

Posted on February 25, 2012 by

I’m happy to see that the vacuum created by O’Reilly Media dropping the annual MySQL Conference has been filled by the good people at Percona in the form of the Percona MySQL Conference, which is in Santa Clara from April 10-12th.

I’ll be speaking at 3:00pm on the 12th. My talk is titled Living with SQL and NoSQL at craigslist: a Pragmatic Approach. The abstract for the talk is:

craigslist uses a variety of data storage systems in its backend systems: in-memory, SQL, and NoSQL. This talk is an overview of how craigslist works with a focus on the data storage and management choices that were made in each of its major subsystems. These include MySQL, memcached, Redis, MongoDB, Sphinx, and the filesystem. Special attention will be paid to the benefits and tradeoffs associated with choosing from the various popular data storage systems, including long-term viability, support, and ease of integration.

If you’re interested in learning more about our data systems at craigslist, drop by.

Then, on the following day (April 13th), the folks behind Sphinx, Drizzle, and SkySQL are putting on a one-day mini-conference that features speakers on a variety of interesting topics. I’ll be speaking as part of Sphinx Search Day 2012. I’m looking forward to meeting Ivinco there as well.

I don’t yet have an abstract for my Sphinx talk, but it’ll probably be similar to a brown bag talk I recently gave at craigslist. It was an overview of how we’re using a mix “traditional” batch-updating Sphinx and Real-Time Sphinx to power some new features.

Hope to see you there!

Posted in Uncategorized | 1 Comment

Money vs. Interesting Work

From the comments of Never Make Counter Offers on Bram Cohen’s weblog,

…Money is really like a chronic disease, my financial situation is something that intrudes on what I want to do, and prevents me from doing it unless I keep it treated. Either the work is interesting, or it isn’t what I want to do with my life. (You turn 40 at the same time regardless of what you’ve been doing, the question “was it worth it” is not a dollar value question.)

I suspect all real geeks are like this. Being paid very well to sit in a cubicle with nothing to do (been there, quit that) is low-grade torture.

– Rob Landley

Aside | Posted on February 4, 2012 by | 4 Comments

magit makes git awesome in emacs

Posted on February 2, 2012 by

I recently came across the Meet Magit video which is an excellent introduction to the magit emacs mode for git. I use both Emacs and Git a lot at work and at home, so having something a bit more comprehensive and useful than just the standard version control integration that you get with Emacs by default sounded good.

But watching that video, I realized that no only it magit completely awesome, I actually learned a fair amount about git in the process. I’d almost say it’s worth watching even if you have no interest in magit but do feel like there’s more you could know about git.

Having seen the video, I find that I most often refer to the Magit Cheatsheet and the Magit User Manual. If you’re an emacs and git user, you probably owe it to yourself to spend 15-20 minutes watching the video and trying out magit. I think you’ll quickly realize how useful it is.

Posted in craigslist, programming | Leave a comment

craigslist Charitable Fund Donates to the Perl Foundation

Posted on January 24, 2012 by

I’m very happy to see the announcement on the Perl Foundations’s home page this morning.

According to craigslist CEO and former Perl Hacker Jim Buckmaster, “craigslist has gloried in and relied upon Perl for most of its software development for more than a decade. craigslist Charitable Fund is honored to recognize the wizardry and generosity of the Perl community, help ensure the ongoing maintenance of Perl5, and contribute to the future evolution of Perl.” He added, “It was unclear at first how best to give something back to Perl. Fortunately there was more than one way to do it.”

And I’m very happy that craigslist (my employer) recognizes the value that Perl has provided over the years in all the work we do at craigslist. I’m sure the $100,000 will go a lon way toward helping with ongoing Perl development and maintenance this year.

Here’s to more great stuff from Perl in 2012!

Discussion also on Hacker News.

Oh, by the way. We’re hiring great Perl hackers at craigslist. Let me know if you’re interested. :-)

Posted in craigslist, perl | 1 Comment