Long time no blog. With the Rim Fire raging up here, I’ve been active on Facebook and Twitter, though.
We shot some pictures of the fire fighting aircraft this evening from the Pine Mountain Lake Marina before dinner.
I’m in the process of rebuilding full-text indexes for a good sized document collection that lives in a sharded MongoDB cluster. And the funny thing about this is that I don’t really use MongoDB that much. I mean we put data into it day after day, but I don’t personally have to interact with it that often. For this particular use case it “just works” the vast majority of the time I don’t have to think about it.
I like that.
But this particular task involves slurping ALL the data out of that cluster and onto a cluster of sharded Sphinx servers so I can re-index the roughly 3 billion documents. That’s all well and good, but since our MongoDB cluster isn’t terribly performance sensitive, it is built on old-fashioned (am I allowed to use that phrase?) spinning disks. And you know what that means, right?
Yeah, seek time matters. A lot.
If this was hitting our production MySQL clusters, I wouldn’t care nearly as much. Those all use one flavor or another of flash stoarge. In fact, we’ve been using SSDs long enough and in enough places that I’m spoiled at this point. I sort of cringe every time I have to deal with disk seeks. That’s so five years ago.
Anyway, I knew this would be an issue so I tried to be clever. I dumped all the document IDs from Mongo in advance, doing so in a way that give them to me in “disk order” so that when I later had to fetch them for indexing, I’d be able to minimize the seeking and hopefully maximize the throughput.
Well, that plan kind of half worked. You see, I had made the assumption that “disk order” on one member of a replica set would be the same as “disk order” on another member of the set. That appears not to be the case. So I had to work around this by telling the indexer processes not to use the mongos routing server, instead talking directly to the mongod on the specific server(s) that I fetched the ids from originally.
I look forward to a few more years from now, when we really do view spinning disks as “the new tape” and use them mainly for archival tasks.
Hey there, I decided to try to catch up with you and I just got back from him. So I got it yet. My My Sweet diarrhea certainly help. I can tell you a little more about it later and got a couple of things to touch base with you on. I know you’re kind of busy concert in the bay area today, but if you get. If we could chat. So anyways. Things are fine here. Just leave me. Cats and what they Hi Sweetie. So, yeah. On the process coming along so I guess. Later on out. I’ll straining. But anyways, I will hopefully catch up with you sometime soon. Okay bye.
I guess the lesson here is that adding “My Sweet diarrhea” to just about anything would make it funny.
(This was all prompted by Facebook posting about a friend’s “best voicemail transcript ever”, though hers was from Vonage.)
There has been some interesting discussion on-line recently about how to handle database (meaning MySQL, but really it applies to other systems too) failover. The discussion that I’ve followed so far, in order, is:
As Rick James (from Yahoo) notes in the comments on Baron’s posting, they take the same approach that I still advocate and which we use at Craigslist: no automated failover. Get a human involved. But try to make it as easy for that human to do two very important things:
It’s that simple.
Peter’s posting gets at the heart of the matter for me. While it’d be fun (and scary) to try and build a great automated system to detect failures and Do The Right Thing, it’s also a really hard problem to solve. There are lots of little gotchas and if you get it wrong, the amount of pain you can bring is potentially enormous.
At Craigslist, we share some similarities with Yahoo. We own our own hardware and it is installed in space that we manage. We try to select good hardware and take good care of it. And things still fail (of course). But the failures are not so frequent that we’re constantly worried about the next MySQL master that’s going to die in the middle of the night.
Rick pointed at MHA in his comment. I need to have a look at it and/or point some of my coworkers at it. I didn’t realize it existed and spent a couple weeks creating a custom tool to help with #1 above. In the event of a master failure, it looks at all available slaves, finds the most suitable candidates, presents a list, and allows the operator to choose a new master. Once selected, the script then tries to automate as much of the switching as possible.
Though I’ve stared at the code quite a bit and tried to reason about the ways it might fail, and I feel pretty good about it, we’ve never actually used it. And that’s OK, really. We have a nicely documented playbook of what to do in that sort of situation already. It has served us well. And, as I said, it doesn’t happen that often. All the script does it try to automate existing practice so that we can turn 10-20 minutes of “read-only” time into less than 5 minutes.
There’s a point at which you start to wonder if that savings is worth the risk of a tricky to spot bug finding its way in and turning 20 minutes into many hours of late night pain. I’m not sure where I stand on that in this particular case. Something like Galera Cluster for MySQL is interesting too, but I kinda feel like it pays not to be an early adopter here too. If we had a lot of problems with master failures, I’d surely feel differently.
Ten years from now a “college education” is going to look radically different from when I went to school. And I think that’s a good thing, especially when you consider the skyrocketing costs of “higher education” and the miserable job market that recent graduates have faced.
This all started for me when I first saw MIT’s Open Courseware and then when Standford offered a few Computer Science courses on-line. I had actually signed up for Andrew Ng’s Machine Learning class but never made time in my schedule to participate. Since then, Andrew and Daphne Koller have kicked things up a notch by starting Coursera. They’ve built a platform that allows instructors to distribute their courses to many, many people on-line at a very low cost.
If you haven’t seen it, take a minute and browse the list of courses. There are 124 at the time of this writing, and that’s up from just a few weeks ago. I’ve already signed up for several (check my Coursera Profile), one of which starts tomorrow.
By figuring out how to make great instruction available to literally millions of people worldwide every year, and solving some of the harder problems associated with class sizes that are a factor of 100 more than what most instructors are using to handling (even with Teaching Assistants), Coursera is on to something–something potentially quite big.
I think the institution of college is about to undergo some very interesting changes. Few of us are able to predict the final outcome, but it’s going to be very interesting to watch–and maybe even more interesting to actually participate! Both Kathleen and I have signed up for some classes. I’m really looking forward to expanding my Computer Science and Programming horizons a bit and trying out a new style of learning and participation.
A few weeks ago I got a new iPad and have generally been quite happy with it. When paired with a wireless keyboard, I can even do some basic (lightweight) “work” on it as a remote terminal. But there’s a rather surprising issue that both Kathleen and I have encountered: file uploads from the browser (Safari) are simply not supported.
So if you shoot a picture with the camera, you can’t hop on to your WordPress dashboard, upload the image, and blog about it. At first I thought it was just me (or us) but a bit of searching reveals that this is, in fact, a “feature” of sorts.
One suggestion I’ve seen is to use a third-party browser, such as iCab. While I’m not opposed to paying a few bucks for something that actually works, the comments on that app make me think it’s not quite the solution either.
I guess you could argue that I “just” need an app that uploads to whatever site it is that I’m using at the time, but that’s pretty unrealistic. I’m a bit puzzled why Apple doesn’t let you browse your media libraries (photos and video, at least) to upload to web sites from mobile Safari. It stikes me as a very, very common need.
No every web site is “big” enough to be able to afford developing their own app for the iPhone and iPad.
So this makes me wonder if I’m just missing something. Is there a reasonably well known workflow that accomplishes what I want? If you’re using an iPhone or iPad, how do you handle this?
In the last year or so, I’ve been working to reduce my computer inventory so that I spend less time administering things and more time actually using them. As part of that, Kathleen and I recently got new iPads (the retina display is amazing).
After a bit of tinkering around, it occurred to me to try pairing it with the Apple Wireless Keyboard. It’s clearly not the same as a laptop but it’s definitely a replacement for a netbook or low-end notebook. Kathleen took to it right away, so now we each have one.
It’s going to be interesting to see how much I’m able to do with an iPad, wireless keyboard, and decent Internet access.
I’m happy to see that the vacuum created by O’Reilly Media dropping the annual MySQL Conference has been filled by the good people at Percona in the form of the Percona MySQL Conference, which is in Santa Clara from April 10-12th.
I’ll be speaking at 3:00pm on the 12th. My talk is titled Living with SQL and NoSQL at craigslist: a Pragmatic Approach. The abstract for the talk is:
craigslist uses a variety of data storage systems in its backend systems: in-memory, SQL, and NoSQL. This talk is an overview of how craigslist works with a focus on the data storage and management choices that were made in each of its major subsystems. These include MySQL, memcached, Redis, MongoDB, Sphinx, and the filesystem. Special attention will be paid to the benefits and tradeoffs associated with choosing from the various popular data storage systems, including long-term viability, support, and ease of integration.
If you’re interested in learning more about our data systems at craigslist, drop by.
Then, on the following day (April 13th), the folks behind Sphinx, Drizzle, and SkySQL are putting on a one-day mini-conference that features speakers on a variety of interesting topics. I’ll be speaking as part of Sphinx Search Day 2012. I’m looking forward to meeting Ivinco there as well.
I don’t yet have an abstract for my Sphinx talk, but it’ll probably be similar to a brown bag talk I recently gave at craigslist. It was an overview of how we’re using a mix “traditional” batch-updating Sphinx and Real-Time Sphinx to power some new features.
Hope to see you there!
From the comments of Never Make Counter Offers on Bram Cohen’s weblog,
…Money is really like a chronic disease, my financial situation is something that intrudes on what I want to do, and prevents me from doing it unless I keep it treated. Either the work is interesting, or it isn’t what I want to do with my life. (You turn 40 at the same time regardless of what you’ve been doing, the question “was it worth it” is not a dollar value question.)
I suspect all real geeks are like this. Being paid very well to sit in a cubicle with nothing to do (been there, quit that) is low-grade torture.
I recently came across the Meet Magit video which is an excellent introduction to the magit emacs mode for git. I use both Emacs and Git a lot at work and at home, so having something a bit more comprehensive and useful than just the standard version control integration that you get with Emacs by default sounded good.
But watching that video, I realized that no only it magit completely awesome, I actually learned a fair amount about git in the process. I’d almost say it’s worth watching even if you have no interest in magit but do feel like there’s more you could know about git.
Having seen the video, I find that I most often refer to the Magit Cheatsheet and the Magit User Manual. If you’re an emacs and git user, you probably owe it to yourself to spend 15-20 minutes watching the video and trying out magit. I think you’ll quickly realize how useful it is.