Sharding is Hard

Reading the Foursqure/MongoDB post-mortem I’m struck by a few things. First, the folks at 10gen are doing a great job of being very open and upfront about what happened and how they hope to solve this problem so that it doesn’t happen again. As someone currently working on a MongoDB deployment myself, I can tell you that a team like that is worth its weight in gold.

Secondly, I’m reminded of the issues that Reddit had a while back with Cassandra. In both cases, servers were arguably under provisioned and running close to limits. And when it came time to add more servers to alleviate the load, the process actually made things worse (because of data migration happening) and didn’t offer immediate relief (because the original servers needed to be compacted).

I suspect that we’ll see this happen more often as time goes on. And there will be high-profile cases. They may or may not be with Cassandra or MongoDB, but the strategies they use are likely also being used by other sharding-friendly NoSQL database engines.

There is no single solution. This is partly a human issue related to planning and monitoring. You really have to stay on top of your growth rates. And it’s also a deficiency in software systems. There may be a need for more sophisticated or aggressive data migration and compaction routines.

All this goes to show that the folks who like to say “just shard your data and you’ll scale forever” are really over-simplifying the problem. The scary part is that I’m not sure many of them realize how much they’re glossing over.

Sharding is hard.

About Jeremy Zawodny

I'm a software engineer and pilot. I work at craigslist by day, hacking on various bits of back-end software and data systems. As a pilot, I fly a Flight Design CTSW and high performance gliders in the northern California and Nevada area. I'm also the original author of "High Performance MySQL" published by O'Reilly Media. I still speak at conferences and user groups on occasion.
This entry was posted in mongodb, nosql, programming, tech. Bookmark the permalink.

One Response to Sharding is Hard

  1. Razi Sharir says:

    I have to admit that I have not yet seen a “sharding-friendly” technique or methodology and will love to get pointers to one.
    Touching the application layer each time a change is required might not qualified for the term “friendly”. It could be related to as tedious, not inclusive or even as risky, let alone effective or efficient…
    Sharding is a great way to deal with inherent traditional SQL DB inability to scale out. Having said that, it will provide for scaling this particular mid tier and address the symptom, it will not address the bottom underlying problem.
    Using NoSQL is a great solution provided you know what you’re getting in return and the limitations imposed.

    Razi Sharir (blog.xeround.com)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s