We’ve been running a version of Redis 2.2 RC at Craigslist for a few months now and it has been flawless. It has rapidly become the backbone of one of our internal systems.
When I upgraded from the 2.0 series and started testing 2.2, I made a few changes to our configuration. The meat of the config file looks like this:
appendonly no appendfsync no rdbcompression yes maxmemory 6gb maxmemory-policy volatile-lru maxmemory-samples 3 glueoutputbuf yes hash-max-zipmap-entries 512 hash-max-zipmap-value 512 activerehashing yes vm-enabled no
We’re running 4 instances on each of our servers. Two are masters and two are slaves of instances on another server. So our 10 machine cluster has 40 redis-server instances, half of which are masters and half of which are slaves.
The “maxmemory” and “maxmemory-policy” directives are fairly new (part of 2.2) and allow us to make sure that a single redis-server instance doesn’t go over 6GB. Since there are 4 per box, that means redis in total should never use more than 24GB (even though the server has 32GB). We don’t use BGSAVE, so the only time a redis-server should fork and trigger any serious copy-on-write (COW) behavior is when a new slave is attaching to a master.
Now here’s the interesting bit. All the servers are configured identically. But it turns out that the master doesn’t synthesize a delete event into the replication stream when it evicts (deletes) a key due to reaching maxmemory and having to use the LRU policy we’ve chosen. This is different than the normal expire/ttl mechanism that applies to “volatile” keys in redis. In that case, the master controls which keys are removed and sends a delete to each slave when that happens.
The result is that the master and slave are no longer identical when you hit maxmemory and doing LRU-based deletes. And the longer this runs, the more they can drift apart. This is a little surprising.
The discussion going on now revolves around whether or not this is the right default behavior. I’m a bit on the fence but leaning toward “keep the master and slave consistent” because I’m a fan of the principle of least surprise.
If this matters to you, now is a good time to speak up.