Redis using 2GB of memory on 70MB data set - the fix
For Ariejan.net I use redis to cache pages and shards. This works great and all, but today I noticed something alarming:
redis Running 3d 12h 4m 0.0% 45.2% [1829556 kB]
Yes, that’s about 1.7 GB of RAM. That’s way too much for what I cache. Let’s see what redis has to say for itself:
$ redis-cli
redis 127.0.0.1:6379> info
redis_version:2.2.12
...
connected_clients:5
connected_slaves:0
used_memory:71626608
used_memory_human:68.31M
used_memory_rss:1873465344
mem_fragmentation_ratio:26.16
Well, that’s awkward. The OS is reporting 1.7GB memory usage, while Redis claims to store a mere 68MB. What’s happening here!
You may have noticed that I included mem_fragmentation_ratio in the snippet above as well. It’s at a whopping 26.16, meaning that for every byte I store, 26.16 bytes of memory are used. This explains the 1.7GB memory usage.
But, how do I get rid of this? My system has enough RAM to cope redis as is, but it’s not a comforting thought to leave Redis running like this.
As it turns out, there isn’t a lot you can do about this. Redis 2.2 uses malloc which causes the fragmentation. One alternative is to add a slave redis-server, migrate your data and then switch the slave to master. Although this is reported to work well, it’s not a good solution to the problem.
Fortunately, Redis 2.4 on Linux by default does not use malloc anymore. Instead it uses jemalloc. From the redis README:
Redis is compiled and linked against libc malloc by default, with the exception of jemalloc being the default on Linux systems. This default was picked because jemalloc has proven to have fewer fragmentation problems than libc malloc.
The only logical step to take is to upgrade to Redis 2.4.
If you’re already running (or just upgraded to) 2.4 you can easily check if your redis is using jemalloc:
$ redis-cli
redis 127.0.0.1:6379> info
redis_version:2.4.9
mem_fragmentation_ratio:1.11
mem_allocator:jemalloc-2.2.5
I’ve taken this step and the mem_fragmentation_ratio samples I’ve measured have all been in the 1.1-1.4 regions.
