I recently migrated my Tor relay from a Fedora 32 container to a NixOS container. For some reason, it magically stared using a lot less CPU, but also a bit more memory.

Today's weird thing that I'm too afraid to investigate: moved my Tor relay container from Fedora 32 to NixOS. Same configuration, only went from v0.4.5.8 to v0.4.5.9 in the process.

Now it's using 1/5 of the CPU, a bit more RAM and it yields ~the same bandwidth 🤔🧐 pic.twitter.com/kWy4NyOcaI

— Alekos Filini (@afilini) August 28, 2021

[archived]

After a few days it was clear that it was struggling with the memory, I could see a few big drops in memory usage where the kernel swapped out a lot of pages.

In the official docs¹ docs they recommend recompiling with a different malloc implementation (which they also use in the official release tarballs), but being lazy I didn’t really want to spend time configuring NixOS to compile the package locally with non-standard flags. So I decided to try the quick route first: changing the malloc implementation at runtime.

NixOS supports a few different memory allocators². Most of them are designed to aid development, by trying to help finding undefined behaviors in the usage of allocated memory. There is one provider that can help us, though: jemalloc.

jemalloc³ is designed to reduce memory fragmentation, which is very likely the cause of the high memory usage of my relay, since it doesn’t seem to be connected to the bandwidth provided but just slowly grows over time.

So, I tried adding environment.memoryAllocator.provider = "jemalloc" to my configuration, restarting Tor, and…