Observing Python in production
• Mark Eschbach
So a Python application I’m responsible for is like the cookie monst of memory. Whenever I give it more memory it just chews through it and keeps going. I’m a bit concerned because the application has reasonable amount of memory. I’m hoping this is really about tuning and I can just move a few knobs to encourage more efficient memory usage. Let’s see.
According to theano Python never releases memory to the operating system and will retain small object lists indefinitely. So that isn’t encouraging. This site might provide more insight (removed because it’s a duplicate). Let me just start with I really dislike fixed column width websites at this point. Even when I’m zooming in it’s got issues. For the massive amount of text it is, on my display it shows rather small. And it looks like a copy of the other article.
At this point I’m wondering if tuning Gunicorn to cycle workers at a more agressive rate might be a better approach. I’ve got to admit I’m a bit disappointed in many dynamic languages never returning memory back to the operating system. Especially with automatic memory management, I would hope the memory management system would go through a compaction phase to free up whole pages at a time. My true hypothesis for CPython’s apporach is the simplicity of relying on the stdlibc memory management friends, which I don’t think return pages back to the operating system. Just a guess and I’m not going to check at this point.
Changing tact to looking at mitigating the problem
So the real issue is the Django workers are consuming enough memory for me to be concerned they will randomly die off. A realistic answer would be to severly limit the number of transactions to be serviced before the worker is restarted. This would effectively allow the application to return some of the memory back to the operating system. In theory we could also tune the pools to be large enough to support all use cases on the application but that would require more memory than I would feel comfortable with. I’ll also add the prefork and see how the application responds. I still haven’t figured out a good cap for the number of trasactions though.
An issue has crept up with old iOS clients and API drift again. Whenever I encounter this my default response is just force an upgrade. A benefit of working on a SaaS platform is we can push changes when we need to. Modern mobile appilcation development is setup to automatically push our updates to the clients. This significantly reduces the overhad of dealing with API drift. Additionally our entire release and engineering systems are setup to only support the last version, currently released version, and the next version of our application. Just irks me when they claim we need to support clients which are months old because someone hasn’t updated.