Something Emporium

Archived News for June 2009

• Monitoring Per-Process Memory Usage with Munin

  Something Emporium and a few associated websites are run off my VPS server from Rimuhosting. (I used to work for Rimuhosting: they're awesome.) Recently, I moved from running a traditional mod_php/mpm_prefork setup to a multi-threaded mod_fcgid/mpm_worker setup.

  In a mod_fcgid PHP setup, a few persistant php-cgi workers are set up and then requests for dynamic content are farmed out to them. By stripping PHP out of the apache workers, you get a setup that uses far less memory to serve each connection, and you're free to use a multi-threaded MPM (PHP is not thread-safe, so you can't with mod_php).

  OK, so, I've set this up, which is a process with spotty documentation. I've got it working nicely, and I've been able to reduce the stack's memory footprint considerably and increase its performance.

  The snag so far is this:

  

  This graph shows total memory usage on the server, catagorised by type, as we transitioned from mod_php to mod_fcgid. It's a graph provided by Debian's default install of Munin. This VPS has 224MB of memory, so it's something I have to watch fairly closely. You'll notice the sawtooth shape of the graph after the changeover. Here's a closeup of that:

  

  This is pretty good evidence of some sort of memory leak. There's a process there that's allocating memory and can't (or won't) deallocate it. It's being reset at fairly regular intervals, either by monit or perhaps by some other worker handler: apache, mod_fcgid. Actually, it could be almost any process. Damn.

  So, my next task was to identify the culprit. Which comes down to the following question: which processes are using memory in this sawtooth pattern? Sounds like a task for Munin, huh. But out of the box, Debian's Munin doesn't (didn't: try multips_memory from version 1.4.2) have a recipe for graphing the memory usage of individual processes.

  Which lead me to write a Munin "plugin", multimemory. Based off multips, this plugin lets you specify a bunch of process names, and it'll graph the total memory usage of processes matching each one. It's available at GitHub if anyone wants it.

  It gave me the following graph, which, twelve hours later, makes the culprit in the case of the leaking memory pretty clear.

  

  Next steps: wait for 5.3's new garbage collector. Well, probably more likely I'll just jump ship to 5.3 sooner, stable version be damned. That, and reduce the number of requests each php-cgi gets to serve before it's replaced.

  Apparently, this isn't PHP's fault:

  PHP is glue. It is the glue used to build cool web applications by sticking dozens of 3rd-party libraries together and making it all appear as one coherent entity through an intuitive and easy to learn language interface. The flexibility and power of PHP relies on the stability and robustness of the underlying platform. It needs a working OS, a working web server and working 3rd-party libraries to glue together. When any of these stop working PHP needs ways to identify the problems and fix them quickly. When you make the underlying framework more complex by not having completely separate execution threads, completely separate memory segments and a strong sandbox for each request to play in, feet of clay are introduced into PHP's system.

  Sure, I get it, you like having your own process to play around in. But dammit, PHP, why don't you clean up after yourself each request? Is this the experience of everyone who runs PHP with fcgid?

  Dominic

  Sun, 21/6/09 – 1:56pm

  12 comments

Menu