Apache mpm_event module running out of slots when reload

I’m finally back, sorry for my long absence, but i moved a year ago to San Francisco to work at ThousandEyes and my life is quite busy at this moment.

So, it seems that the new Apache version 2.4 stopped considering the mpm_event module as experimental and changed it to stable. But i don’t think is as stable as it should be, at least in Ubuntu Trusty, i haven’t tried other distros.

The mpm_event module is basically an improvement of the mpm_worker, it changes how the requests are handled by the Apache threads by creating a main thread to listen all of them and delegating the actual work to other threads freeing the main thread to attend other requests.

There is a bug, easy to reproduce, that makes your Apache server run out of slots to attend requests, if you execute an Apache reload (like the logrotate conf that the Ubuntu Apache package has…) some slots become G in the Apache scoreboard, meaning “Gracefully finishing”, which is expected, the problem is that some of them never comes back as an available slot.

To reproduce it, open two consoles, in the first one:

$ while true ; do service apache2 reload; done

And in the second one:

$ watch -n 1 "apache2ctl status | tail -n30"

You will need some requests to hit your server, because i suspect that the slots that hangs in “Gracefully finishing” are the ones that had an open connection.

Continue reading