Apache mpm_event module running out of slots when reload

I’m finally back, sorry for my long absence, but i moved a year ago to San Francisco to work at ThousandEyes and my life is quite busy at this moment.

So, it seems that the new Apache version 2.4 stopped considering the mpm_event module as experimental and changed it to stable. But i don’t think is as stable as it should be, at least in Ubuntu Trusty, i haven’t tried other distros.

The mpm_event module is basically an improvement of the mpm_worker, it changes how the requests are handled by the Apache threads by creating a main thread to listen all of them and delegating the actual work to other threads freeing the main thread to attend other requests.

There is a bug, easy to reproduce, that makes your Apache server run out of slots to attend requests, if you execute an Apache reload (like the logrotate conf that the Ubuntu Apache package has…) some slots become G in the Apache scoreboard, meaning “Gracefully finishing”, which is expected, the problem is that some of them never comes back as an available slot.

To reproduce it, open two consoles, in the first one:

$ while true ; do service apache2 reload; done

And in the second one:

$ watch -n 1 "apache2ctl status | tail -n30"

You will need some requests to hit your server, because i suspect that the slots that hangs in “Gracefully finishing” are the ones that had an open connection.

Wait for a little bit and you will see your scoreboard getting more and more G until you end up without no open slots and your Apache web server stops serving requests.

For example:

 GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG
 GGGGGGGGGGG.........................GGGGGGGGGGGGGGGGGGGGGGGGG___
 _________________W____GGGGGGGGGGGGGGGGGGGGGGGGG.................
 .................................GGGGGGGGGGGGGGGGGGGGGGGGG______
 _____RW___________________________W_________....................
 ................................................................
 ..........GGGGGGGGGGGGGGGGGGGGG........GGGGGGGGGGGGGGGGGGGGGGGGG
 ..........GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG....
 ......................................GGGGGGGGGGGGGGGGGGGGGGGGG.
 ...GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG.............
 ..........GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG....
 ..............................................GGGGGGGGGGGGGGGGGG
 GGGGGGG.........................................................
 ..........GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG
 ..........GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG....
 ........................................

   Scoreboard Key:
   "_" Waiting for Connection, "S" Starting up, "R" Reading Request,
   "W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
   "C" Closing connection, "L" Logging, "G" Gracefully finishing,
   "I" Idle cleanup of worker, "." Open slot with no current process

Hopefully, there is a workaround, which is switching back to the event_worker module. You will probably lose a bit of performance, but in my opinion, in most cases, the worker module will work as well as the event module. And as far as i’ve read on the internet, if you use SSL (which is the normal usage in most of websites today) the mpm event module falls back to mpm_worker, therefore, it doesn’t make much sense to use mpm_event if you have SSL.

So:

$ a2dismod mpm_event
$ a2enmod mpm_worker
$ service apache2 restart

You can keep the same configuration parameters for both, they behave in the same way and have the same defaults.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s