Why would MySQL crash when WordPress attacked?

WordPress brute force attacks are increasingly common at the time I write this – September 2014. Recently a server I look after was coming under attack and then after a few minutes the site would display ‘Error establishing a Database Connection’. error-establishing-database-connectionWhen I logged into the server the MySQL service wasn’t running.

My first thoughts were that something in the attack was causing MySQL to crash, There was no information in the MySQL error.log to indicate why though so I spent time looking at plugins to help mitigate a brute force attacks, which tend to focus on the WordPress wp-login.php and xmlrpc.php files:

Disable XML-RPC will disable WordPress XML-RPC functionality meaning that nothing can be done to your content, however this does nothing to prevent initial database queries so MySQL can still be affected when running this, and it does nothing about wp-login.php either.

BruteProtect – I’ve seen comments that it will protect xmlrpc but I’ve seen it in action during an attack and as of version 2.2.2 it did nothing to stop it.

NinjaFirewall – there are a lot of configuration options but this one does the job. It sits in front of WordPress and so it can intercept the attack before all the database activity starts up. This worked great when I used it during an attack.

However you may host multiple WordPress sites on a single server and it could be tedious having to install this and configure it for each site, plus there is going to be duplication of what the code is doing and Apache still has to handle all the incoming requests.

It may be better to stop any attacks at the firewall because that’s the single point of entry to your server. This approach still requires a plugin – WP fail2ban but this one will log attacks to your systems auth.log which can be picked up by fail2ban and automatically added for a temporary period to your firewall rules. It is more complicated than the other plugins to install but once a malicious IP has been blocked it can’t affect any of the sites on your server will keep the system load much lower.

After getting the fail2ban solution in place the database was brought down yet again. I ended up looking in kern.log and at the time of the attack was the line ‘apache2 invoked oom-killer’ – that revealed the problem… Apache was using so many processes that it was running out of memory and as this particular server runs off an SSD the swap file is disabled. As there was nowhere else to get memory from it started killing off other processes, the biggest of these was MySQL.

What a relief to find the cause – MySQL wasn’t crashing at all. The solution was just to reduce the MaxClients value in apache2.conf – the default value of 150 is way too big for a modest sized server. If each process takes 40 MB then 150 processes require 6 GB of RAM. Getting a realistic value for this requires a little load testing plus the use of some tools to see how the memory usage increases as Apache has to handle more requests.

Ubuntu modern.IE extracting VM files

If you follow the instructions on the www.modern.ie site to extract a VirtualBox file from a set of downloaded .rar files you are asked to run the following command:

./IE11.Win8.1.For.LinuxVirtualBox.part1.sfx

This gives the error:

bash: ./IE11.Win8.1.For.LinuxVirtualBox.part1.sfx: No such file or directory

The solution is to run:

unrar e IE11.Win8.1.For.LinuxVirtualBox.part1.sfx

Is it a good idea to use a swap file on a cloud server?

ElasticHosts recommend: ‘You should configure your operating system without swap, using additional memory if necessary. This will significantly improve performance, since drives are provided over network RAID.’

I haven’t been brave enough to try this yet – but a brief bit of research showed that the following commands could be used.

#force the kernel not to use swap (default = 60)
sysctl vm.swappiness=0
#or  disable the swap file
swapoff -v /swapfile
# to confirm … show memory and swap usage
free -m

Logrotate not restarting Apache

Spotted an interesting problem where adding the ‘compress’ directive to logrotate stopped it from doing its daily restart of Apache.

After a bit of diagnosis it turns out that the gzip compression used by logrotate can be very sensitive to non-letter characters in filenames. My log files were all named with an underscore instead of the dot character e.g. access_log instead of access.log.
I think I did this initially to make it easier to find them e.g. locate *_log would give me all the httpd log files I was interested in, instead of locate *.log giving every log file on the system.

I’ve renamed them all back to the more conventional access.log and it is all working as it should.

Lesson learned, don’t try to be too clever in IT.

More on lightweight Linux distros

After some initial experiments with Zenwalk, I have tried 3 more lightweight Linux distros, with the aim of finding something that will run well on an older PC with only 256MB of RAM, but provide a user friendly interface and be capable of performing day to day office tasks.

They are:

Here is my very superficial review, please don’t take it too seriously.

Puppy Linux is the smallest of the three and the ISO file is only 131MB. It is optimized for running from CD or USB. Although it tries to be user friendly it is some way behind the other two for non-technical users. There are lots of strange application names and icons in the menus, so for now I’m putting that aside.

lubuntu is a project that is intended to lead to an official derivative of the Ubuntu operating system that is “lighter, less resource hungry and more energy-efficient”, using the LXDE desktop environment. I installed this to hard disk using an Oracle VirtualBox Virtual Machine.

Pros:
On booting it consumed only 56MB of RAM so there was plenty left over for running applications.
Cons:
It uses chromium as the default browser, and this doesn’t have Adobe Flash installed.

Peppermint OS has the goal of providing a fast, web-centric operating system that’s easy to learn and effective when put in use. The default desktop interface is LXDE. I’ve also run this as a virtual machine.

Pros:
On booting, it consumed 86MB of RAM, a little higher than lubuntu but still very reasonable.
Firefox is the default browser and Adobe Flash is also installed.
There is an update manager that lets you know when package updates are available and a very nice software installer for locating and installing additional applications.
Cons:
I’ll post them here after I’ve spent some more time investigating…

In conclusion Peppermint OS looks like the best option for my requirements at the moment, but it is only through day-to-day use that I’ll uncover any issues.