Happy New Year, and…

A small but annoying bug was reported over Christmas by Android users, where pictures were not showing properly on web pages served by The DEOSS Community Server. The odd thing was it only affected some browsers. Users with Firefox for Android were unaffected.

After some investigation it transpired  that mod_security disliked the way some browsers presented their HTTP responses. Anyway, the problem has now been fixed. There was no server downtime.

Posted in Server News | Comments Off on Happy New Year, and…

All WP sites upgraded

All the WordPress-based sites here on the DEOSS Community Server have just been upgraded to version 3.8. The upgrade appears to have gone very smoothly and there was no downtime on any of the affected sites.

Posted in Server News | Comments Off on All WP sites upgraded

Merry Christmas 2013

Firstly, Merry Christmas. It seems to come round faster every year doesn’t it! Now, I must apologise for the lack of recent blog updates. Actually quite lot has happened on the DEOSS Community Server lately and it has been hard to keep track of it all. So here’s a very brief summary:-

  1. Automated plugin updates via rsync over ssh for all WordPress-based sites. Keeping all those plugins updated was becoming very tiresome and was causing delays in updating: A new backend script on the control server means that all updates can be made promptly and easily.
  2. Upgrade of all wordpress sites to 3.7.1
  3. Routine updates to the Debian operating system. These are generally performed on a weekly basis.
  4. Tweaks to the anti DDS (denial of service attack) system. Some users were reporting error 500’s when trying to perform certain functions on their sites. These issues were mostly due to overzealous settings in an Apache module called mod_security. Mod_security is an excellent module but its documentation is not as clear as it might be. And it is a complex bit of kit too with configuration settings that have far-ranging implications for a server such as ours. Therefore configuring it is something of a dark art. Nevertheless, hopefully all the issues have now been resolved.

That’s it for now. More routine work is currently being undertaken this week. No downtime is anticipated.

 

Posted in Good News, Server News | Comments Off on Merry Christmas 2013

DDoS attack update

Good news is that DEOSS is now back on line and running reasonably well. However, I don’t anticipate that the bad guys will be giving up anytime soon.

Consequently, I have been forced to impose some very tight security restrictions in order to fight of this ongoing DDoS attack. The server logs indicate that we are currently experiencing 1 attack every 4 seconds – though at one point it was 10 attacks per second! To fight them off I have deployed a WAF (web application firewall) called ModSecurity – amongst a number of other measures. It is a very powerful tool and is holding up well. However its configuration is very poorly documented and its settings are virtually incomprehensible!

In addition I have had to create my own directives to respond to the particular attack we are experiencing. This means that while I finetune the new security measures to suit all the various web applications on DEOSS, you may get experience some “error 403’s“. This is the default response of the WAF to anything it considers a threat. So if you are a DEOSS customer and you experience unexpected “403’s” then please let me know about them, through the normal channels.

Posted in Good News, Server News | Comments Off on DDoS attack update

Server outage due to suspected DDoS attack

At around 13:30UTC today I received a mail for a customer saying that he could not access this site.

Naturally I got on the case straight away. My investigations found that his site was actually accessible, but only after waiting an unacceptably long period of time. And occasionally the page never appeared at all.  On studying the server logs, I found there was quite a bit of activity. Most notably we had three search engine “robots”simultaneously crawling the web server. And it seems there one a lot of Windows 8 laptop users in mainland China interested in one of our image sites. Strange, I thought…

Whilst one might expect the server to slow down a bit with all this traffic, if would not explain such slow speed. Added to which examining all the running processes indicated that e MySQL server was using most of the server’s processing power, along with a number of power hungry Apache2 threads. I restarted both APache2 and MySQL. This caused a temporary speed up. But then things went back to a snail’s pace.

So I was forced to reboot the entire server at 14:25UTC. It remained offline for around seven minutes. That resolved the speed issue for about ten minutes. So I studied the logs again, particularly all those Chinese Windows 8 laptops and noticed they were all looking at the same few files.

I came to the conclusion that we were under a DDoS attack. Without boring readers with all the gory details I  attempted to harden our server against such attacks, using a method similar to that detailed here.  Unfortunately that didn’t work. All it did was ban the Google bot from crawling the site – which was a somewhat less than desirable outcome.

So I tried a more dramatic approach – using a web cache application called “Varnish“:-

Sadly that didn’t fix the problem either.

At the moment only one site bearing the brunt of attack, and thus hogging computing resources needed by other sites, and the DEOSS mail server. So I have temporarily taken that site off line while my investigations continue. The rest of the web server now appears to be running reasonably satisfactorily – though this is only a very temporary fix.

I will be closely monitoring the situation, while I seek a more permanent and adaptive solution.

Posted in Bad News, Server News | Comments Off on Server outage due to suspected DDoS attack

Server upgrades completed

A large round of server upgrades were undertaken between 2013-01-31 and 2013-02-01. These have now been completed successfully. There were no major issues and no server downtime. 

As usual, the entire DEOSS Community Server is now being mirrored to an off-site remote backup server.

Posted in Good News, Server News | Comments Off on Server upgrades completed

Server upgrades in progress

A number of software upgrades on the DEOSS community server are currently underway. Most of these are fairly routine and we do not anticipate any downtime.

Posted in Server News | Comments Off on Server upgrades in progress

Debian upgrade completed successfully

Well, it’s 03:00 UTC and I am pleased and relieved to report that the upgrade from Debian “Lenny” to Debian “Squeeze” seems to be successful. We had two periods of downtime of around twenty minutes each.

First one was caused by the latest MySQL not wanting to install properly. Unfortunately my fix on the test server did not work here. But I eventually did get it running, by editing one of its configuration files, uninstalling the old mySQL Server – which seems not to have uninstalled cleanly – and issuing and “apt –configure” command on the new MySQL Server 5.1.

Then we had a further twenty minutes downtime when I rebooted the server. Turns out the server had run continuously for 510 days without a reboot. Try doing that on a Windows machine! 🙂 Anyway, when it rebooted, “fsck” (a Linux disk checking tool) insisted on checking the RAID disks. Eventually they passed OK and the machine booted fine.

I have checked the web and  mail servers and everything seems to be fine. I can send and receive emails – certainly to myself. The web server must be working otherwise I would not be able to post this message! There are other tests I need to conduct. However, I’m exhausted and I intend to catch some sleep now and return to it refreshed in a few hours, so that I can undertake further tests. Meantime the entire server is being “rsync’d” to a remote backup server.

Please note:- this is a very major system upgrade. So if you find that anything is not working as expected for you then please let me know ASAP.

Posted in Good News, Server News | Comments Off on Debian upgrade completed successfully

Major planned server upgrade 2012-04-21 and 2012-04-22

I have to do a major operating system upgrade to the DEOSS server this weekend. I have already piloted the upgrade on a test server and it worked out OK. Well there was a nasty issue with the MySQL serve failing to start, but I found a fix and noted it well!

Whilst I don’t anticipate any major issues on the main server, it is a very complex upgrade and there may be some downtime. So you may find that your email is a little flaky for a day or two or that some web pages are temporarily unavailable. However, I hope to have it all running again smoothly by Monday morning 2012-04-23. I will post in this site blog once the upgrade is successfully completed.

Posted in Server News | Comments Off on Major planned server upgrade 2012-04-21 and 2012-04-22

Server upgrades complete

A large round of server upgrades announced 2012-03-05 have now been completed successfully. There were no major issue and no server downtime.  The entire DEOSS Community Server is now being mirrored to an off-site remote backup server.

Posted in Good News, Server News | Comments Off on Server upgrades complete