Server outage due to suspected DDoS attack

At around 13:30UTC today I received a mail for a customer saying that he could not access this site.

Naturally I got on the case straight away. My investigations found that his site was actually accessible, but only after waiting an unacceptably long period of time. And occasionally the page never appeared at all.  On studying the server logs, I found there was quite a bit of activity. Most notably we had three search engine “robots”simultaneously crawling the web server. And it seems there one a lot of Windows 8 laptop users in mainland China interested in one of our image sites. Strange, I thought…

Whilst one might expect the server to slow down a bit with all this traffic, if would not explain such slow speed. Added to which examining all the running processes indicated that e MySQL server was using most of the server’s processing power, along with a number of power hungry Apache2 threads. I restarted both APache2 and MySQL. This caused a temporary speed up. But then things went back to a snail’s pace.

So I was forced to reboot the entire server at 14:25UTC. It remained offline for around seven minutes. That resolved the speed issue for about ten minutes. So I studied the logs again, particularly all those Chinese Windows 8 laptops and noticed they were all looking at the same few files.

I came to the conclusion that we were under a DDoS attack. Without boring readers with all the gory details I  attempted to harden our server against such attacks, using a method similar to that detailed here.  Unfortunately that didn’t work. All it did was ban the Google bot from crawling the site – which was a somewhat less than desirable outcome.

So I tried a more dramatic approach – using a web cache application called “Varnish“:-

Sadly that didn’t fix the problem either.

At the moment only one site bearing the brunt of attack, and thus hogging computing resources needed by other sites, and the DEOSS mail server. So I have temporarily taken that site off line while my investigations continue. The rest of the web server now appears to be running reasonably satisfactorily – though this is only a very temporary fix.

I will be closely monitoring the situation, while I seek a more permanent and adaptive solution.

Posted in Bad News, Server News | Comments Off on Server outage due to suspected DDoS attack

Server upgrades completed

A large round of server upgrades were undertaken between 2013-01-31 and 2013-02-01. These have now been completed successfully. There were no major issues and no server downtime. 

As usual, the entire DEOSS Community Server is now being mirrored to an off-site remote backup server.

Posted in Good News, Server News | Comments Off on Server upgrades completed

Server upgrades in progress

A number of software upgrades on the DEOSS community server are currently underway. Most of these are fairly routine and we do not anticipate any downtime.

Posted in Server News | Comments Off on Server upgrades in progress

Debian upgrade completed successfully

Well, it’s 03:00 UTC and I am pleased and relieved to report that the upgrade from Debian “Lenny” to Debian “Squeeze” seems to be successful. We had two periods of downtime of around twenty minutes each.

First one was caused by the latest MySQL not wanting to install properly. Unfortunately my fix on the test server did not work here. But I eventually did get it running, by editing one of its configuration files, uninstalling the old mySQL Server – which seems not to have uninstalled cleanly – and issuing and “apt –configure” command on the new MySQL Server 5.1.

Then we had a further twenty minutes downtime when I rebooted the server. Turns out the server had run continuously for 510 days without a reboot. Try doing that on a Windows machine! 🙂 Anyway, when it rebooted, “fsck” (a Linux disk checking tool) insisted on checking the RAID disks. Eventually they passed OK and the machine booted fine.

I have checked the web and  mail servers and everything seems to be fine. I can send and receive emails – certainly to myself. The web server must be working otherwise I would not be able to post this message! There are other tests I need to conduct. However, I’m exhausted and I intend to catch some sleep now and return to it refreshed in a few hours, so that I can undertake further tests. Meantime the entire server is being “rsync’d” to a remote backup server.

Please note:- this is a very major system upgrade. So if you find that anything is not working as expected for you then please let me know ASAP.

Posted in Good News, Server News | Comments Off on Debian upgrade completed successfully

Major planned server upgrade 2012-04-21 and 2012-04-22

I have to do a major operating system upgrade to the DEOSS server this weekend. I have already piloted the upgrade on a test server and it worked out OK. Well there was a nasty issue with the MySQL serve failing to start, but I found a fix and noted it well!

Whilst I don’t anticipate any major issues on the main server, it is a very complex upgrade and there may be some downtime. So you may find that your email is a little flaky for a day or two or that some web pages are temporarily unavailable. However, I hope to have it all running again smoothly by Monday morning 2012-04-23. I will post in this site blog once the upgrade is successfully completed.

Posted in Server News | Comments Off on Major planned server upgrade 2012-04-21 and 2012-04-22

Server upgrades complete

A large round of server upgrades announced 2012-03-05 have now been completed successfully. There were no major issue and no server downtime.  The entire DEOSS Community Server is now being mirrored to an off-site remote backup server.

Posted in Good News, Server News | Comments Off on Server upgrades complete

More server upgrades

A number of significant server upgrades are currently underway. Whilst this is going to be a relatively long job, we don’t anticipate any significant server downtime. We will post again when the upgrades are complete

Posted in Server News | Comments Off on More server upgrades

Server software upgrades

All the main content management systems hosted here on the DEOSS community server have just been upgraded, including all those using Coppermine, eGroupware, Joomla, Moodle, phpBB, Tikiwiki and WordPress.

A significant number of Wordplus plugins such as the contact forms and galleries also required upgrading. Meantime the Moodle sites have also received a fairly major upgrade. We have also done some routine Debian updates over the last few days. Joomla and WordPress contact form users will receive a test email via their various contact forms shortly to verify that their systems are working correctly.

All works were successfully completed without any server downtime. The entire DEOSS Community Server was then RSync’d to a remote backup server early this morning.

Posted in Good News, Server News | Comments Off on Server software upgrades

Routine server upgrades

A number of routine server upgrades have just been completed. This includes critical security updates to Moodle, as well as routine updates to all sites using Coppermine, eGroupware, Joomla,  phpBB, Tikiwiki and WordPress. All third-party WordPress plugins have also been upgraded.

We have also undertaken routine Debian updates. All updates were  completed successfully and there was no server downtime. DEOSS Community Server is now undergoing a full backup to a remote server.

Posted in Server News | Comments Off on Routine server upgrades

Server upgrades

All the main content management systems hosted here on the DEOSS community server have been upgraded, including all those using Coppermine, eGroupware, Joomla, Moodle, phpBB, Tikiwiki and WordPress. A significant number of Wordplus plugins such as the contact forms and galleries also required upgrading. WordPress contact form users will receive a test email via their contact forms shortly to verify that their systems are working correctly.

We have also done routine Debian updates over the last few days. All this work has now been successfully completed with no server downtime.

The entire server is currently being RSync’d to a remote backup server.

On a slightly sour note, the experimental eyeOS system on GarfNet has been indefinitely suspended. This is due to a long term bug in eyeOS that remains unfixed. This bug causes a very nasty PHP error, that prevents eyeOS from working. We have no date for resuming theeyeOS service – nor can we say whether it will even be resumed at all. All other services are unaffected.

Posted in Server News | Comments Off on Server upgrades