Yes, the server move is complete. The new server is fantastic. And we have a new server configurator, which will make the job of managing all the projects on the DEOSS community server much easier and safer.
Now, we know there are a few bugs and outstanding issues with some projects. These are being fixed right now. And there will be some software upgrades to individual projects. But in principle, the hard work is done! 🙂
In a nutshell, we have to move out of our existing server, that currently resides in a secure data centre in Manchester, and move into a new server, located in York. This was not my idea, I hasten to add. I have nothing against Manchester! 😉
It has come about due to restructuring on the part of our upstream service provider, Bytemark. Obviously this is causing me massive disruption that I am trying very hard not to pass on to my users. However some disruption is inevitable, I’m afraid.
To sweeten the pill, Bytemark has done me a rather good deal where we get a faster and more highly-specified server for very little extra money. I am also taking the opportunity to improve the facilities that the server offers. Over the years customers have requested various features that the old server was either unable to provide, or provided rather poorly. For example:-
I am also taking the opportunity to weed out a lot of cruft that has accumulated over the years. To make most efficient use of the available computing resources, a number of redundant or barely-used services will be withdrawn:-
The whole move has to be completed by midnight 2014-10-31. However I am hoping to have the majority of this work done much earlier than this.
A small but annoying bug was reported over Christmas by Android users, where pictures were not showing properly on web pages served by The DEOSS Community Server. The odd thing was it only affected some browsers. Users with Firefox for Android were unaffected.
After some investigation it transpired that mod_security disliked the way some browsers presented their HTTP responses. Anyway, the problem has now been fixed. There was no server downtime.
All the WordPress-based sites here on the DEOSS Community Server have just been upgraded to version 3.8. The upgrade appears to have gone very smoothly and there was no downtime on any of the affected sites.
Firstly, Merry Christmas. It seems to come round faster every year doesn’t it! Now, I must apologise for the lack of recent blog updates. Actually quite lot has happened on the DEOSS Community Server lately and it has been hard to keep track of it all. So here’s a very brief summary:-
That’s it for now. More routine work is currently being undertaken this week. No downtime is anticipated.
Good news is that DEOSS is now back on line and running reasonably well. However, I don’t anticipate that the bad guys will be giving up anytime soon.
Consequently, I have been forced to impose some very tight security restrictions in order to fight of this ongoing DDoS attack. The server logs indicate that we are currently experiencing 1 attack every 4 seconds – though at one point it was 10 attacks per second! To fight them off I have deployed a WAF (web application firewall) called ModSecurity – amongst a number of other measures. It is a very powerful tool and is holding up well. However its configuration is very poorly documented and its settings are virtually incomprehensible!
In addition I have had to create my own directives to respond to the particular attack we are experiencing. This means that while I finetune the new security measures to suit all the various web applications on DEOSS, you may get experience some “error 403’s“. This is the default response of the WAF to anything it considers a threat. So if you are a DEOSS customer and you experience unexpected “403’s” then please let me know about them, through the normal channels.
At around 13:30UTC today I received a mail for a customer saying that he could not access this site.
Naturally I got on the case straight away. My investigations found that his site was actually accessible, but only after waiting an unacceptably long period of time. And occasionally the page never appeared at all. On studying the server logs, I found there was quite a bit of activity. Most notably we had three search engine “robots”simultaneously crawling the web server. And it seems there one a lot of Windows 8 laptop users in mainland China interested in one of our image sites. Strange, I thought…
Whilst one might expect the server to slow down a bit with all this traffic, if would not explain such slow speed. Added to which examining all the running processes indicated that e MySQL server was using most of the server’s processing power, along with a number of power hungry Apache2 threads. I restarted both APache2 and MySQL. This caused a temporary speed up. But then things went back to a snail’s pace.
So I was forced to reboot the entire server at 14:25UTC. It remained offline for around seven minutes. That resolved the speed issue for about ten minutes. So I studied the logs again, particularly all those Chinese Windows 8 laptops and noticed they were all looking at the same few files.
I came to the conclusion that we were under a DDoS attack. Without boring readers with all the gory details I attempted to harden our server against such attacks, using a method similar to that detailed here. Unfortunately that didn’t work. All it did was ban the Google bot from crawling the site – which was a somewhat less than desirable outcome.
So I tried a more dramatic approach – using a web cache application called “Varnish“:-
Sadly that didn’t fix the problem either.
At the moment only one site bearing the brunt of attack, and thus hogging computing resources needed by other sites, and the DEOSS mail server. So I have temporarily taken that site off line while my investigations continue. The rest of the web server now appears to be running reasonably satisfactorily – though this is only a very temporary fix.
I will be closely monitoring the situation, while I seek a more permanent and adaptive solution.
A large round of server upgrades were undertaken between 2013-01-31 and 2013-02-01. These have now been completed successfully. There were no major issues and no server downtime.
As usual, the entire DEOSS Community Server is now being mirrored to an off-site remote backup server.
A number of software upgrades on the DEOSS community server are currently underway. Most of these are fairly routine and we do not anticipate any downtime.
Well, it’s 03:00 UTC and I am pleased and relieved to report that the upgrade from Debian “Lenny” to Debian “Squeeze” seems to be successful. We had two periods of downtime of around twenty minutes each.
First one was caused by the latest MySQL not wanting to install properly. Unfortunately my fix on the test server did not work here. But I eventually did get it running, by editing one of its configuration files, uninstalling the old mySQL Server – which seems not to have uninstalled cleanly – and issuing and “apt –configure” command on the new MySQL Server 5.1.
Then we had a further twenty minutes downtime when I rebooted the server. Turns out the server had run continuously for 510 days without a reboot. Try doing that on a Windows machine! 🙂 Anyway, when it rebooted, “fsck” (a Linux disk checking tool) insisted on checking the RAID disks. Eventually they passed OK and the machine booted fine.
I have checked the web and mail servers and everything seems to be fine. I can send and receive emails – certainly to myself. The web server must be working otherwise I would not be able to post this message! There are other tests I need to conduct. However, I’m exhausted and I intend to catch some sleep now and return to it refreshed in a few hours, so that I can undertake further tests. Meantime the entire server is being “rsync’d” to a remote backup server.
Please note:- this is a very major system upgrade. So if you find that anything is not working as expected for you then please let me know ASAP.
