Server upgrade (our biggest ever!)

Today saw the completion of our biggest ever server upgrade. This will allow us a lot ore room for expansion, along with faster, smoother sites, and easier day-to-day management. However we are sad to be saying goodbye to our trusty upstream provider Bytemark. We had our first Bytemark server back in 2006 and as a company Bytemark has served us well, ever since.

Bytemark was taken over by Iomark, and is no longer actively supporting the “dedicated host” type machine that we have enjoyed for all these years. However, thanks to a very helpful chap at Iomark, I have been able to secure a deal from one of it’s Iomark “sister companies”, Rapidswitch. This deal allowed us a smooth transition period to a new dedicated Debian Linux host. It enabled us to move all sites and services across from the old Bytemark machine to a new one hosted by our new upstream provider, one-by one, testing each one before moving the next one. In addition to allowing a relatively smooth and orderly transition, this deal has also provided us with a dedicated machine with significantly higher specification than the old one.

Most sites moved across very smoothly with only a few minutes downtime whilst the planet’s DNS system caught up with the various nameserver changes. However, some sites suffered slightly longer delays due to an issue that to be frank, I had not really made sufficient allowance for. The new server runs latest Debian stable c/w latest PHP. The old server was running Debian “old stable” c/w PHP 7.4. This meant that some older WordPress plugins had stopped working because they have not been updated to PHP 8.*. So it was necessary to remove these plugins, and where necessary, find more modern replacements. Anyway, after a little scratching around, all the issues arsing from this nave now been resolved.

The performance of this new Rapidswitch server is quite impressive. And its 64GB RAM should ensure we don’t have any “out of memory” errors any time soon. So all-in-all, despite the upheaval I would say this has been a good move for us, and will certainly leave us better placed to deal with the technological challenges the future will inevitably bring.

Posted in Good News, Server News | Comments Off on Server upgrade (our biggest ever!)

Server upgrade

I had to take the DEOSS server off-line for a couple of hours this evening for a routine server operating system update. Essentially it involved upgrading its underlying Debian GNU/Linux operating system from version 10 (“buster” aka “old stable”) to version 11 (“bullseye” aka “stable”).

As Debian upgrades go, this proved to be one of my less eventful ones. Pretty much everything upgraded as expected. It also gave me the opportunity to revisit some configuration issues.

So I’m pleased to report that after several reboots and some fairly extensive server-side testing, the e-mail server and the web server both upgraded successfully and appear to be functioning correctly. And there certainly seems to be a significant improvement in web server performance.

This upgrade should also be good news for those with sites needing updated plug-ins. Some plug-ins refused to upgrade on the version of PHP shipped with Debian 10. Debian 11 has a later version of PHP, which allows these plug-ins to update correctly.

I have subsequently visited and logged-in to all sites hosted on this server and successfully updated all plug-ins. Everything appears to be functioning correctly.

However, with an upgrade of this magnitude, there may still be a few configuration issues and tweaks needed over the next few days. So, if you are a DEOSS customer and you are still having any issues, please contact me through the normal channels.

Posted in Good News, Server News | Comments Off on Server upgrade

Urgent hard disk replacement

Yesterday, 2022-11-28, we needed to do some fairly urgent maintenance work on the DEOSS Community Server. One of its hard disks was failing. The server resides in a data centre in York, and Bytemark (my upstream provider) engineers undertook the work. This is a fairly common operation for its engineers. Nevertheless, there was still the possibility of unforeseen technical issues.

The server uses two disks in a “RAID 1” arrangement, so that if one disk fails, the data is retained on the other disk. When the faulty disk is replaced, the RAID will then write a copy of the data back to the new disk.

Work was initially scheduled for Monday, 2022-11-28, 10:00 UTC. However we hit a small snag. I back-up the server daily to a local machine via RSync. So data loss was unlikely. However if the repair did fail completely, then I wanted to ensure a replacement server could be implemented quickly, without securely squirting 200GiB data across the internet.

So, in addition to my local off-site backups, Bytemark kindly provided me with a backup system that is local to the server, just in case things went wrong. Having a complete dataset in the same building as the server would enable us to build a new server much more quickly if things went badly wrong.

We planned to create this additional backup over the previous weekend. But unfortunately, the backup volume was limited to 100GiB. Bytemark fixed that limitation Monday morning. Which meant that the full backup did not complete until later that day.

Consequently, the the disk swap was not undertaken until 16:00 on Monday afternoon. Fortunately for users, the server remained up for all that time. The swap itself took about 30 minutes. However the reboot took over two hours as a program called ‘fsck’ checked the remaining RAID disk for errors, prior to rebuilding the RAID across onto the replacement second RAID disk. Consequently, the Server went live and became usable again at approximately 18:38. This meant we had a total of 2 hours and 38 minutes actual downtime. This was a little longer than I hoped. But considering the circumstances, it really wasn’t too bad.

However, the server still ran rather slowly for a few more hours as the RAID array rebuilt itself across onto the new disk. The RAID completed at around 23:00 Monday evening. Since then, I have performed fairly extensive tests on the server and all seems well. However, if you are DEOSS customer and still have any problems with your service, then please contact me though the usual channels.

(RAID=’redundant array of inexpensive disks’.)

Posted in Server News | Comments Off on Urgent hard disk replacement

Email issues fixed

Following our recent (2022-02-13) server upgrade, users were having problems collecting their IMAP /POP3  emails from the server. It turned out to be a relatively minor Dovecot configuration issue – just one line in a config file, in fact!

Anyway, this issue is resolved and the email system is now working as expected.

Posted in Good News, Server News | Comments Off on Email issues fixed

Major server upgrade

We’ve just undertaken another major server upgrade. This time it was a full operating system upgrade. This inevitably caused a little unavoidable downtime. The server was offline for about an hour today. But it is up and running again now. We are now running Debian 10, c/w all its associated bits and pieces.

The web-server and its underlying databases appear to be performing correctly. However it seems the upgrade has caused some unforeseen issues with  our email service. We are currently working on these and hope to have them fixed very shortly.

Posted in Good News, Server News | Comments Off on Major server upgrade

Another server upgrade

In addition to the usual routine back-end site updates, we just undertook a fairly serious security upgrade to the sever too. Consequently, DEOSS Community Server was down for around 30 minutes this afternoon. While undertaking the upgrade we took the opportunity to run fsck and check the hard drives for integrity. Everything is fine and we are now back on-line again with security patches applied c/w the latest Linux kernel.

Posted in Server News | Comments Off on Another server upgrade

Server upgrade

We are planning some fairly major work on the DEOSS server over the Bank Holiday weekend. This will include a major operating system upgrade. It will also entail several server reboots. There may also be some further temporary unavoidable outages and glitches while we fix any bugs. We plan to have everything all up and running again properly by Tuesday 2020-04-14.

Meantime enjoy your Weekend.

Posted in Server News | Comments Off on Server upgrade

Server Outage

We suffered a short server outage yesterday afternoon, around 16:40 UTC. Turns out that one of the two hard disk’s in the server’s RAID1 array had developed a fault. So we had to take the server off line and replace the disk. The sever was back up and running by 17:10 UTC.

Users may have noticed that the server was a bit slow early evening as the RAID rebuilt itself, copying data to the replacement disk. This process completed circa 21:00 UTC, and everything seems to be running normally again.

I was first alerted to the issue when the server sent me an automated message around 04:00 UTC, alerting me to a failing hard drive. I ran some tests to confirm we had a problem and find our more about what was actually wrong. At around 08:00 I reported the issue to Bytemark, our up-stream provider, the company that actually houses our rack-mount server in York, UK. Once again, Bytemark sorted this issue very promptly and efficiently, literally within hours of us first reporting it. 🙂

Posted in Bad News, Good News, Server News | Comments Off on Server Outage

Server upgrade

We have just completed a fairly major server OS and software upgrade today. Frankly this was long overdue and was delayed because we knew it would be quite a messy business and could result in some server downtime.

Fortunately the mail-server was only down for a few minutes – though it did go down several times during the afternoon. Unfortunately however, the web-server upgrade took rather longer to fix and was off line for several hours – for which we apologise. Anyway, we are all up and running. We chose a weekend – and rather a warm one at that – so hopefully the inconvenience to our customers was minimal. 🙂

Posted in Server News | Comments Off on Server upgrade

Moving to HTTPS

You may have noticed that when you access your site, there is a change in the way the URL looks in the address-bar on your browser? In a long-overdue upgrade, all our customer sites have been upgraded from HTTP (hypertext transfer protocol) to HTTPS (hypertext transfer protocol secure).

What is the difference?

In an ordinary HTTP session, data that flows to/from the server and the user’s computer is sent in plain text, this means that hackers connected in between may be able to see what is being transmitted.

Using HTTPS, the server and the user’s computer agree on a “code” between them, and then they encrypt the data flowing between them using that “code”, so that no one in between can read them. This keeps your information much safer from hackers.

Redirect

Customer sites have been re-configured, so that all requests for the old HTTP pages are redirected to the new HTTPS version instead. This redirection should happen automatically, with no intervention form the user. A few sites required a little page-by-page tweaking, too. But as far as I am aware, all customer sites have now been upgraded successfully.

Internal projects

Our own sites, deoss.net and deoss.com have also been upgraded to https. All our other projects will follow shortly.

More information

The Wikipedia has a very good page that describes HTTPS and its relative advantages:-

https://en.wikipedia.org/wiki/HTTPS

Posted in Good News, Server News | Comments Off on Moving to HTTPS