Open Mind Status

For reasons unknown at this point, the backups that were restored through the night are actually from February of this year.  This is of course a major blow to the hard work the team put in through the night as well as causing further inconvenience for users.

We are presently downloading the offsite backups which are up to date and we shall then have to start the restoration of accounts from scratch.

As the offsite backups are already in tar format, the process shouldn’t take as long as it did last night.

Obviously this is a major setback but we would appreciate your patience and understanding whilst we try and resolve this asap.

Please note that to keep accounts in sync we have deliberately stopped all services on the Linux server whilst we restore from the recent backups

Update 09:35 – Backups from 15th April are now being downloaded t the server.  We expect this to be complete by approximately 10:45

Update 10:11 – Approximately 5GB of data left to download before we can start restoring. This should start at around 10:30

Update 10:50 – Currently terminating accounts so that they can be recreated with the correct data.

Update 12:10 – The restoration of accounts has now begun in alphabetical order.  Each account takes between 30-60 seconds to fully restore so it is hard for us to give an ETA of the completion. As each account is restored however, it will immediately become active again for web traffic and email.

Update 12:18 – Restoration 15% complete

Update 12:31 – Restoration 25% complete

Update 12:45 – Restoration 30% complete

Update 13:11 – We’re just resolving a minor issue with cPanel and we will then be able to continue with the account restoration.

Update 13:27 – Earlier problems with cPanel have been resolved and we are continuing with the restoration albeit in blocks of 10 domains at a time to prevent server overloads.

Update 13:52 – We are now well over half way through the restoration process.

Update 14:22 – We are now restoring the last batch of accounts and this should be completed within the hour.  Thank you for your patience during this very frustrating period.

Update 14:51 – There are currently around ten accounts left to restore…

We are aware of an issue where PHP files are not being parsed rather the browser is trying to download the file.  This is being treated as a high priority and will be dealt with as soon as the restoration task completes.

Update 15:19 – All accounts have now been full restored.  We are currently recompiling Apache/PHP to resolve the above issue.

We are aware of an issue with the MySQL server on Linux Server One and we are working to resolve this as quickly as possible.

Update 18:36 – The server is failing to respond due to an excessively high CPU usage.  Wea re attempting to reboot the server now.

Update 18:40 – It looks like the reboot has not resolved the issue, technicians are on-site and are investigating further.

Update 19:29 – We have gained access to the server but it does look like a major issue with the boot disk. We are running a disk repair at the moment to try and resolve the boot issue.

Update 20:24 – Unfortunately it does look like the primary hard disk on the server is fatally corrupted.  The procedure is now that we will have to replace the disk, reinstall the OS and cPanel and then start restoring accounts.  Note that this process will take several hours to complete but we will give you constant updates as to the progress.

Update 21:05 – The faulty drive has now been replaced and the operating system re-installed. We are currently installing cPanel/WHM

Update 22:33 – cPanel/WHM has now been re-installed and we are in the process of securing it for the restoration of accounts.

Update 02:14 – We have now commenced restoring user account.  This will be done alphabetically and we shall proceed as quickly as possible.

Update 02:38 – Account zero-a restored

Update 02:58 – “B” accounts restored

Update 03:27 – “C” accounts restored

Update 03:42 – “D” accounts restored

Update 03:53 – “E” accounts restored

Update 04:34 – “F” accounts restored

Update 05:00 – “G” & “H” accounts restored

Update 05:10 – “I” accounts restored

Update 05:12 – “J” accounts restored

Update 05:19 – “K” accounts restored

Update 05:25 – “L” accounts created

Update 05:41 – “M” accounts restored

Update 05:47 – “N” accounts restored

Update 06:00 – “O” accounts restored

Update 0612 – “P” accounts restored

Update 06:14 – “Q” accounts restored

Update 06:18 – “R” accounts restored

Update 06:47 – “S” accounts restored

Update 07:04 – “T” accounts restored

Update 07:06 – “U” accounts restored

Update 07:18 – “W” accounts restored

Update 07:23 – “Z” accounts restored

All user accounts have nw been restored from the backups taken at around 2am yesterday morning. Would allusers please carefully check their account and report any issues through the helpdesk: https://www.openmindhosting.co.uk/support/

A RFO (Reason For Outage) will be issued shortly.