Open Mind Status

We shall this evening be upgrading the primary Windows mail server and relocating all user mailboxes to a new dedicated server. We apologise for the short notice for this work but it has become necessary as the current server is starting to show signs of instability and we would rather act now to resolve the issue quickly thus avoiding any unnecessary disruption to your service.

The work will involve moving all user account currently hosted on server IS-3712 to our new server IS-11329. There will be no changes required by you as we will be migrating the mailboxes and data as well as moving the current mail IP address to the new server. We have already synchronised the mailbox data to the new box to ensure the migration is as smooth as possible.

In addition, whilst the migration is being carried out, mail will be delivered to our backup mail server and once complete, will be delivered as normal to your mailbox to ensure no mail is lost in the transition.

The anticipated time line and updates is shown below. You can also follow updates via out twitter account http://twitter.com/openmindhosting

Please note that you will only be affected by this work if you have a shared or reseller Windows hosting plan. All dedicated mail server clients and Linux plans will NOT be affected.

Anticipated Schedule

Please note this is a planned time line for this work and may be subject to change. All actions will be carried out this evening, 26th April 2010:

• 19:00 – The mail server will be stopped and access to the Helm control panel/webmail access disabled for the duration of the move. We shall then begin a final synchronisation of the mailbox data. This synchronisation will take approximately 30 minutes to complete.
• 19:30 – We will carry out some final checks on the data and reconfigure the new mail server to use the existing mail IP address.
• 19:45 – Once the mail data has been synchronised we will begin to restore accounts on the new server. Each domain takes between 5 and 20 seconds and the domains are restored alphabetically.  There are approximately 1,300 domains to restore on this particular mail server. As each domain is restored, mail will start flowing into the new mail server and will also start arriving from the backup mail server. We anticipate, and this is a very rough estimate, that the restoration of accounts will take about 4 hours to complete.
• c. 23:45 – The mail restoration should be complete. Helm will be restarted and access to webmail will be restored.

Please note that the above timings are subject to change but we will keep you updated throughout the move both here and on our twitter account at http://twitter.com/openmindhosting

We are currently installing new software on each of the Windows cluster servers. This process does involve stopping IIS for about 2 minutes whilst the software installs.

Currently we are installing on the primary mail server and this should be completed shortly.

Update 10:57 – Mail server installation completed.

Update 11:32 – MySQL4 database server installation in progress

Update 12:53 – This work is now complete.

We are aware of a delay in receiving email.  This has been caused by a backlog in the mail server queue which we are resolving at present.

Update 14:29 – We have identified the issue and working to resolve it now. It looks like emails are sitting in the spool ready to be delivered but not actually leaving the server.

Update 14:47 – Mail is starting to work it’s way through now but this may take some time to complete due to the previous backlog.

Update 15:12 – We have found the issue to be associated with the spam check we carry out on all email coming into the server. As a temp. measure to clear the backlog, we have removed the spam checks and will re-enable them once the queue has been cleared.

This will result in a higher level of spam being received for a short period.

Update 15:25 – The backlog has now been cleared.  We will now start re-enabling the anti-spam services we have in place one by one whilst monitoring the delivery spool until we are satisfied all is OK.

It looks like we have a disk failure on this server resulting in MySQL4 databases becoming non-operational.

Technicians are currently working on the issue and we shall post an update as soon as we have one…

UPDATE 15:19 – Unfortunately it is confirmed that the drive has failed and we are currently waiting for technicians to replace it. Once this is done we shall be able to restore data from the backups.

UPDATE 16:18 – The disk has now been replaced and we have started to restore data from the backups.  This should take no more than 2 hours to complete.

UPDATE 16:47 – Data has now been restored, we are in the process of reconfiguring the database server.

UPDATE 19:18 – Unfortunately our batch restore script is failing to restore databases correctly so to ensure zero data loss we are having to restore databases one by one.  There are currently 181 databases on the server and each one takes 30-60 seconds to restore.  Databases will be restored in alphabetical order so we appreciate your patience whilst we carry out this work.

UPDATE 20:44 – We are now restoring databases beginning with “H” and are more than 50% through the restoration work.

UPDATE 23:03 – All databases have now been restored, if you experience any further problems then please do not hesitate to get in touch with our support team.  We shall be issuing a full RFO (Reason For Outage) tomorrow.

We shall be carrying out some essential maintenance on the Helm control panel this morning so it will be unavailable for approximately 30 minutes.

UPDATE 12:34 – This work has now been completed.

We are currently in the process of upgrading our primary Windows statistics server up to the latest release of SmarterStats.

During the up;grade users will not be able to access their stats reports. We shall update this post once the work has been completed.

UPDATE 11:21 – This work has now been completed and stats are once again available to all Windows network users.

We will shortly commence an upgrade to our primary and secondary DNS servers.

The DNS servers will continue to operate throughout the upgrade so there will be no interruption in service. During the upgrade however, you will not be able to add, edit or remove and DNS records from existing zones within Helm.

We will update this post once the upgrade has been completed.

UPDATE 12:24 – The upgrade has now been completed with zero downtime or loss of data. We hope you will notice an increase in speed with DNS server requests from now on.