The net result of this programme will be a network with greater capacity and redundancy, complete with a ‘future proofing’ capability enabling us to offer you new services like IPv6 and MPLS in the very near future.

We are planning to implement this upgrade in two phases over the coming weeks, with the first scheduled window being 1st December between the hours of 01:00 am and 06:00 am GMT. During this window clients will experience a very short period of connectivity interruption of less than two minutes as we change the routing configuration from the current equipment to four fully redundant network segments. The second phase, details of which will be released in a separate communication, will involve the migration of the rack uplinks to the new network segments. Once again as a consequence of implementing Phase 2, clients will also experience a period of less than two minutes of connectivity interruption.

We apologise for any inconvenience this may cause, and hope that you’ll appreciate that we are undertaking this action to ensure that we can continue to provide the best hosting service available.

In the meantime, you can collect email through the webmail service at http://webmail.helmcontrolpanel.net and logging in with your email address and password.

Update 10:33 – This problem has now been resolved.

Update 19:00 – The following email has been sent to all affected users:

You are receiving this email as you currently have a Windows hosting plan with Open Mind Hosting.

IMPORTANT! This issue ONLY affects users who have sites on server IS-03716. It does NOT affect users on server IS-08313. In short, if your web site is available then you are not affected.

At around 5:30pm we detected an issue with server IS-03716 which is used to serve website requests on our shared hosting platform. On investigation it appears that the primary disk for this server may be in the process of failing. To prevent any data loss we have taken the server offline whilst we perform a full diagnostic and attempt to repair the disk.

Based on the results of these tests, we may have to reinstall the operating system and/or retrieve your web site data from backups which were taken at around 1am this morning.

The net result of this is that all web sites on the affected server will not be currently available. This does not affect email or databases as they are both on separate servers.

We shall provide updates as soon as we have them via our status board at http://www.openmindstatus.com so please try and avoid submitting a ticket asking for updates as we post progress on the status board for all to see. Once we have restored service, a further email will be sent.

Obviously this is an inconvenience to you but rest assured we are working as quickly as possible to resolve the situation.

Update 21:20 – We are currently running a check disk on both drives to rule out any drive corruption.

Update 02:30 – the disk check is still continuing. Note that this the preferred method to formatting the server. A further update will be issued shortly.

Update 04:00 – the chkdsk is still running but should be completed shortly.

Update 09:00 – Unfortunately due to the time chkdsk is taking to complete it does indicate that the level of corruption is unsalvageable . A decision has been made to reload the OS to prevent further downtime. A new status post has been made to track the restoration progress.

The maintenance will commence within the net hour and will last approximately 30 minutes.

During this time mail services will not be available. Further updates will be given when the work commences and once it is complete.

18:06 – The upgrade has now commenced.

18:14 – The upgrade has now completed and mail services restored.

A full change log may be found here:

http://www.smartertools.com/smartermail/releasenotes/v8.aspx

During this time, mail services will be unavailable to users and mail will be redirected to our offsite mail server.

Once the upgrade is complete, access will be restored for users and mail sent to the backup server will be delievered as normal.

We expect the upgrade to take approximately 15 minutes…

18:47 – The upgrade process has now commenced…

19:04 – The upgrade has now been completed and mail services have returned to normal.

For more information on SmarterMail v8, please visit the SmarterTools site:

http://www.smartertools.com/smartermail/mail-server-software.aspx

Date: 14/03/2011

Time: 19:00

Duration: < 50 minutes

Affected Service(s): IS-03093, IS-03384, IS-03587, IS-03711, IS-03712, IS-03713, IS-03714, IS-03715, IS-03716, IS-03717, IS-03718, IS-03719, IS-05965, IS-07420, IS-07931, IS-08313, IS-08532, IS-08817, IS-08859, IS-08985, IS-11329, IS-12110, IS-12290, IS-12441, IS-12812, IS-13332, IS-13333, IS-13468, IS-13520, IS-13740, IS-13782, IS-13880, VS-01020, VS-01143, VS-01834

LINX have made an announcement that they are experiencing problems. This was affecting any traffic coming to the Open Mind Hosting network via LINX.  Any traffic transiting to Open Mind Hosting from any other peering points would have been unaffected

Our network engineers responded quickly and have shutdown our sessions at LINX. Traffic has been routed to alternative peers and should be flowing normally. Subsequent announcements from LINX have suggested that they have isolated the cause and are looking to fix it, but the Open Mind Hosting sessions at LINX will remain offline until further notice.

It is important to stress that the connectivity issues are outside of our control and should you be affected you should contact your Internet Service Provider for assistance.

If anyone is still experiencing any adverse effects please contact us immediately. We will continue to monitor the situation and will restore traffic via LINX only when we are confident they have fixed the problems within their network.

It is not clear at this time where the issue lies but engineers are on site and are investigating as a high priority.

Update 09:06 – Resolution and RFO:

Date: 17/11/2010

Time: 07:40

Outage Type: Connectivity Interruption

Duration: < 55 minutes

Affected Service(s): IS-03093, IS-03384, IS-03587, IS-03711, IS-03712, IS-03713, IS-03714, IS-03715, IS-03716, IS-03717, IS-03718, IS-03719, IS-05965, IS-06110, IS-07420, IS-07931, IS-08313, IS-08532, IS-08817, IS-08859, IS-08985, IS-09830, IS-11329, IS-12110, IS-12290, IS-12441, IS-12812, VS-01020, VS-01143, VS-01695

At 08:00 this morning the cluster experienced similar issues to those on the 18/11/10. This again only affected a proportion of the traffic coming through RSH North side. However, those affected will have seen packet loss and increased latency.

Having identified the issue we have taken preventative steps. The cluster is now operating correctly and in a stable manner. We will provide further information as it is available but services are operating within normal parameters again. We apologise for the recurrence of this issue and are continuing to communicate with Cisco.

If you are experiencing any problems as a result of this please do not hesitate to contact our support team immediately and we will look into this for you.

12:15 – This has now been fully resolved. In addition we have upgraded PHP to the latest stable release.

We are monitoring the situation and will keep users updated with more information as and when we have it.

Update 10:38

Date: 18/10/2010
Time: 09:20
Duration so far: 1 hour, 18 minutes
Affected Service(s): IS-03093, IS-03384, IS-03587, IS-03711, IS-03712, IS-03713, IS-03714, IS-03715, IS-03716, IS-03717, IS-03718, IS-03719, IS-05965, IS-06110, IS-07420, IS-07931, IS-08077, IS-08313, IS-08532, IS-08817, IS-08859, IS-08985, IS-09830, IS-09906, IS-11329, IS-11743, IS-11959, IS-12110, IS-12290, IS-12441, IS-12812

There is currently a network issue affecting RSH North. At this time we believe this to be caused by a network routing problem at one of our peering connections. This problem is only affecting a proportion of traffic via RSH North.

Update 11:17

The issue affecting RSH North is ongoing. A proportion of the traffic into RSH North will currently be experiencing increased latency and packet loss at varying levels. In some instances this may appear to be complete connectivity loss.

We are investigating the cause and will provide further updates as soon as we possibly can.

Update 13:24:

The initial problem with a peering point that occurred earlier this morning has led to a problem within the VSS routing cluster that services RSH North. Our network team quickly eliminated as many causes as possible. This issue was then escalated to Cisco and we are currently involved in joint investigation to try and discern the underlying problem. As soon as we have any progress from this work we will inform you immediately.

Update 13:59:

Network connectivity has now been restored across all the servers however we are waiting fr an official all clear from the data centre before showing this issue as resolved.

Summary

* The data centre will be enacting our procedures for a controlled power down and power restore to RSH North. This will happen between the hours of 00:00 and 08:00 on Saturday 02/10/10.
* These procedures are being put into action to allow essential repairs to the main power systems in RSH North. For further details on these repairs please see ‘Technical explanation of cause’.
* As part of procedures for restoration of power there will additional staff on site. They will be checking that all services have power restored and pro-actively investigating any issues reported by our monitoring system.

Sequence of events

00:00:

* Our monitoring will begin automatically executing a soft shutdown on affected servers.

01:00

* Mains power will be systematically shut down.
* Your service will not be available during this time.

05:00

* We aim to begin restoring power systematically across the site, in stages.
* Technical staff will begin pro-actively investigating and restoring service to any problematic servers based on the monitoring system.

08:00

* we aim to have all servers back online.

Technical explanation of cause

* The data centre recently installed a fourth 500kva UPS to add resilience and capacity to our power systems.
* The data centre was designed with this in mind, so the appropriate connections exist on the Schneider UPS Output Panel (that feeds all servers from a common busbar) to connect this UPS.
* During the installation of the UPS, engineers discovered there was a fault with the panel which meant the additional UPS could not be connected.
* To repair this fault, we need to electrically isolate the panel, as engineers cannot work on the panel while it is live for safety reasons.
* The manufacturer of the panel, Schneider, is sending a team of engineers to repair the panel so it can accept the fourth 500kva UPS.

Scheduling an automated shutdown on a server
We are strongly advising all unmanaged dedicated server clients to put in place on of the following methods dependent on the OS installed.

Shared/reseller hosting clients and managed dedicated server owners do NOT need to take any action as we will be powering the server down for you. VPS servers without a “VS” prefix will NOT be affected by this maintenance window and do NOT need to take any action.

Windows Server 2003

1. Click Start, All Programs, Accessories, System Tools, Scheduled Tasks
2. Double-click on Add Scheduled Task, and Click Next
3. Click Browse… to select the program to run
4. Enter C:\Windows\System32\Shutdown.exe, and click OK
5. Under “Perform this Task”, click “One Time Only”, and press Next
6. Enter the date and time to shutdown your computer: 00:00 on 02/10/2010, and click Next (BE SURE TO ALTER THE TIME IF YOU’VE CHANGED YOUR TIME ZONE!)
7. Enter your password in both fields and click Next
8. Check “Open advanced properties for this task when I click Finish”, and click Finish
9. Add “/s /t 30″ to the end of the Run box, so it looks like the following:
C:\WINDOWS\system32\shutdown.exe /s /t 30
10. Click OK, Reenter your password, and click OK
Windows Server 2008

1. Click Start, All Programs, Accessories, System Tools, Task Scheduler
2. Click “Create Basic Task” on the actions pane on the right-hand side
3. Enter “Shutdown” as the Name, and click Next
4. Under “When do you want the task to start”, select “One Time”, and click Next
5. Enter the date and time to shutdown your computer: 00:00 on 02/10/2010, and click Next (BE SURE TO ALTER THE TIME IF YOU’VE CHANGED YOUR TIME ZONE!)
6. Select “Start a Program”, and click Next
7. Enter “shutdown” (without quotes) in the “Program/Script” box, and enter “/s /t 30″ (without quotes) in the “Add Arguments” box, and click Next
8. Check “Open the Properties dialog for this task when I click Finish”, and click Finish
9. Click “Run whether the user is logged on or not”, and click OK
10. Enter your password, and click OK
Linux

Ensure you have the “at” service installed, for running tasks on a schedule. This can be installed in the following way:
Debian 5: aptitude install at
Debian 4: apt-get install at
CentOS: yum install at
Fedora: yum install at

1. Log into your server using SSH
2. Enter: “at 00:00 Oct 02″ (without quotes) and press Enter
3. After the at> prompt, enter “/sbin/shutdown -h now” (without quotes) and press Enter
4. Enter CTRL-D
5. Enter “atq” and press Enter to check that

DNS services

* During the maintenance the DNS name server services provided by us will cease to operate with the exception of ns2, ns5 and ns6.nixcontrolpanel.com name servers.
* It will not be possible to change your DNS records during the maintenance.
* Please ensure any DNS changes for holding pages are made prior to 00:00.

We realise that this is a large scale maintenance window and you will have received notification on how to claim a credit for your account despite our network SLAs having no provision for scheduled maintenance such as this.