Open Mind Status

We have been today informed by the data centre where our servers are located that they will be migrating several racks of servers to their new purpose built location a short distance from the current data centre at Blue Square.

The move will involve powering down the servers in the affected racks, packing and transporting them to the new location and then re-racking the server and testing.

During the move the server will obviously be unavailable for a short period so this announcement serves as notice of intended downtime. We have servers distributed over several racks in the data centre so please refer to the schedule and affected servers below:

Rack V2 – Move Date 21st June 2009
Move Window: 22:00 for 7 hours
Downtime: < 2 hours

Affected Servers:
IS-3711: Helm control panel & MS SQL Server
IS-3712: Mail and FTP services
IS-3713: MySQL4 databases
IS-3714: ColdFusion MX7 webserver
IS-3715: ColdFusion MX6 webserver

IS-3716: Windows webserver
IS-3717: Private dedicated server
IS-3718: MySQL5 databases

Rack V4 – Move Date 22nd June 2009
Move Window: 22:00 for 7 hours
Downtime: < 2 hours

Affected Servers:
IS-7931: Private dedicated server

Rack V6 – Move Date 23rd June 2009
Move Window: 22:00 for 7 hours
Downtime: < 2 hours

Affected Servers:
IS-5965: Private dedicated server
IS-7994: Private dedicated server

Rack W1 & W6 – Move Date 24th June 2009
Move Window: 22:00 for 7 hours

Downtime: < 2 hours
Affected Servers:
IS-3587: Private dedicated server
IS-3093: Linux shared server (cPanel/WHM Users)

Rack W3 & W5 – Move Date 25th June 2009

Move Window: 22:00 for 7 hours
Downtime: < 2 hours
Affected Servers:
IS-3295: Linux shared server (DirectAdmin Users)
IS-7420: Private dedicated server

IS-7627: Private dedicated server

If your server is not listed above then a scheduled time has not been set at this point. We will be informing these users at a later stage.

If your server IP address begins with 95.154.xxx.xxx then you are already located within the new facility and will not need to be moved.

Questions about the move.
We realise that you may have questions about this move so hopefully the summaries below will help. Obviously if you have a question not covered below then please do contact the support desk.

Will my data be safe?

Yes it will. We already back up all shared hosting accounts plus dedicated servers who use our off-site service to a separate data centre in Manchester. In addition, we will be carrying out secondary backups to internal servers.

NB: Although we are taking every precaution with your data we are NOT responsible for any data loss howsoever caused. You are therefore STRONGLY advised to have up-to-date backups before the move.

Will I need to make any changes to my site?
No. You will not need to make any changes to files or databases.

Will the IP addresses change after the move?
At this time the IP addresses will remain unchanged after the move.

I don”t know if I am affected by this move?

If you are a shared hosting customer then you will be affected by the move. If you have a dedicated server with us then you need to locate your server ID in the list above for details of when your service will be re-located.

I don”t see my server listed?
Either the server has not been scheduled yet for a move in which case we will inform you at a later stage or your server is already located in the new facility.

What if I have any questions?
Please use our support desk if you have any questions about this notification:

https://www.openmindhosting.co.uk/support/

We have now received the following explanation from the data centre surrounding the circumstances of Sundays power failure. As expected, there was little the data centre could have done to prevent/for see this occurrence and we are now satisfied that the problem has been fully resolved.

We began investigation on Sunday, working with our suppliers to find the cause of the outage. The initial investigations appeared to have identified a phase detection relay which may have failed in the generator auto-changeover panel. This could have caused the panel to think there was a mains power failure when in fact there had not been.

The UPS systems worked as expected without fault, however due to the mains being forced off by the potentially failed relay, and power from the generators also locked out, the UPS batteries fully discharged after 15 minutes resulting in a critical power failure.

Engineers went to collect a new relay, along with other spares that maybe needed from the manufactures in North Yorkshire, and arrived back on the early hours of Monday morning.

We also took the step to manually disable the mains auto changeover system in order to avoid any potential further mains cuts. In the event of a real power outage on site we would operate the generator changeover procedure manually using onsite staff.

On Monday, the switch panel vendor was on site this morning and conducted several tests of the board. A decision has been taken to replace the Phase Failure Relay (PFR) for a new one.

We then completed six mains failure tests over the next 48 hours, in order to fully test the systems that have been replaced to ensure a full switch over to generator takes place in the event of a real power cut.

Today, we have collated the information so far to provider a fuller description of events. At 4.43am on Sunday morning the building lost mains power. The building suffered a power failure which caused the automatic systems to start the generator, which ran as expected. The system is then design to switch off the Air Circuit Breaker (ACB) to the mains feed, and close the ACB to the generator, thus supplying the UPS with generator power. This worked as expected and the generator took the load. Approximately 2 minutes later, the power cut ended, and power was restored switching down the generator and operating the ACB’s to switch back to mains, which all worked as planned.

Shortly after this there was a further power cut, which re-started the above sequence, in that the generator started (successfully), the mains ACB opened (successfully) and the signal was sent to close the generator ACB. This signal was sent to the ACB, however the ACB failed to close, thus meaning that the generator could not supply the UPS with power during the power cut. The UPS worked as expected and took the load. During this time the mains came back on. The ACBs have a physical and electrical interlocking system, which prevents both ACBs from being operated at the same time, thus preventing the possibility of both mains and generators feeding the load, which would result in a severe failure. Because the signals were sent to the generator ACB to close, but it never did, the interlocking systems got into a state of deadlock, where they were both stuck in an ‘open’ position, thus leaving the UPS with no feed, resulting in the batteries draining down after 15 minutes, and the system loosing the critical load.

Work started Sunday and continued on Monday to look at the electrical circuitry that controls the electrical side of the interlocks, as well as the mains phase failure relay, which detects a mains failure. This was tested as OK, however it was decided to replace critical parts with new spares to rule out any issues. After this was completed, we conducted a mains failure test which failed in the same way it did on Sunday morning. We restored mains manually at this point.

Work then commenced to look at a possible failure of the manual interlock system, which could cause the same issue. Work continued to check and replace certain parts of this system before we re-ran a mains failure situation. This test passed and the system worked as expected. We then decided to re-run the test, to ensure the issue had been fixed. The next generator test was completed, however the test failed with the same result as the first failure. Mains was again restored manually.

Due to all electrical circuitry testing and operating OK, and all manual interlocks working OK, the board vendor then started to look at a possible fault with the generator ACB. Firing pins in the ACB were tested and passed, which led the board manufacturer to suspect there was an intermittent issue with the generator ACB. This ACB is manufactured by APC/Schneider Electric/Merlin Gerin (now all the same company). As this ACB is under warranty our board vendor did not want to strip the ACB and look for issues, preferring that Merlin Gerin engineers look at this component directly.

Merlin Gerin were contacted on Monday night, and provided telephone support to the board vendors, however this was un-successful. Merlin Gerin have now agreed that an emergency support engineer needs to look at the unit in situ with the hope of swapping the failed component and re-testing. A Merlin Gerin emergency support engineer is currently on site with the required spares. Representatives from both the UPS manufacture and the control board vendor are also on site.

After the fifth mains failure test we believe Merlin Gerin have now identified the issue with the ACB. The suspected faulty component was replaced, and then there were a further three mains failure tests to validate the correct operation. The further tests have been successful and not exhibited the problematic behaviour previously identified.

We are experiencing a very high volume of calls which is preventing us from working through the outstanding issues.

Whilst we appreciate there are still problems we would ask that you either send a support ticket to support@openmindhosting.com or visit www.openmindsupport.com to let us know if you are having problems.