The following servers are relevant to this message:

IS-03093, IS-03384, IS-03587, IS-03711, IS-03712, IS-03713, IS-03714, IS-03715, IS-03716, IS-03717, IS-03718, IS-03719, IS-05965, IS-06110, IS-07420, IS-07759, IS-07931, IS-08077, IS-08313, IS-08532, IS-08817, IS-08859, IS-08985, IS-09830, IS-09906, IS-10952, IS-11329, IS-11743

On the 5th of July 2010 there will be a maintenance window to expand the capacity of the UPS systems. An additional UPS control unit and its associated batteries will be added to the existing cluster. This is to ensure the UPS system continues to provide fully redundant power backup as the site grows. This work creates an “at risk” period for your service. This maintenance is not service-affecting, but during the maintenance period services are at an increased risk.

During the maintenance period the critical load will be transferred seamlessly from the UPS to the raw mains. When the maintenance is complete it will be transferred back seamlessly.

We have confirmed with Southern Electric that they have no maintenance scheduled during this period.

This should be fully resolved within 24 hours.

This is affecting all shared/dedicated clients.

Currently we are waiting for the data centre to update us as to the nature of the problem and expcted resolution.

Updae 01:45 – The network is now back online, we are communicating with the data centre to find the cause of the outage.

The reason for the closure is to ensure that all invoices and orders are charged at the correct rate of VAT when the VAT rate changes from 15% to 17.5% at midnight tonight.

Once the billing system re-opens, users will be able to once again place new orders and access existing accounts.

Over Christmas and the New Year, our support services will be scaled down to critical support only. The changes are as follows:

Wednesday 23rd December: Normal service
Thursday 24th December: Telephone lines close at 6pm
Friday 25th December: Critical issues only
Saturday 26th December: Critical issues only
Sunday 27th December: Normal Service

Wednesday 30th December: Normal service
Thursday 31st December: Telephone lines close at 6pm
Friday 1st January: Critical issues only
Saturday 2nd January: Normal service

Where the support above is for critical issues only, we will be able to provide support ONLY if you have suffered a complete loss of service. Non-critical issues will be dealt with as soon as normal support levels resume.

The helpdesk will be monitored over the Christmas/New Year breaks and an emergency telephone system will be in place to handle CRITICAL issues only. If the call is deemed to be non-critical then you will be advised to submit a support ticket and it will be dealt with when normal support levels resume.

All servers will be monitored as normal 24/7 and on-site technicians will be available to deal with network/hardware issues as standard.

VAT Changes from 1st January 2010
===========================

As you may be aware, the rate of VAT will rise from 15% to 17.5% on January 1st 2010. There will be no change in the existing inc. VAT pricing and if you are not registered for VAT there is no action for you to take.

Business clients who are registered for VAT will see a small increase in the vat exclusive element of their invoice but as they are able to claim back the VAT paid to us and other VAT registered suppliers, the knock on effect is minimal.

The automated billing system will be paused from 30th December 2009 until 2nd January 2010 to allow us to ensure that pricing is correct and that invoices generated show the correct level of VAT. If you have any questions about the VAT change and if you are affected then please do contact the billing department.

Future Plans for 2010
================

Here at Open Mind Hosting, 2009 has proved to be an exceptional year in terms of growth and our hosting base has grown by over 150% in the last twelve months. This is a trend we fully expect to continue through 2010 and we have some very exciting plans in place to support this growth and bring new features to our clients. These include the launch of our integrated support/billing system, expansion of our VPS/Dedicated and reseller offerings, launch of our new corporate site and the possibility of launching cloud hosting which is currently under investigation.

Finally may we take the opportunity to thank you for your business over the last twelve months and wish you and your family a very Happy Christmas and prosperous New Year.

Kind Regards,

Phil Williams
Managing Director
Open Minded Solutions Limited
Tel: 08456 445 789
Int. Tel: 044 8456 445 789

Important Links:
Support Forums: http://www.openmindhosting.co.uk/forum/
Support Portal: http://www.openmindhosting.co.uk/support/
Network Status: http://www.openmindstatus.com

We will be performing maintenance which will affect your service. This will last about 20 minutes, and during this time, your service will go offline.

Date: 21 July 2009
Window: 02:00 AM for 1 Hour
Duration: < 20 minutes

This maintenance is to upgrade the IOS firmware on our RSH.North core router, from our new Maidenhead facility. Currently the network is configured with dual uplinks from our core router to the edge switches in each rack. This upgrade will enable clustering capabilities on the core router and once the core router will be configured as a cluster, it will increase the availability of the service in case of a core router failure.
This will improve redundancy across our network ensuring maximum availability for all our services.

We apologise for any inconvenience this may cause, please do not hesitate to contact us if you have any queries or questions regarding this maintenance window.

The move will involve powering down the servers in the affected racks, packing and transporting them to the new location and then re-racking the server and testing.

During the move the server will obviously be unavailable for a short period so this announcement serves as notice of intended downtime. We have servers distributed over several racks in the data centre so please refer to the schedule and affected servers below:

Rack V2 – Move Date 21st June 2009
Move Window: 22:00 for 7 hours
Downtime: < 2 hours

Affected Servers:
IS-3711: Helm control panel & MS SQL Server
IS-3712: Mail and FTP services
IS-3713: MySQL4 databases
IS-3714: ColdFusion MX7 webserver
IS-3715: ColdFusion MX6 webserver

IS-3716: Windows webserver
IS-3717: Private dedicated server
IS-3718: MySQL5 databases

Rack V4 – Move Date 22nd June 2009
Move Window: 22:00 for 7 hours
Downtime: < 2 hours

Affected Servers:
IS-7931: Private dedicated server

Rack V6 – Move Date 23rd June 2009
Move Window: 22:00 for 7 hours
Downtime: < 2 hours

Affected Servers:
IS-5965: Private dedicated server
IS-7994: Private dedicated server

Rack W1 & W6 – Move Date 24th June 2009
Move Window: 22:00 for 7 hours

Downtime: < 2 hours
Affected Servers:
IS-3587: Private dedicated server
IS-3093: Linux shared server (cPanel/WHM Users)

Rack W3 & W5 – Move Date 25th June 2009

Move Window: 22:00 for 7 hours
Downtime: < 2 hours
Affected Servers:
IS-3295: Linux shared server (DirectAdmin Users)
IS-7420: Private dedicated server

IS-7627: Private dedicated server

If your server is not listed above then a scheduled time has not been set at this point. We will be informing these users at a later stage.

If your server IP address begins with 95.154.xxx.xxx then you are already located within the new facility and will not need to be moved.

Questions about the move.
We realise that you may have questions about this move so hopefully the summaries below will help. Obviously if you have a question not covered below then please do contact the support desk.

Will my data be safe?

Yes it will. We already back up all shared hosting accounts plus dedicated servers who use our off-site service to a separate data centre in Manchester. In addition, we will be carrying out secondary backups to internal servers.

NB: Although we are taking every precaution with your data we are NOT responsible for any data loss howsoever caused. You are therefore STRONGLY advised to have up-to-date backups before the move.

Will I need to make any changes to my site?
No. You will not need to make any changes to files or databases.

Will the IP addresses change after the move?
At this time the IP addresses will remain unchanged after the move.

I don”t know if I am affected by this move?

If you are a shared hosting customer then you will be affected by the move. If you have a dedicated server with us then you need to locate your server ID in the list above for details of when your service will be re-located.

I don”t see my server listed?
Either the server has not been scheduled yet for a move in which case we will inform you at a later stage or your server is already located in the new facility.

What if I have any questions?
Please use our support desk if you have any questions about this notification:

https://www.openmindhosting.co.uk/support/

We began investigation on Sunday, working with our suppliers to find the cause of the outage. The initial investigations appeared to have identified a phase detection relay which may have failed in the generator auto-changeover panel. This could have caused the panel to think there was a mains power failure when in fact there had not been.

The UPS systems worked as expected without fault, however due to the mains being forced off by the potentially failed relay, and power from the generators also locked out, the UPS batteries fully discharged after 15 minutes resulting in a critical power failure.

Engineers went to collect a new relay, along with other spares that maybe needed from the manufactures in North Yorkshire, and arrived back on the early hours of Monday morning.

We also took the step to manually disable the mains auto changeover system in order to avoid any potential further mains cuts. In the event of a real power outage on site we would operate the generator changeover procedure manually using onsite staff.

On Monday, the switch panel vendor was on site this morning and conducted several tests of the board. A decision has been taken to replace the Phase Failure Relay (PFR) for a new one.

We then completed six mains failure tests over the next 48 hours, in order to fully test the systems that have been replaced to ensure a full switch over to generator takes place in the event of a real power cut.

Today, we have collated the information so far to provider a fuller description of events. At 4.43am on Sunday morning the building lost mains power. The building suffered a power failure which caused the automatic systems to start the generator, which ran as expected. The system is then design to switch off the Air Circuit Breaker (ACB) to the mains feed, and close the ACB to the generator, thus supplying the UPS with generator power. This worked as expected and the generator took the load. Approximately 2 minutes later, the power cut ended, and power was restored switching down the generator and operating the ACB’s to switch back to mains, which all worked as planned.

Shortly after this there was a further power cut, which re-started the above sequence, in that the generator started (successfully), the mains ACB opened (successfully) and the signal was sent to close the generator ACB. This signal was sent to the ACB, however the ACB failed to close, thus meaning that the generator could not supply the UPS with power during the power cut. The UPS worked as expected and took the load. During this time the mains came back on. The ACBs have a physical and electrical interlocking system, which prevents both ACBs from being operated at the same time, thus preventing the possibility of both mains and generators feeding the load, which would result in a severe failure. Because the signals were sent to the generator ACB to close, but it never did, the interlocking systems got into a state of deadlock, where they were both stuck in an ‘open’ position, thus leaving the UPS with no feed, resulting in the batteries draining down after 15 minutes, and the system loosing the critical load.

Work started Sunday and continued on Monday to look at the electrical circuitry that controls the electrical side of the interlocks, as well as the mains phase failure relay, which detects a mains failure. This was tested as OK, however it was decided to replace critical parts with new spares to rule out any issues. After this was completed, we conducted a mains failure test which failed in the same way it did on Sunday morning. We restored mains manually at this point.

Work then commenced to look at a possible failure of the manual interlock system, which could cause the same issue. Work continued to check and replace certain parts of this system before we re-ran a mains failure situation. This test passed and the system worked as expected. We then decided to re-run the test, to ensure the issue had been fixed. The next generator test was completed, however the test failed with the same result as the first failure. Mains was again restored manually.

Due to all electrical circuitry testing and operating OK, and all manual interlocks working OK, the board vendor then started to look at a possible fault with the generator ACB. Firing pins in the ACB were tested and passed, which led the board manufacturer to suspect there was an intermittent issue with the generator ACB. This ACB is manufactured by APC/Schneider Electric/Merlin Gerin (now all the same company). As this ACB is under warranty our board vendor did not want to strip the ACB and look for issues, preferring that Merlin Gerin engineers look at this component directly.

Merlin Gerin were contacted on Monday night, and provided telephone support to the board vendors, however this was un-successful. Merlin Gerin have now agreed that an emergency support engineer needs to look at the unit in situ with the hope of swapping the failed component and re-testing. A Merlin Gerin emergency support engineer is currently on site with the required spares. Representatives from both the UPS manufacture and the control board vendor are also on site.

After the fifth mains failure test we believe Merlin Gerin have now identified the issue with the ACB. The suspected faulty component was replaced, and then there were a further three mains failure tests to validate the correct operation. The further tests have been successful and not exhibited the problematic behaviour previously identified.

Whilst we appreciate there are still problems we would ask that you either send a support ticket to support@openmindhosting.com or visit www.openmindsupport.com to let us know if you are having problems.