We are currently experiencing a major outage at the data centre our newtork is located within.
This is affecting all shared/dedicated clients.
Currently we are waiting for the data centre to update us as to the nature of the problem and expcted resolution.
Updae 01:45 – The network is now back online, we are communicating with the data centre to find the cause of the outage.
At 13:10 we received an update from LINX to say that their emergency maintenance steps had restored stability and they were advising all peers to re-establish sessions. We followed that advice and began restoring our LINX sessions. This process was completed at 13:48, however a small number of our peers have kept their sessions administratively down. This is their choice but may impact traffic if you would normally cross their network to reach us. We have been monitoring the situation this afternoon and have seen traffic levels return to normal. There have been no further reports of problems.
LINX are planning further ‘tidy up’ maintenance tonight, but according to their descriptions of the works, they do not affect the Open Mind Hosting connections. We are hopeful that this is a conclusion to this incident and that LINX will maintain the resumed normal service.
If any one is still experiencing difficulties please do not hesitate to contact us. We will do what we can to investigate any issues and resolve them for you if we can. However, please bear in mind that the cause of all the problems over the last 24 hours has been outside the Open Mind Hosting network and we may not be able to act for you.
LINX are still reporting that they have ongoing problems and do not know when these will be resolved. For this reason we have started disabling all peering via LINX and traffic will be rerouted via other connections.
This should resolve any outstanding problems with traffic flow via LINX, but this process will take some time to complete. We cannot just switch off a connection like LINX without ensuring that the traffic is properly handled, or there will be negative effects for all clients. Once this process is complete, we will provide further updates.
At 20:10 the issue at LINX resurfaced and traffic via this connection was no longer flowing. Once again this was outside our sphere of control and it was outside the Open Mind Hosting network. Traffic flow was restored at around 20:30.
We are still waiting to hear from LINX regarding the cause of these problems, so we cannot be certain they will not resurface again.
We will keep you updated with more information as soon as we have it.
We have been seeing a return of traffic flow via LINX since around 17:40 and it appears to be holding steady.
There has been no announcement from LINX yet. As soon as we have any further information we will issue a further update. As the connection appears to be operating correctly again this is likely to be tomorrow morning. We are still seeing no issues at all with the Open Mind Hosting network and all traffic coming via our other peers is flowing as normal. The issue seems to be within LINX’s infrastructure.
It seems that the London Internet Exchange (LINX) are currently experiencing problems. Our connections to LINX are no longer carrying traffic and this is having an impact on some clients who would normally cross LINX to reach us. We do not currently have information on the precise cause but the issue has been reported by other IPs connecting at LINX.
We will provide more information as soon as we have it, but unfortunately it seems that the problem is outside our network and outside our control.
Date: 09/12/2009
Time: 17:20
Effect: Increased latency and packet loss for some routes into the our network
19:54 – The sever move will begin shortly and all services will be stopped on the old server until after the move is complete
20:12 – Accounts are currently being backed up on the old server
21:01 – Admin accounts are now being restored on the new server
21:17 – Reseller accounts are now being restored
21:20 – User accounts with shared IPs are now being restored
21:39 – A minor issue has been encountered and we are currently rebuilding PHP/Apache
21:46 – Currently restoring fixed IP accounts
22:18 – All user accounts now restored, name server IP changes now in progress
This is advanced notice of the maintenance work to be carried out on the DirectAdmin Linux server. If you use DirectAdmin to administer your site then you will be affected by this move.
Our monitoring system has picked up a minor issue with this server so as a precaution, all accounts will be moved to a new server.
The work has been scheduled for Monday 12th October from 8pm
Further updates will be posted closer to the time.
We have now received the all clear from the data centre:
Please be advised that we are now able to officially provide an all-clear notification to all clients regarding the network disturbances, outages, interruptions to service and emergency maintenance experienced by clients in RSH-North since Midday, Thursday 24th September.
To clarify, normal service has been resumed for the large majority of clients since before 3am however we are now able to issue a full all clear.
The NAS service continues to experience problems and this will be dealt with separately.
A full report will be issued as soon as possible but in the meantime all clients are encouraged to contact us by telephone immediately if they are experiencing any problems at this time.
Currently all servers are offline as the data centre are once again attempting to resolve the issues with the core server.
Further updates will be posted as we have them
UPDATE: 21:37 – Approximately 40% of our servers are now coming back online…
UPDATE – 01:33 – All servers up and running. A full RFO will be issued shortly…
UPDATE – 02:11 – After removing the VSS cluster technology from the RSH North zone routers and restoring the routers to a similar configuration to the ones which served RHC stably for over two years, considerable progress was made to restore connectivity for 90% of clients. However, service is not stable and clients will see differing levels of connectivity ranging from complete loss of service to severely degraded.
Obviously we are continuing to work on this with Cisco.
Please be assured that further updates will be provided as and when we have them.