Open Mind Status

The London Internet Exchange (LINX) is currently suffering from a network issue that is affecting traffic. We immediately shut down our connections to LINX until they resolve this problem. Traffic has been rerouted over our other peering points until the situation is resolved.

Date: 16/03/2010
Time: 22:30
Effect on service: Increased latency for traffic that would normally reach us via LINX
Duration: Ongoing

The only information that we have from LINX at this time is that they have acknowledged an issue and are investigating the problem. This problem has only affected traffic that reaches our network via LINX, which due to our diverse peering points with many of the major ISPs has meant minimal impact on incoming and outgoing traffic. The Open Mind Hosting network has remained stable throughout.

Traffic that would normally reach us via LINX will now be rerouted and connect to us via our other peering points. This may result in a slight increase in latency for that traffic, if it involves additional hops. We are now waiting on LINX for updates as to the nature of the problem, likely resolution time and ultimately for an all clear notification.

If anyone is experiencing problems with their service, please do not hesitate to contact us.

At 13:10 we received an update from LINX to say that their emergency maintenance steps had restored stability and they were advising all peers to re-establish sessions. We followed that advice and began restoring our LINX sessions. This process was completed at 13:48, however a small number of our peers have kept their sessions administratively down. This is their choice but may impact traffic if you would normally cross their network to reach us. We have been monitoring the situation this afternoon and have seen traffic levels return to normal. There have been no further reports of problems.

LINX are planning further ‘tidy up’ maintenance tonight, but according to their descriptions of the works, they do not affect the Open Mind Hosting connections. We are hopeful that this is a conclusion to this incident and that LINX will maintain the resumed normal service.

If any one is still experiencing difficulties please do not hesitate to contact us. We will do what we can to investigate any issues and resolve them for you if we can. However, please bear in mind that the cause of all the problems over the last 24 hours has been outside the Open Mind Hosting network and we may not be able to act for you.

LINX are still reporting that they have ongoing problems and do not know when these will be resolved. For this reason we have started disabling all peering via LINX and traffic will be rerouted via other connections.

This should resolve any outstanding problems with traffic flow via LINX, but this process will take some time to complete. We cannot just switch off a connection like LINX without ensuring that the traffic is properly handled, or there will be negative effects for all clients. Once this process is complete, we will provide further updates.

At 20:10 the issue at LINX resurfaced and traffic via this connection was no longer flowing. Once again this was outside our sphere of control and it was outside the Open Mind Hosting network. Traffic flow was restored at around 20:30.

We are still waiting to hear from LINX regarding the cause of these problems, so we cannot be certain they will not resurface again.

We will keep you updated with more information as soon as we have it.

We have been seeing a return of traffic flow via LINX since around 17:40 and it appears to be holding steady.

There has been no announcement from LINX yet. As soon as we have any further information we will issue a further update. As the connection appears to be operating correctly again this is likely to be tomorrow morning. We are still seeing no issues at all with the Open Mind Hosting network and all traffic coming via our other peers is flowing as normal. The issue seems to be within LINX’s infrastructure.

It seems that the London Internet Exchange (LINX) are currently experiencing problems. Our connections to LINX are no longer carrying traffic and this is having an impact on some clients who would normally cross LINX to reach us. We do not currently have information on the precise cause but the issue has been reported by other IPs connecting at LINX.

We will provide more information as soon as we have it, but unfortunately it seems that the problem is outside our network and outside our control.

Date: 09/12/2009
Time: 17:20
Effect: Increased latency and packet loss for some routes into the our network

We have now received the all clear from the data centre:

Please be advised that we are now able to officially provide an all-clear notification to all clients regarding the network disturbances, outages, interruptions to service and emergency maintenance experienced by clients in RSH-North since Midday, Thursday 24th September.

To clarify, normal service has been resumed for the large majority of clients since before 3am however we are now able to issue a full all clear.

The NAS service continues to experience problems and this will be dealt with separately.

A full report will be issued as soon as possible but in the meantime all clients are encouraged to contact us by telephone immediately if they are experiencing any problems at this time.