Planned Linux Server Migrations

May 19th, 2011

In line with our server replacement program we have made the decision to retire the current Linux servers and replace them with higher spec machines in order to continue providing our customers with the high level of service they would expect. The new servers have now been provisioned and we are planning to move all Linux shared/reseller hosting accounts between Friday and Sunday this week.

The accounts will be migrated for clients and will happen at off peak times to minimise any impact to your services. There is nothing you need do as we will migrate all your data for you but we do recommend having an up to date backup of all your files and databases. During the migration you or your visitors may see a message indicating that the move is in progress but this is normal and will clear itself within about 60 minutes.

A further notification will be sent out and also posted on our status board during the move.

If you have any questions or concerns then please do not hesitate t get in touch with the support team.

Upgrade of Primary Mailserver (IS-11329)

March 28th, 2011

We will shortly begin upgrading the primary mail server on IS-11329 to the lastest stable release of SmarterMail.

During this time, mail services will be unavailable to users and mail will be redirected to our offsite mail server.

Once the upgrade is complete, access will be restored for users and mail sent to the backup server will be delievered as normal.

We expect the upgrade to take approximately 15 minutes…

18:47 – The upgrade process has now commenced…

19:04 – The upgrade has now been completed and mail services have returned to normal.

For more information on SmarterMail v8, please visit the SmarterTools site:

http://www.smartertools.com/smartermail/mail-server-software.aspx

Power Outage for VPS Host Node IS-13782

March 23rd, 2011

At approximately 8:14am this morning, on site technicians detected a power outage for server IS-13782. On investigation it was found a power cable had become faulty which was subsequently replaced.

Power to the server was restored shortly afterwards and VPS servers are now back online although some may take a little longer to boot than others.

We apologise for the inconvenience and should your VPS not be responding as normal, please contact the support team.

Connectivity Issues at Linx

March 15th, 2011

Some users may be experiencing connectivity issues to our services due to an outage at LINX based in London.

Date: 14/03/2011

Time: 19:00

Duration: < 50 minutes

Affected Service(s): IS-03093, IS-03384, IS-03587, IS-03711, IS-03712, IS-03713, IS-03714, IS-03715, IS-03716, IS-03717, IS-03718, IS-03719, IS-05965, IS-07420, IS-07931, IS-08313, IS-08532, IS-08817, IS-08859, IS-08985, IS-11329, IS-12110, IS-12290, IS-12441, IS-12812, IS-13332, IS-13333, IS-13468, IS-13520, IS-13740, IS-13782, IS-13880, VS-01020, VS-01143, VS-01834

LINX have made an announcement that they are experiencing problems. This was affecting any traffic coming to the Open Mind Hosting network via LINX.  Any traffic transiting to Open Mind Hosting from any other peering points would have been unaffected

Our network engineers responded quickly and have shutdown our sessions at LINX. Traffic has been routed to alternative peers and should be flowing normally. Subsequent announcements from LINX have suggested that they have isolated the cause and are looking to fix it, but the Open Mind Hosting sessions at LINX will remain offline until further notice.

It is important to stress that the connectivity issues are outside of our control and should you be affected you should contact your Internet Service Provider for assistance.

If anyone is still experiencing any adverse effects please contact us immediately. We will continue to monitor the situation and will restore traffic via LINX only when we are confident they have fixed the problems within their network.

Network Issues

November 17th, 2010

We are currently experiencing a network issue affecting all dedicated servers and shared/reseller hosting plans.

It is not clear at this time where the issue lies but engineers are on site and are investigating as a high priority.

Update 09:06 – Resolution and RFO:

Date: 17/11/2010

Time: 07:40

Outage Type: Connectivity Interruption

Duration: < 55 minutes

Affected Service(s): IS-03093, IS-03384, IS-03587, IS-03711, IS-03712, IS-03713, IS-03714, IS-03715, IS-03716, IS-03717, IS-03718, IS-03719, IS-05965, IS-06110, IS-07420, IS-07931, IS-08313, IS-08532, IS-08817, IS-08859, IS-08985, IS-09830, IS-11329, IS-12110, IS-12290, IS-12441, IS-12812, VS-01020, VS-01143, VS-01695

At 08:00 this morning the cluster experienced similar issues to those on the 18/11/10. This again only affected a proportion of the traffic coming through RSH North side. However, those affected will have seen packet loss and increased latency.

Having identified the issue we have taken preventative steps. The cluster is now operating correctly and in a stable manner. We will provide further information as it is available but services are operating within normal parameters again. We apologise for the recurrence of this issue and are continuing to communicate with Cisco.

If you are experiencing any problems as a result of this please do not hesitate to contact our support team immediately and we will look into this for you.

ColdFusion MX7 Server – IS-3714

November 10th, 2010

We are currently experiencing a minor issue with this server and users may experience a loss of connectivity. We are currently working on the issue and will update this post when it is resolved.

12:15 – This has now been fully resolved. In addition we have upgraded PHP to the latest stable release.

Network Issues

October 18th, 2010

The primary data centre is currently having some network connectivity issues and this is affecting a small proportion of our servers causing timeouts and unresponsiveness.

We are monitoring the situation and will keep users updated with more information as and when we have it.

Update 10:38

Date: 18/10/2010
Time: 09:20
Duration so far: 1 hour, 18 minutes
Affected Service(s): IS-03093, IS-03384, IS-03587, IS-03711, IS-03712, IS-03713, IS-03714, IS-03715, IS-03716, IS-03717, IS-03718, IS-03719, IS-05965, IS-06110, IS-07420, IS-07931, IS-08077, IS-08313, IS-08532, IS-08817, IS-08859, IS-08985, IS-09830, IS-09906, IS-11329, IS-11743, IS-11959, IS-12110, IS-12290, IS-12441, IS-12812

There is currently a network issue affecting RSH North. At this time we believe this to be caused by a network routing problem at one of our peering connections. This problem is only affecting a proportion of traffic via RSH North.

Update 11:17

The issue affecting RSH North is ongoing. A proportion of the traffic into RSH North will currently be experiencing increased latency and packet loss at varying levels. In some instances this may appear to be complete connectivity loss.

We are investigating the cause and will provide further updates as soon as we possibly can.

Update 13:24:

The initial problem with a peering point that occurred earlier this morning has led to a problem within the VSS routing cluster that services RSH North. Our network team quickly eliminated as many causes as possible. This issue was then escalated to Cisco and we are currently involved in joint investigation to try and discern the underlying problem. As soon as we have any progress from this work we will inform you immediately.

Update 13:59:

Network connectivity has now been restored across all the servers however we are waiting fr an official all clear from the data centre before showing this issue as resolved.

Network Power Systems Maintenance

September 16th, 2010

On 2nd October 2010 between the hours of midnight and 8am, the data centre housing our shared, reseller and dedicated server clients will be carrying urgent and substantial maintenance to the primary power supply system and this is advanced notification of the expected downtime schedule that will affect the majority of our client services. A full breakdown, schedule and client action is given below:

Summary

* The data centre will be enacting our procedures for a controlled power down and power restore to RSH North. This will happen between the hours of 00:00 and 08:00 on Saturday 02/10/10.
* These procedures are being put into action to allow essential repairs to the main power systems in RSH North. For further details on these repairs please see ‘Technical explanation of cause’.
* As part of procedures for restoration of power there will additional staff on site. They will be checking that all services have power restored and pro-actively investigating any issues reported by our monitoring system.

Sequence of events

00:00:

* Our monitoring will begin automatically executing a soft shutdown on affected servers.

01:00

* Mains power will be systematically shut down.
* Your service will not be available during this time.

05:00

* We aim to begin restoring power systematically across the site, in stages.
* Technical staff will begin pro-actively investigating and restoring service to any problematic servers based on the monitoring system.

08:00

* we aim to have all servers back online.

Technical explanation of cause

* The data centre recently installed a fourth 500kva UPS to add resilience and capacity to our power systems.
* The data centre was designed with this in mind, so the appropriate connections exist on the Schneider UPS Output Panel (that feeds all servers from a common busbar) to connect this UPS.
* During the installation of the UPS, engineers discovered there was a fault with the panel which meant the additional UPS could not be connected.
* To repair this fault, we need to electrically isolate the panel, as engineers cannot work on the panel while it is live for safety reasons.
* The manufacturer of the panel, Schneider, is sending a team of engineers to repair the panel so it can accept the fourth 500kva UPS.

Scheduling an automated shutdown on a server
We are strongly advising all unmanaged dedicated server clients to put in place on of the following methods dependent on the OS installed.

Shared/reseller hosting clients and managed dedicated server owners do NOT need to take any action as we will be powering the server down for you. VPS servers without a “VS” prefix will NOT be affected by this maintenance window and do NOT need to take any action.

Windows Server 2003

1. Click Start, All Programs, Accessories, System Tools, Scheduled Tasks
2. Double-click on Add Scheduled Task, and Click Next
3. Click Browse… to select the program to run
4. Enter C:\Windows\System32\Shutdown.exe, and click OK
5. Under “Perform this Task”, click “One Time Only”, and press Next
6. Enter the date and time to shutdown your computer: 00:00 on 02/10/2010, and click Next (BE SURE TO ALTER THE TIME IF YOU’VE CHANGED YOUR TIME ZONE!)
7. Enter your password in both fields and click Next
8. Check “Open advanced properties for this task when I click Finish”, and click Finish
9. Add “/s /t 30″ to the end of the Run box, so it looks like the following:
C:\WINDOWS\system32\shutdown.exe /s /t 30
10. Click OK, Reenter your password, and click OK
Windows Server 2008

1. Click Start, All Programs, Accessories, System Tools, Task Scheduler
2. Click “Create Basic Task” on the actions pane on the right-hand side
3. Enter “Shutdown” as the Name, and click Next
4. Under “When do you want the task to start”, select “One Time”, and click Next
5. Enter the date and time to shutdown your computer: 00:00 on 02/10/2010, and click Next (BE SURE TO ALTER THE TIME IF YOU’VE CHANGED YOUR TIME ZONE!)
6. Select “Start a Program”, and click Next
7. Enter “shutdown” (without quotes) in the “Program/Script” box, and enter “/s /t 30″ (without quotes) in the “Add Arguments” box, and click Next
8. Check “Open the Properties dialog for this task when I click Finish”, and click Finish
9. Click “Run whether the user is logged on or not”, and click OK
10. Enter your password, and click OK
Linux

Ensure you have the “at” service installed, for running tasks on a schedule. This can be installed in the following way:
Debian 5: aptitude install at
Debian 4: apt-get install at
CentOS: yum install at
Fedora: yum install at

1. Log into your server using SSH
2. Enter: “at 00:00 Oct 02″ (without quotes) and press Enter
3. After the at> prompt, enter “/sbin/shutdown -h now” (without quotes) and press Enter
4. Enter CTRL-D
5. Enter “atq” and press Enter to check that

DNS services

* During the maintenance the DNS name server services provided by us will cease to operate with the exception of ns2, ns5 and ns6.nixcontrolpanel.com name servers.
* It will not be possible to change your DNS records during the maintenance.
* Please ensure any DNS changes for holding pages are made prior to 00:00.

We realise that this is a large scale maintenance window and you will have received notification on how to claim a credit for your account despite our network SLAs having no provision for scheduled maintenance such as this.

Mailserver Upgrade

August 25th, 2010

23:00 – The upgrade has now commenced and mail services have been halted. Mail is currently being delivered to our backup mail server.

23:10 – The upgrade has been completed and access restored. We’re checking everything has gone through smoothly but please contact the help desk should you have any questions or problems.

Mailserver Maintenance

August 25th, 2010

This evening we shall be upgrading the primary mailserver to the latest release of SmarterMail. This work will take about 30 minutes to complete and will occur between 10pm and midnight tonight.

Mail services will be stopped during the upgrade but mail will continue to be delivered to our backup mailserver.

Once completed, users will have access to a completely revamped webmail interface as well as many other new options. For full details please visit the SmarterTools site:

http://www.smartertools.com/smartermail/mail-server-software.aspx