Archive for June, 2008

Unscheduled Power Outage

Sunday, June 29th, 2008

At approximately 5:05am this morning, the Blue Square data centre where our servers are located suffered a power blackout.  We have received the following announcement from the data centre in the last 30 minutes:

We’re writing to give further information on the unscheduled outage that happened earlier on today, at approximately 5:05am.

The cause of the outage was a power failure. Power was restored earlier this morning, and over 95% of servers are back online. We have twice the number of staff in as is normal for this time of day, and we are working hard to ensure the remaining servers are brought online as quickly as possible.

We will be issuing a full RFO (Reason For Outage) to all of our clients later on in the week, when we have made a full investigation. As we’re sure you can appreciate, our current focus is to get all servers back online.”

We are obviously closely monitoring the situation and will post a full reason for outage once we receive it from the data centre. This issue has affected all servers within the data centre, not just ourselves, so please do be patient and we will issue an update as soon as possible.

Mailserver Problems

Thursday, June 12th, 2008

We have rectified an issue with both the primary and backup mail servers.

Microsoft updates were applied to both of these machines earl;y this morning which required a reboot. When the servers rebooted, the SmarterMail service failed to restart as it was conflicting with another service on the box. Our monitoring service immediately picked this up and escalated the issue.

As this was the first time these boxes had been rebooted since mid-March, the problem was not discovered/rectified earlier.

We have now taken steps to ensure that both servers respond correctly after a reboot and would like to apologise for an inconvenience caused.