Saturday 18 August 2012

Torbay Bus Routes: Our Current Server Outage

Dear Torbay Bus Routes User,

Yesterday (Friday, 17th August, 2012) afternoon, our website and e-mails went down. Our server is usually reliable, so it was unfortunate when the website went down in the afternoon. I, Allan Wagner, am responsible for the websites reliability and I must start by apologising.


Since I was made aware of the issue, I have been kept informed about what our data centre is doing to resolve the issue, and even though our main server won’t be up for the next few days, I have organised for a replacement server to be brought in within 24 hours (as of 22:00, Saturday 18th August).


Once again, I am deeply sorry about this outage, and as soon as normal service is restored, I will be looking at ways to avoid this from occurring again.


I also wanted to provide more detail for the more technically minded users, who might be interested in the suspected cause (not completely confirmed yet). Between 18:00 to 19:00 yesterday evening, I found out that the Torbay Bus Routes website was coming up with a 404 (Server Not Found) error. Straight away, I got in touch with our providers asking them to look into this issue. It turned out that our service had in fact been down since around 16:00, or possibly earlier.


Later that evening, I was updated that the motherboard in our server was being replaced due to a suspected faulty Ethernet port. However, upon the data centre testing the repair, they had the wrong motherboard, and the kernel would not boot up the server. The data centre are now trying to source the correct part. Unfortunately, with it being the weekend, this could delay the replacement hunt further!


Thankfully, backups are kept. So we are now setting up a replacement server and we are changing over the DNS (domain name server) settings to point to our new temporary server. The moral of the story is: always keep a back up!


Technology is fantastic, when it works. But all we can do is learn what went wrong here, and make sure this problem will not happen again. 


We can’t wait to return to the World Wide Web tomorrow, and we thank you for your understanding and patience!

From the Director, Elliot Massey:

I'd also like to apologise for this outage. We understand just how frustrating it is for our users, and it's even more frustrating for us knowing we can't do anything about it. 

This has also delayed our development of MyRoute, our latest innovation, but we are just thankful that the server did not go down before uploading our new bus map (although no one can see it now!). I personally initiated the DNS change, and I can assure our users that it is working its way through the system.

We're all hoping for it to be back up soon, and everything's crossed for a quick and painless DNS change.

With very best wishes from Allan and I.

If anyone requires bus times or information, please tweet us (@torbaybuses), or write on our Facebook wall.

No comments:

Post a Comment

Comment