@14:49: Our monitoring has been triggered by a failure on Rampage.3v0.net we are investigating and further updates will be made shortly.

@14:54: We have access to the server currently on some ports but not full access, as soon as we can get into the server to fully investigate the issue we will do so. From the current state of the server we believe there is a load spike on the server for an as yet unknown reason.

@15:33: We are now rebooting the server, if all goes well when the server comes back up we will be back online very soon, we will update again as soon as the reboot is completed.

@15:44: We have rebooted the server and the server has come back ok, but web-services are still down, we are working to restore this.

@16:28: We are having to force the server to perform an FSCK  (File System Check) as we are seeing locked files when trying to start web and database services, this may take some time, further updates will be issued hourly.  Once again we are very sorry for this outage and are doing everything we can to resolve it as soon as possible.

@17:31: The first run of FSCK has completed we are now attempting to bring the server back online and investigate the issue further. I will make a further update as soon as more information is available.

@19:33: I am VERY relieved to tell you we have now resovled the issue on Rampage and everything is now running as normal, we will continue to monitor the server over the weekend.

@ 20:13: (Tim) Thanks very much to all our customers who were affected by this outage for remaining calm during this situation, we work very hard to make sure that any server issue is dealt with as fast as possible and customers are kept in the loop AT ALL TIMES, uptime and customer satisfaction are the two most important things for us with no exception. Also thanks to the whole team who pulled together on this one, great job guys. We were working on fixing this within 45 seconds of the error occuring, unfortunately some things just take some time to fix, especially if you’re doing many disk checks.

For the non technical, green means good!

All is well again in the Evo rack. Thanks to everyone.

We will maintain a VERY close eye on Rampage over the coming days.

This entry was posted on Friday, September 5th, 2008 at 2:50 pm and is filed under Issues: Resolved. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.