Hi there, these are the last updates on the server issue - we will keep you informed every 30 - 60 minutes with new news.

LATEST @ 16:00 - SERVER IS BACK ONLINE! We are thoroughly checking it over.

Posted on: 11 Feb 2008 10:15 AM

Hi there,

Just to give you an update on the situation:

The server was rebooted early this morning and has been performing a disk check since. We hope it will be back very soon but we currently have no ETA.

Unfortunately a disk check takes as long as it takes and is forced to run every X times the server is rebooted, it is checking the consistency of approximately 900GB of data.

We will keep you in the loop & mail you back as soon as its fixed.

Posted on: 11 Feb 2008 10:40 AM

Vortex Server - further update.

Disk Consistency check has finished however the server refuses to boot, we are assessing the server for hardware failure and running another FSCK.

We will keep you updated.

Posted on: 11 Feb 2008 11:26 AM

Further update on Vortex.

The second FSCK (disk consistency check) is still running, the server will reboot automatically once that has finished - we will then be able to see the state of play in full.

At the moment it is not looking like a hardware failure, no monitoring devices have picked up any physical faults.

Please rest assured that we do understand the severity of the problem and we are doing everything within our power to make it right.

We will update you in the next 30 - 60 minutes.

Posted on: 11 Feb 2008 12:00 PM

Vortex - Further Update:

Some customers are interested in the intricacies of how the servers work, some customers just want their sites online again so I will explain where we are now and try to address the balance:

The bottom line: At this stage we’d expect to see the server back in a time frame of 1 - 6 hours depending on how many more times we have to perform a disk check.

More info for those who want it: We have been checking the server for hardware faults, none have been found so it is most likely ‘only’ a file system problem. (A file system problem can be bad, yes, however a hardware fault on (for example) a Raid Array Controller would be a LOT more problematic).

The good news is that the FSCK has gotten further without manual intervention - it is currently at 23%, the previous time it managed to get to 11% before it required intervention and failed.

We will most likely need to run FSCK 1 or 2 more times before the server will become operational, each time we run it we would expect to get it back into a better state.

I will update you again in 30 - 60 minutes (or sooner if there is news in the meantime).

Posted on: 11 Feb 2008 12:43 PM

Vortex Server - Update:

The FSCK is progressing well, we haven’t had to manually intervene and it hasn’t crashed out - I estimate this disk check should finish around 13:30 - 14:00, the moment it finishes the staff will diagnose it and I will reply you again.

Expect the next update within the next 60-90 minutes, I’ll reply earlier if the FSCK crashes out or if anything changes.

Posted on: 11 Feb 2008 02:06 PM

Vortex Situation - Update.

The FSCK is still progressing well, no intervention required - however we cannot see the % complete as that is no longer visible on the screen.

It could be as little as 1hr or as much as 6hrs.

I will reply every hour with a standard update if there is no further news, however I will reply sooner if anything else happens - I will make sure to keep you in the loop at all times.

We are on top of the situation and hope to have it worked out as soon as possible.

Posted on: 11 Feb 2008 05:08 PM

Vortex Server - Update. Total downtime 13 hours.

Hi, I am INCREDIBLY relieved to tell you that this server is back online now. Thankfully our team did their jobs impeccably behind the scenes, big thanks to them.

I am truly sorry for the outage today, we will spend the rest of the day actively monitoring this server and will work on a solution to make disk checking a lot quicker, this may require some overnight maintenance downtime but I’m sure you’ll understand that it will be worth it in the long run.

I will also work out some compensation for each customer who has written to us, I’ll arrange for each customer to have a month free.

I hope by the amount of updates Ive given that you do realise that we do care about the quality of service we offer - uptime, customer relations. It’s so important to the ethos of Evohosting, I will make steps in the future to make our public site entirely transparent so you can see what goes on behind the scenes.

Again, I’m truly sorry for the inconvenience caused and will do my best to make sure this cannot happen again.

But first I need to go make a cup of tea as I’ve been sitting here for 13 hours.

This entry was posted on Monday, February 11th, 2008 at 12:03 pm and is filed under Resolved Issues. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.