@ 9.40 (sorry 10.40 - I forgot that the time changed last night!): We are currently experiencing DNS issues on Firestar.3v0.net. We are looking into the cause of this and will update this post as more information shortly.

@11.15 - We have now resolved the issue, this was caused by a fault in the server’s firewall which we have now fixed. Apologies for the outage and delayed response to tickets, we will work through and reply to all as fast as we can, although first priority was to resolve the issue, as always.

I will keep this post up-to-date with events occuring on Vortex this evening/morning.

@ 02:40 - The filesystem on Vortex has gone read only. We are remounting the filesystem but may need to reboot and do a manual FSCK which will take some time.

@ 03:03 - Still remounting the file system.

@ 03:18 - NOC staff will reboot the server shortly. If it goes into FSCK we could be here a while.

@ 03:57 - Unfortunately the server requires an FSCK and will be down until this completes, this was already been started a little while ago.

@ 04:32 - FSCK is progressing, usually this will take 6 hours. Best case scenario would mean the server is back at 10am.

@ 05:06 - As mentioned in a comment, I am prepping a new server (a shiny 8 x 2.5ghz one this time), once Vortex is back up we will copy all accounts to the new server, switch nameserver ips and rsync databases. (Still waiting for FSCK by the way….)

@ 05:38 - Still FSCKing away.

@ 06:16 - FSCK ‘almost’ done, it might need another.. Its currently complaining about several “multiply claimed block”, this basically involves the NOC engineer repeatedly hitting ‘Y’ until it stops.. It then reboots, we then see if it needs another FSCK or if we can bring it back up and start the migration. In the meantime I continue to prep up the new server — almost done now.

@ 06:41 - Same as before…

@ 07:01 - Same as before, new server is prepped so I’m heading down to Bluesquare now to plug it in. Am only a few minutes away - will update the blog after I’ve plugged in the new server and checked out Vortex myself (will try to report back by 08:00).

@ 08.14 - Replacement server built and Vortex 20% through a 2nd fsck which we kicked off at 7.35? we will update with latest status by 9.00

@ 08:33 - Vortex is currently at 50% of the FSCK.

@ 09:07 - 70%

@ 09:29 - Still the same, nothing further to report, presuming this is the final FSCK we’ll do then hopefully it should be finished by 11am. If we have to do another one then we’re starting to look towards mid-late afternoon.

@ 11:01 - FSCK failed at 85% - server rebooted, going for a third FSCK. Currently at 5% (restarted it 10:55)

@ 11:21 - FSCK at 13.7%

@ 11:45 - 44.5%

@ 12:13 - 70.1%

@ 12:39 - Percentage meter has gone off the screen, it is currently fixing multiple claimed blocks. It did this for a long time earlier, we simply have to see how long it takes and how far it gets.

@ 13:01 - Still the same, server is churning through data - screen hasn’t updated so hasn’t found anymore of the above as yet.

@ 13:30 - It has churned through a LOT of data, just waiting to see if it completes okay this time.

@ 14:19 - It got a lot further then failed again so I have had to kick it off again (eta to finish around 5:30pm based on previous)

@ 15:30 - Nothing to report, there is no % meter this time round, it is currently on pass 1 and has been since 14:19, I’ll keep checking the server every 5minutes and report back when there is actually something to report. Hang in there everyone.

@ 17:26 - The FSCK is on pass 1D, possibly another 60-90 minutes until it has finished.

@ 18.56 - It Lives! Vortex is now coming back online, it may take a while before databases and everything are all up and running if you see any issues please open a support ticket. We will update here when we are moving everything over to the new server. - Toby H.

The UK Team are having a well deserved break over Easter. We will not be available via the helpdesk on the following dates:

Friday 21st March
Saturday 22nd March
Sunday 23rd March
Monday 24th March

This means all departments will be closed on those dates with the exception of ‘Web Hosting Support’ which remains open 24 hours a day, 7 days a week, 365 days a year.

This doesn’t mean we’re not montoring services during this period, Toby, Jan-Erik and myself will be on-call alongside our offshore team to continue to monitor the stability of all services 24hrs a day during the Easter period.

I hope you have a good Easter and enjoy your easter eggs, I just sent out 40 of them and quite frankly I don’t want to see another one this year - they haunt me in my sleep.

Best regards,

Tim.

Sorry, we have run out of free easter eggs. We have fulfilled all orders over £29.99 with a free easter egg until now (16/03/2008 @ 16:55)

Spend £29.99 or more this month in a single payment on either a new hosting plan, renewing your existing hosting plan, upgrading your hosting or simply topping up and we’ll send you a wonderous Cadbury’s Easter Egg!
* This offer is open to both new and existing customers! *

Things you need to know:

• Order must be made & paid for between 1st March - 16th March.
• Product will be shipped 17th March first class mail.
• One egg per shipping address & customer (ie if you buy 30 hosting accounts we can send only 1 egg).
• Customer must spend over ?29.99 in one transaction.
• Offer IS redeemable for NEW AND EXISTING Customers - New orders, upgrades, top-ups, renewals!
• Offer available while stocks last - We’ll put up a blog post when we’re out.
• There is no alternative cash value if we run out.
• We can ONLY ship to Great Britain and Northern Ireland, if you live outside this area you’re welcome to an easter egg but will have to pay shipping charges. (Open a ticket with sales and we’ll work out shipping costs).
• Easter eggs are the 190 - 200 gram size.
• If your item doesn’t get to you we cannot send another one.
• We cannot guarantee which easter egg you’ll get, however if for example you absolutely hate caramel let us know and we’ll try to make a special arrangement, but no promises due to stock limitations!