Our registrar has put this notice up so I am passing it on…

Question
Why can’t I register a .co.uk domain?
Answer

9th May 2008, 5:48 AM PST

We are currently experiencing issues with the registration of .uk domains in our system. You may find that your .uk order fails with the following error: “Invalid domain contacts”.

Please be advised that this has been reported, and that our developers are working on this as a top-priority issue. Currently we do not have an ETA for resolution. We will post any updates here.

Thank-you for your patience.

We’ve dropped the prices of domain registrations for the moment.

com, net, org = £4.49 per year.

co.uk = £6.49 for 2 years.

If anyone has issues with the Vortex > Hydra migration please open a ticket, this blog post is to log what’s happened.

@ 0000 - Migration started.

@ 0015 - Databases rsync’d

@ 0130 - Sites running.

@ 0231 - MySQL won’t restart after being stopped - Fixing.

@ 0254 - MySQL issue was a file permission problem, all sorted now.

@ 0256 - Checked: http, https, pop3, smtp, webmail, ftp, dns - all services looking good.

@ 0257 - Starting final rsync of home directories

@ 0448 - So far I’ve found only a few issues with the migration, I continue to check through each main domain - please open a ticket if you have discovered a new issue.

Rsync of /home continues… I’ll do it again tomorrow night aswell.

That’s it for now - I’ll update this further if there are system wide issues.

@ 9.40 (sorry 10.40 - I forgot that the time changed last night!): We are currently experiencing DNS issues on Firestar.3v0.net. We are looking into the cause of this and will update this post as more information shortly.

@11.15 - We have now resolved the issue, this was caused by a fault in the server’s firewall which we have now fixed. Apologies for the outage and delayed response to tickets, we will work through and reply to all as fast as we can, although first priority was to resolve the issue, as always.

I will keep this post up-to-date with events occuring on Vortex this evening/morning.

@ 02:40 - The filesystem on Vortex has gone read only. We are remounting the filesystem but may need to reboot and do a manual FSCK which will take some time.

@ 03:03 - Still remounting the file system.

@ 03:18 - NOC staff will reboot the server shortly. If it goes into FSCK we could be here a while.

@ 03:57 - Unfortunately the server requires an FSCK and will be down until this completes, this was already been started a little while ago.

@ 04:32 - FSCK is progressing, usually this will take 6 hours. Best case scenario would mean the server is back at 10am.

@ 05:06 - As mentioned in a comment, I am prepping a new server (a shiny 8 x 2.5ghz one this time), once Vortex is back up we will copy all accounts to the new server, switch nameserver ips and rsync databases. (Still waiting for FSCK by the way….)

@ 05:38 - Still FSCKing away.

@ 06:16 - FSCK ‘almost’ done, it might need another.. Its currently complaining about several “multiply claimed block”, this basically involves the NOC engineer repeatedly hitting ‘Y’ until it stops.. It then reboots, we then see if it needs another FSCK or if we can bring it back up and start the migration. In the meantime I continue to prep up the new server — almost done now.

@ 06:41 - Same as before…

@ 07:01 - Same as before, new server is prepped so I’m heading down to Bluesquare now to plug it in. Am only a few minutes away - will update the blog after I’ve plugged in the new server and checked out Vortex myself (will try to report back by 08:00).

@ 08.14 - Replacement server built and Vortex 20% through a 2nd fsck which we kicked off at 7.35? we will update with latest status by 9.00

@ 08:33 - Vortex is currently at 50% of the FSCK.

@ 09:07 - 70%

@ 09:29 - Still the same, nothing further to report, presuming this is the final FSCK we’ll do then hopefully it should be finished by 11am. If we have to do another one then we’re starting to look towards mid-late afternoon.

@ 11:01 - FSCK failed at 85% - server rebooted, going for a third FSCK. Currently at 5% (restarted it 10:55)

@ 11:21 - FSCK at 13.7%

@ 11:45 - 44.5%

@ 12:13 - 70.1%

@ 12:39 - Percentage meter has gone off the screen, it is currently fixing multiple claimed blocks. It did this for a long time earlier, we simply have to see how long it takes and how far it gets.

@ 13:01 - Still the same, server is churning through data - screen hasn’t updated so hasn’t found anymore of the above as yet.

@ 13:30 - It has churned through a LOT of data, just waiting to see if it completes okay this time.

@ 14:19 - It got a lot further then failed again so I have had to kick it off again (eta to finish around 5:30pm based on previous)

@ 15:30 - Nothing to report, there is no % meter this time round, it is currently on pass 1 and has been since 14:19, I’ll keep checking the server every 5minutes and report back when there is actually something to report. Hang in there everyone.

@ 17:26 - The FSCK is on pass 1D, possibly another 60-90 minutes until it has finished.

@ 18.56 - It Lives! Vortex is now coming back online, it may take a while before databases and everything are all up and running if you see any issues please open a support ticket. We will update here when we are moving everything over to the new server. - Toby H.

The UK Team are having a well deserved break over Easter. We will not be available via the helpdesk on the following dates:

Friday 21st March
Saturday 22nd March
Sunday 23rd March
Monday 24th March

This means all departments will be closed on those dates with the exception of ‘Web Hosting Support’ which remains open 24 hours a day, 7 days a week, 365 days a year.

This doesn’t mean we’re not montoring services during this period, Toby, Jan-Erik and myself will be on-call alongside our offshore team to continue to monitor the stability of all services 24hrs a day during the Easter period.

I hope you have a good Easter and enjoy your easter eggs, I just sent out 40 of them and quite frankly I don’t want to see another one this year - they haunt me in my sleep.

Best regards,

Tim.

Sorry, we have run out of free easter eggs. We have fulfilled all orders over £29.99 with a free easter egg until now (16/03/2008 @ 16:55)

Spend £29.99 or more this month in a single payment on either a new hosting plan, renewing your existing hosting plan, upgrading your hosting or simply topping up and we’ll send you a wonderous Cadbury’s Easter Egg!
* This offer is open to both new and existing customers! *

Things you need to know:

• Order must be made & paid for between 1st March - 16th March.
• Product will be shipped 17th March first class mail.
• One egg per shipping address & customer (ie if you buy 30 hosting accounts we can send only 1 egg).
• Customer must spend over ?29.99 in one transaction.
• Offer IS redeemable for NEW AND EXISTING Customers - New orders, upgrades, top-ups, renewals!
• Offer available while stocks last - We’ll put up a blog post when we’re out.
• There is no alternative cash value if we run out.
• We can ONLY ship to Great Britain and Northern Ireland, if you live outside this area you’re welcome to an easter egg but will have to pay shipping charges. (Open a ticket with sales and we’ll work out shipping costs).
• Easter eggs are the 190 - 200 gram size.
• If your item doesn’t get to you we cannot send another one.
• We cannot guarantee which easter egg you’ll get, however if for example you absolutely hate caramel let us know and we’ll try to make a special arrangement, but no promises due to stock limitations!

We’re experiencing a load hike with Optimus, we cannot currently access it so have asked the datacenter to reboot it.

@ 16:33 - Server is now rebooted, we continue to monitor it to find out the cause of the load hike.

@ 17:57 - Load hike was caused by customer using old Mambo installation which was breached, customer has been suspended.

We are investigating issues with Vortex, this is a repeat of Monday - full report will be emailed to Vortex customers during the following week.

Latest news:

@ 00:39 Sun (Tim) - Kernel recompile completed on this and all UK servers!

@ 03:54 Sat (Tim) - We’re upgrading another server first, unfortunately post-upgrade it won’t find the lan card so we’re rescheduling for tomorrow at midnight…The other server is being worked on before Vortex.

@ 00:53 Sat (Tim) - Kernel upgrade has begun.

@ 22:25 Fri (Tim) - We’re going ahead with the Kernel upgrade as scheduled from around midnight. The server will be rebooted a couple of times during the night, nothing to worry about. I’ll post a report here if there is an issue.

@ 05:52 Fri (Tim) - I had a problem with the kernel update (its a manual rebuild rather than just typing ‘yum update’ then ‘reboot’) so I’m going to do this at 00:00 (Friday night / Sat morning) Its coming back into daylight hours and I cannot allow more downtime on this box if its avoidable so it has to wait until tomorrow night - this won’t involve much downtime (enough to reboot the server, usually 3 minutes however I’m taking no chances).

Issue is resolved for the moment, no hardware failure, server is stable, file system is okay, all sites are up, all services are running, 95% of tickets are replied, just working on the last few now — tomorrow is likely to be very busy on support tickets due to this but we’ll do our best to action your request in our usual 1 - 6 hour SLA.
Once again, I’m really sorry for the inconvenience caused - if you haven’t already claimed a free month for the downtime you can open a ticket with sales and Mark/Jan-Erik/Pete will extend your hosting period by one month. If we have already given you a free month on Monday we cannot do this again.

I’m also copying the backup files to another server so if this happens again in the next few days (It shouldn’t) then we can setup all the sites on another box and switch the nameserver IPs.

Last thing, thankyou to YOU, our customers, for being so understanding and cheering us up with your comments during this stressful time, we don’t like to let you down - myself and the team know the seriousness of this and how this has caused you significant problems. A personal thankyou also to our team for working some long hours this week too - great team, good job guys.

@ 04:13 Fri (Tim) - Kernel update is progressing well, it has been compiling for sometime now.

@ 02:17 Fri (Tim) - IT LIVES! However we’re not quite out of the woods yet. We are to complete a manual Kernel update as a precaution. This is happening right now, we’ll then need to reboot the box. If nothing bad happens then we’ll have it back by 03:30.

@ 01:02 Fri (Tim) - We’re checking on the server each 5 minutes (I’d make Toby sit in there constantly but apparently its against Health and Safety due to extreme noise). Currently the server is on Pass 1D (thats good) of the FSCK check and hasn’t bailed. Hopefully shouldn’t be much longer. While we’re waiting we’re tidying up the support desk (150+ open tickets due to this). Once the FSCK is done we’re performing a kernel upgrade and rebooting the server again. ETA is now 03:00 - 06:00 — we’ll do our ABSOLUTE BEST to get it back online for 06:00.

@ 23:21 Thur (Tim) - Same as an hour ago, but an hour closer. Just playing the waiting game right now. Some people might be interested to actually SEE what we’re seeing … For those who want to see, here you go:

@ 22:21 Thur (Tim) - Disk check failed, performing manual disk check - we started the manual one at 22:06 and expect it to finish at approx 01:00. Uptime ETA 01:00 - 06:00.

@ 21:01 Thur (Tim) - Sitting here in the Bluesquare datacentre chill out room with Toby waiting for the disk check to finish - we’re hoping it’ll be done by midnight although ETA as Mark said below is still the same.

@ 19:17 Thur (Mark) - We have discovered that there is no hardware failure, we are currently running another mandatory disk check on the hard drives. (Current ETA: Midnight - 6am as long as there are no further errors).

@ 16:49 Thur (Tim) - We’re still looking at the ETA’s below as Dell Diags havent finished.

@ 15:56 Thur (Tim) - Disk check finished, Dell diagnostics is running, likely until 6pm.

1550 - 1800 (APPROX) Dell Diagnostics will run.

When diagnostics finish —- if no faults found then we reboot server and see if it wants another disk check or wants to start up. Disk check will then take another 4 - 6 hours.

If faults are found then we call Dell and await engineer arrival.

@ 15:42 Thur (Tim) - Still the same, see the 13:33 update for ETA.

@ 14:42 Thur (Tim) - Still the same as the previous post due to disk check. Nothing new to report as yet.

@ 13:33 Thur (Tim) - Disk check still progressing, we’re estimating approx 6pm before it finishes, Dell are on call also to swap out any hardware - 1- 4 Hr SLA for them to arrive on scene. Best case scenario will mean server is online 4pm - 6pm, worst case 10pm - tomorrow morning.

@ 12:06 Thur (Tim) - We saw some IO Errors before it shutdown, possible harddisk failure. It is running a diskcheck now which (as you will know from Monday) takes 4 - 6 hours.

We are looking into intermittent reboots on the Firestar server, multiple outages of 2 - 10 minutes have been reported this morning.

@ 00:39 Sun (Tim) - Kernel recompile completed on this and all UK servers!

@ 03:55 Sat (Tim) - Post-upgrade kernel does not find lan card - we are working on solution and will try again tomorrow at 00:00.

@ 00:53 Sat (Tim) - Kernel upgrade has begun.

@ 22:38 (Tim) - We’re going ahead with the Kernel upgrade as scheduled from around midnight. The server will be rebooted a couple of times during the night, nothing to worry about. I’ll post a report here if there is an issue.

@ 06:08 (Tim) - Had issue with Kernel upgrade, cannot afford more downtime on this box during peak (06:00 - 23:00) so am leaving it until 00:00 tomorrow along with Vortex.

@ 22:20 (Tim) - Firestar just rebooted again - we will take it down at 00:00 for a little while to perform a kernel upgrade.

@ 16:19 (Tim) - We found a customer running a process he shouldn’t, customer has now been suspended. This is very likely the cause of the problem, we continue to check and monitor the situation.

@ 15:35 (Tim) - Firestar just rebooted itself, a technician is running Dell Diagnostics on it. I’ll update you ASAP.