Server is down (12-13 Feb 2013)

ArithonUK · February 13, 2013, 08:33:56 PM

The server was off line last night, so I just played elsewhere, but it is down again tonight. Is there a problem? Can we get it restarted?

Thanks.

Gone_Away · February 13, 2013, 09:02:43 PM

[TABLE=class: Table]
[TR]
[TD=class: FieldName]Name: [/TD]
[TD=class: FieldValue] dMw UK:City Maps, Adaptive Tix, FREE.BEER

[/TD]
[/TR]
[TR]
[TD=class: FieldName] Connection Info: [/TD]
[TD=class: FieldValue] 81.19.216.82:25200 [/TD]
[/TR]
[TR]
[TD=class: FieldName] Status: [/TD]
[TD=class: FieldValue] Unknown[/TD]
[/TR]
[/TABLE]

At least I could get into the control panel.. :/

"apologies, we had a raid failure on our master game panel server, andys been rushing around all today trying to repair the raid and re-image the system to get the game panel back up asap but weve run into some sql/software issues that are being diagnosed
we will make an announcement as soon as were back live, if you need help accessing your server for anything for the time being log a support ticket"

ArithonUK · February 14, 2013, 08:19:50 PM

Still down.

Gone_Away · February 14, 2013, 08:58:57 PM

From the GamingDeluxe Forums:

I don't know a great deal. I'm not GD staff, but merely a customer who has been asked to fulfil a couple of extra functions on this forum. However, I have been paying close attention to recent issues.

I'm sure that Gaming Deluxe will make a full announcement to customers, as soon as they feel able. A brief announcement was made but had to be taken down being premature. We also know that as part of the failure, email communication systems related to customer controls have also been affected, which has compounded the issues of letting us all know what is happening.

It is unfortunately clear that the Game Control Panel systems we were used to cannot easily be restored. We know there have been hardware failures and even from my limited experience of a simple RAID office server, restoring systems from one hardware configuration onto any other system a challenge. From the limited part of the control panels we can now see it is apparent GD are working on bringing a new control panel software online, rather than trying to resurrect an old one. But we also know that this is advancing a programme for the game panel which was not quite ready. Add to that the probable need to add many of the controlled server and customer details manually (image how many of them there are), and one can see this will take a while. Inevitably the GD staff are seeing a few teething problems during this process frustrating their progress.

If your game servers are not working, or you have other urgent issues needing to be addressed, then please do raise a support ticket. The billig panels do still work normally. Don't chase and bump tickets, that with many systems actually puts you to the bottom of a queue when you add an update. Inevitably, as GD are prioritising the game panel for the benefit of everyone, support may have less resources, compounded by a higher workload. I can only suggest patience.

Please don't get me wrong, I'm not trying to make excuses for GD, just trying to impart the little information I can see in the hope it can alleviate some frustration.

Gone_Away · February 14, 2013, 09:02:18 PM

TL,

Can you pleas submit a ticket for us. I don't seem to have the ability to do so.

NF

TeaLeaf · February 17, 2013, 02:23:20 PM

Crap, I have just seen this post, not sure why I missed it, sorry. I thought you had ticket creation permissions, I will check.

TeaLeaf · February 17, 2013, 02:31:21 PM

I've ticked everything available to me NF, you should be able to do everything. Can you check again and let me know? Although tbh atm, in their new control panel it does not even have a 'submit ticket' option :blink:

TeaLeaf · February 17, 2013, 02:39:33 PM

While I was in there I noticed that none of the patches had been applied by GD, so I patched it from R0 to R37.

Gone_Away · February 17, 2013, 08:43:06 PM

At least it lives again..

It's poor that you had to re-patch it. They should have done that for all of us...

Gone_Away · February 17, 2013, 08:47:12 PM

From Carnage - Head Admin:

Hi Guys

whilst this isnt an official statement, id just like to give people a brief run through on what occured last week,

we discovered a raid deteriation on our primary game panel, and did our best to repair this, the SSD raid continued to shows signs of degradation and needing breaking and repairing

friday the 9th of february at 10:30pm we discovered a breach in the game panel master security
saturday the 10th of february at 2:00am we discovered our implimentation of tighter security controls had been breached by a back door
the game panel was taken offline with immediate effect

saturday the 10th of february, our game panel was completely rebuilt by andy re-imaged, and resecured
saturday the 10th of february, the game panel security was shown to be maliceously insecure be it by sql injection or IIS vulnerabilities

steve smiths drove down to cheltenham, security analysts were called in, game panel devs were called in, the way forward was discussed

with immediate effect it came clear that the planned roll out of our new panel wich we had started to action for a roll out in 3 months time needed immediate deployment to secure the IIS vulns (the new system dos not use IIS) and we needed a clean database even though we found no proof of injection.

a new scratch box with ssd raid on its own vlan was deployed with the new game panel, we started importing new games to it as fast as possible
as of sunday 11th feb everyones servers were running no panels and purely as system services with no access (no game server files were affected in the new panel install)

we started installing new remote software on every single peice of dedicated hardware, steve and andy were at the data center for 72 hours straight going through boxes
we were also securing boxes that had been compromised because of this breach in security

over this week we have completed the install of software and imported peoples servers to a completion of about 95%

game panel features and bugs are now being investigated as the last few accounts get added to the systems, people still not seeing their accounts are probably on our linux machines as this isnt an automated process and has to be done by hand and is quite time consuming, we apologise for that

we will be configuring the panel mod installers and fixing cmdline buildersand features over the coming days, we will still be providing this feature please be patient with us

if your server is still not showing up in your game panel, or you still cant access the new game panel system please log a support ticket and we will get things actioned for you immedietly

all the team at GD would like to apologise to all our clients for the frustrating way this happened, it wasnt perfect or how we had planned it, but was required and we hope you all enjoy the new panel software and features that it brings

TeaLeaf · February 18, 2013, 09:38:41 AM

That would account for the apparent lack of functionaility of the ACP. Let's hope some features become available soon to make it workable. Interesting how they do not feel password resets are needed or recommended given that they've had their trousers around their ankles for the last goodness-knows how long.

ArithonUK · February 18, 2013, 04:29:49 PM

Yes....

"Chosen the right excuse sir?"
"Yes thank you."
"How would you like those lies wrapped sir?"
"What paper do you have?"
"We've got 'hack attack' and 'hardware failure' this week sir."
"Good, I'll take both!"
"Thank you for shopping with 'Spin & Untruth' sir, do come again."

Server is down (12-13 Feb 2013)

ArithonUK

ArithonUK

ArithonUK