PH Burnt Down

[Fredy]

​

TL;DR: Datacenter burnt down. Server was down for ~12 hours and restored from backups. Up to 24 hours of player progress was lost.

​

Today at 1 am GMT @Bolli called me and woke me up to notify me of degraded server performance. We suspected a DDoS attack was taking place and attacking our website/SQL Server.



However, as it turns out this was not the result of some DDoS attack but rather the results of the server overheating and the CPU throttling its frequency to compensate for high temperatures that were caused by the server and the entire datacenter literally burning.


As it turns out, the entire datacenter our server was hosted in went up in flames yesterday. The datacenter we were using was made up of 4 buildings, SBG1-SBG4, with our server hosted in SBG2. The fire started in SBG2 and according to OVH the building is completely destroyed (Some more pictures here).


We were informed of a fire in the building at 3:00 AM GMT today that apparently started at ~1:00 AM GMT, however it was not clear how much damage was caused by the fire. At 4:00 AM GMT it was announced that SBG2 was destroyed entirely and that customers should activate their "Disaster Recovery Plans".
At this point we knew that it all the data was likely lost, so we ordered a new server from a different hosting provider at 5:00 AM GMT which took until 7:30 AM GMT to be setup and available.

From this point we started setting up all the services and restored the backups we had. This took until 2:00 PM GMT at which point the server was almost in a fully functional state and at the very least playable.
The backups of our database are created daily at 2:30 AM GMT but the server was destroyed before the backup for that day could be completed, which means we lost roughly 23 hours of progress on the server. We have noted down a list of all the users who bought premium during that time and we will make sure that you do get it added back (all but one have already received it).

Some data, such as files uploaded to the forums (e.g. avatars) and files.perpheads.com files are only backed up on a weekly basis and the last backup was on the 7th of march, so we lost ~3 days of that, but I doubt that will cause significant issue.

We are evaluating our new hosting provider to see how well the server does and depending on the outcomes of that we might switch to another provider in the future, so be sure to always use the domain name (perpheads.com) to connect to TeamSpeak, Server, etc. rather than using the server's IP.

 

[Ayjay]

very nice, a reason for us to delay V5 again

 

[Double J]

How much for you to not delay it?

 

[Ayjay]

however much that adhd medication costs on the black market

 

[Auris]

@Ayjay i will litterly take out a loan to pay you £1000 to release v5

 

[Smeris]

Aaron I recommend giving these lads a call.

UKAT - UK Addiction Treatment Centres

UKAT provides safe rehabilitation centres all across UK. Discover all rehab services & addiction centres, open 24/7 with medically trained experienced staff.

www.ukat.co.uk

 

[Smeris]

Thanks for the work fredster and good that the backups weren't on a different partition only!

 

[MegaFro]

jesus christ, how often does this happen?? like damn this is so random lmao, thanks for the speedy recovery

 

[Creepis]

Rust was also affected. But as it turns out, players (or Twitter being Twitter) were more worried about 10 days of in-game progression that gets wiped every 30 days instead of human lives that could've been lost in this accident.

Hopefully teaches a lot of people a good lesson why you should keep back-ups of important stuff. Thanks for the work.

 

[ati ツ]

Bro, I just have no idea about something like that, but I think it's good that you notice things like that and fix them immediately so that we can play without a problem.

 

[Smeris]

The servers litteraly get wiped every month why do they care so much

 

[Mxdium]

With 'Make the server hot again' we meant like popularity... Not burning the fucking servers down!

 

[Ashur]

This fire could've been much worse, they seem to be very fortunate only 1 of their buildings has burnt down, was a pretty nasty fire.

 

[Ashur]

I think it's more tragic that people only care about their in-game progression lol, that company could've been burnt down completely, potentially injuring employees that were there, and also thousands of people losing their data,

 

[Bert]

Just remember NA is always an option. You don't need to worry about servers overheating if you get a server in middle of no where Canada!

 

[Clarky]

Unfortunately, server performance is not the best on the new server however it’s playable and it’s great how quickly things got up and running again

 

[ati ツ]

Yes, but it's still nice to see how the server continues to develop and that the owner / staff team do everything to ensure that everything is okay

 

[joey]

Bet you @Wiki committed arson to evade his two week ban

nevermind, it was probably @amir

 

[Efan]

Is it too much to do both?

 

[blobvis 2.0]

Or it was you, maybe you decided to take revenge on perp for getting promoted to honorary again?