PH Burnt Down

Status
Not open for further replies.
Messages
697
Reaction score
5,334
Points
735
style-logo-png.10315

TL;DR: Datacenter burnt down. Server was down for ~12 hours and restored from backups. Up to 24 hours of player progress was lost.

Today at 1 am GMT @Bolli called me and woke me up to notify me of degraded server performance. We suspected a DDoS attack was taking place and attacking our website/SQL Server.

1615395847576.png

However, as it turns out this was not the result of some DDoS attack but rather the results of the server overheating and the CPU throttling its frequency to compensate for high temperatures that were caused by the server and the entire datacenter literally burning.
1615396117511.png

As it turns out, the entire datacenter our server was hosted in went up in flames yesterday. The datacenter we were using was made up of 4 buildings, SBG1-SBG4, with our server hosted in SBG2. The fire started in SBG2 and according to OVH the building is completely destroyed (Some more pictures here).
1615396534407.png

We were informed of a fire in the building at 3:00 AM GMT today that apparently started at ~1:00 AM GMT, however it was not clear how much damage was caused by the fire. At 4:00 AM GMT it was announced that SBG2 was destroyed entirely and that customers should activate their "Disaster Recovery Plans".
At this point we knew that it all the data was likely lost, so we ordered a new server from a different hosting provider at 5:00 AM GMT which took until 7:30 AM GMT to be setup and available.

From this point we started setting up all the services and restored the backups we had. This took until 2:00 PM GMT at which point the server was almost in a fully functional state and at the very least playable.
The backups of our database are created daily at 2:30 AM GMT but the server was destroyed before the backup for that day could be completed, which means we lost roughly 23 hours of progress on the server. We have noted down a list of all the users who bought premium during that time and we will make sure that you do get it added back (all but one have already received it).

Some data, such as files uploaded to the forums (e.g. avatars) and files.perpheads.com files are only backed up on a weekly basis and the last backup was on the 7th of march, so we lost ~3 days of that, but I doubt that will cause significant issue.

We are evaluating our new hosting provider to see how well the server does and depending on the outcomes of that we might switch to another provider in the future, so be sure to always use the domain name (perpheads.com) to connect to TeamSpeak, Server, etc. rather than using the server's IP.
 
Messages
1,170
Reaction score
1,221
Points
660
Location
Israel
jesus christ, how often does this happen?? like damn this is so random lmao, thanks for the speedy recovery
 
Messages
239
Reaction score
210
Points
430
Location
Germany
Bro, I just have no idea about something like that, but I think it's good that you notice things like that and fix them immediately so that we can play without a problem.
 
Messages
2,614
Reaction score
4,226
Points
845
The servers litteraly get wiped every month why do they care so much
 
Messages
683
Reaction score
865
Points
610
Location
in a basement
This fire could've been much worse, they seem to be very fortunate only 1 of their buildings has burnt down, was a pretty nasty fire.
 
Messages
683
Reaction score
865
Points
610
Location
in a basement
I think it's more tragic that people only care about their in-game progression lol, that company could've been burnt down completely, potentially injuring employees that were there, and also thousands of people losing their data,
 
Messages
2,199
Reaction score
3,107
Points
985
Location
United Kingdom
Unfortunately, server performance is not the best on the new server however it’s playable and it’s great how quickly things got up and running again
 
Messages
239
Reaction score
210
Points
430
Location
Germany
Yes, but it's still nice to see how the server continues to develop and that the owner / staff team do everything to ensure that everything is okay
 
Status
Not open for further replies.
Top