PH Burnt Down

Messages
499
Reaction score
3,757
Points
585
Location
McDonalds
style-logo-png.10315

TL;DR: Datacenter burnt down. Server was down for ~12 hours and restored from backups. Up to 24 hours of player progress was lost.

Today at 1 am GMT @Bolli called me and woke me up to notify me of degraded server performance. We suspected a DDoS attack was taking place and attacking our website/SQL Server.

1615395847576.png

However, as it turns out this was not the result of some DDoS attack but rather the results of the server overheating and the CPU throttling its frequency to compensate for high temperatures that were caused by the server and the entire datacenter literally burning.
1615396117511.png

As it turns out, the entire datacenter our server was hosted in went up in flames yesterday. The datacenter we were using was made up of 4 buildings, SBG1-SBG4, with our server hosted in SBG2. The fire started in SBG2 and according to OVH the building is completely destroyed (Some more pictures here).
1615396534407.png

We were informed of a fire in the building at 3:00 AM GMT today that apparently started at ~1:00 AM GMT, however it was not clear how much damage was caused by the fire. At 4:00 AM GMT it was announced that SBG2 was destroyed entirely and that customers should activate their "Disaster Recovery Plans".
At this point we knew that it all the data was likely lost, so we ordered a new server from a different hosting provider at 5:00 AM GMT which took until 7:30 AM GMT to be setup and available.

From this point we started setting up all the services and restored the backups we had. This took until 2:00 PM GMT at which point the server was almost in a fully functional state and at the very least playable.
The backups of our database are created daily at 2:30 AM GMT but the server was destroyed before the backup for that day could be completed, which means we lost roughly 23 hours of progress on the server. We have noted down a list of all the users who bought premium during that time and we will make sure that you do get it added back (all but one have already received it).

Some data, such as files uploaded to the forums (e.g. avatars) and files.perpheads.com files are only backed up on a weekly basis and the last backup was on the 7th of march, so we lost ~3 days of that, but I doubt that will cause significant issue.

We are evaluating our new hosting provider to see how well the server does and depending on the outcomes of that we might switch to another provider in the future, so be sure to always use the domain name (perpheads.com) to connect to TeamSpeak, Server, etc. rather than using the server's IP.
 
Messages
2,881
Reaction score
8,117
Points
1,388
Location
Leeds, England.
very nice, a reason for us to delay V5 again
 
Messages
927
Reaction score
941
Points
580
Location
Israel
jesus christ, how often does this happen?? like damn this is so random lmao, thanks for the speedy recovery
 
Messages
5,987
Reaction score
16,096
Points
925
Location
North Rhine-Westphalia, Germany
Rust was also affected. But as it turns out, players (or Twitter being Twitter) were more worried about 10 days of in-game progression that gets wiped every 30 days instead of human lives that could've been lost in this accident.

Hopefully teaches a lot of people a good lesson why you should keep back-ups of important stuff. Thanks for the work.
 
View previous replies…
Messages
1,210
Reaction score
4,165
Points
630
Location
Scotland
Literal basement dwellers, I couldn't believe the entitlement of the people REEing on twitter demanding Facepunch compensate them for their loss of Rust progress.
 
Messages
183
Reaction score
127
Points
480
Location
Germany
Bro, I just have no idea about something like that, but I think it's good that you notice things like that and fix them immediately so that we can play without a problem.
 
Messages
505
Reaction score
738
Points
605
Location
United Kingdom
Unfortunately, server performance is not the best on the new server however it’s playable and it’s great how quickly things got up and running again
 
Messages
183
Reaction score
127
Points
480
Location
Germany
Yes, but it's still nice to see how the server continues to develop and that the owner / staff team do everything to ensure that everything is okay
 

Ashur

Communication Banned
Messages
643
Reaction score
825
Points
580
Location
in a basement
This fire could've been much worse, they seem to be very fortunate only 1 of their buildings has burnt down, was a pretty nasty fire.
 
Messages
995
Reaction score
4,976
Points
580
Location
Parts Unknown
Imagine hosting in an actual shipping container with no fire protection, there is a reason why i avoid OVH lol.
Can't wait for the fire sales to start ;)
 
View previous replies…
Messages
759
Reaction score
429
Points
510
OVH gonna be tryna dump off servers cheap real soon i bet once they get up and running probably gonna save the server loads of cash if they go with them again
 
Messages
183
Reaction score
127
Points
480
Location
Germany
I think it's just a shame that it burned down, but I think it's much more important that no one was injured or, dear god, no one died. I hope so. At the moment I know what you think, but that's my opinion.
 
Messages
895
Reaction score
727
Points
590
Location
Under Lewis088's bed
you really are trying to show people you're reformed aren't you lmao
 
View previous replies…
Messages
183
Reaction score
127
Points
480
Location
Germany
@Clarky Thank you Clarky and flugs,I told you I have changed I will go through with this and show the admins / staff members that I have really changed. 2 days full active in the forum and also help in the Shourtbox chat.
 
Messages
24
Reaction score
16
Points
35
Location
Israel
My friend who plays Rust said the EU servers of it were also burnt down, meaning they can lose their skins and other stuff. Maybe your server and Rust's EU based on the same datacenter.
 
Messages
2,459
Reaction score
3,884
Points
655
Location
The Hague
Only the rust dedicated servers were lost. Those are wiped monthly. Skins and player data is stored on Steam servers. Yes, PH was on the same datacenter.
 
Messages
1,023
Reaction score
1,864
Points
865
Location
United Kingdom
View previous replies…
Top