- Reaction score
Today at 1 am GMT @Bolli called me and woke me up to notify me of degraded server performance. We suspected a DDoS attack was taking place and attacking our website/SQL Server.
However, as it turns out this was not the result of some DDoS attack but rather the results of the server overheating and the CPU throttling its frequency to compensate for high temperatures that were caused by the server and the entire datacenter literally burning.
As it turns out, the entire datacenter our server was hosted in went up in flames yesterday. The datacenter we were using was made up of 4 buildings, SBG1-SBG4, with our server hosted in SBG2. The fire started in SBG2 and according to OVH the building is completely destroyed (Some more pictures here).
We were informed of a fire in the building at 3:00 AM GMT today that apparently started at ~1:00 AM GMT, however it was not clear how much damage was caused by the fire. At 4:00 AM GMT it was announced that SBG2 was destroyed entirely and that customers should activate their "Disaster Recovery Plans".
At this point we knew that it all the data was likely lost, so we ordered a new server from a different hosting provider at 5:00 AM GMT which took until 7:30 AM GMT to be setup and available.
From this point we started setting up all the services and restored the backups we had. This took until 2:00 PM GMT at which point the server was almost in a fully functional state and at the very least playable.
The backups of our database are created daily at 2:30 AM GMT but the server was destroyed before the backup for that day could be completed, which means we lost roughly 23 hours of progress on the server. We have noted down a list of all the users who bought premium during that time and we will make sure that you do get it added back (all but one have already received it).
Some data, such as files uploaded to the forums (e.g. avatars) and files.perpheads.com files are only backed up on a weekly basis and the last backup was on the 7th of march, so we lost ~3 days of that, but I doubt that will cause significant issue.
We are evaluating our new hosting provider to see how well the server does and depending on the outcomes of that we might switch to another provider in the future, so be sure to always use the domain name (perpheads.com) to connect to TeamSpeak, Server, etc. rather than using the server's IP.