The world is currently online!

Welcome to Emps-World!

Register now to gain access to all of our forum features. Once registered and logged in, you will be able to create topics, post replies, send private messages, manage your profile, chat with other players in the shoutbox and much more. Once you sign in, this message will disappear.

Pages: 1
0 Members and 2 Guests are viewing this topic.

Offline Thomy

  • *
  • 3067
    • View Profile
Incident March 10th - Datacenter Burnt Down
« on: March 12, 2021, 11:44:29 am »
Hello everybody!

The datacenter emps was hosted on burnt down completely in the night of March 10th. There were no casualties other than a few bytes being lost in the fire. :)

I've already hired new server hardware and was able to get the game back up and running on the same day. There was a rollback of roughly 20 hours to the most recent backup I had. I spent the my past 2 days restoring the entire emps server. We are getting there but a few things are still missing.

Since our forum is more or less completely back online I'm continuing my report here:
--- 10th of march ---
7:40: contacted support and waiting for an answer:
8:01: data center is literally on fire ( looking into data recovery and setting up a new server now
8:20: hiring new hardware and trying to get everything back up and running again. no ETA on this, hopefully today
9:30: new server hired in germany :) now setting it up (no ETA when it will be finished, I hope today)
11:25: started importing backup from yesterday night. there will be a rollback until 9th of march 5am
12:30: new server partially up and running. might be ready in the evening today
13:25: few problems with database imports. some of you might already reach on the new server. nothing there works yet but the page should display
16:27: database import finished. testing homepage and bringing game server back online
16:47: game server successfully running and login worked with player data from yesterday. doing more investigating before it goes live
16:57: looking good! you will be able to reconnect whenever your computer decides to update DNS (takes up to 24h)
17:08: game server back online. working on web services now
18:00: I'll continue tomorrow. wiki, forum and recovery options still need to be done
--- 11th of march ---
6:45: fixed an issue with model loading (donator items, demon weapons, etc.)
7:45: gotta go to work today, will be back in the afternon (skipped office yesterday already)
12:30: continuing work - looking into missing models
16:00: few base homepage components have been fixed (hiscores, ge, logs, mod tools, etc.)
19:00: missing assets (models, icons, etc.) have been fixed. developed a new system for easier deployment and backups of assets
--- 12th of march ---
9:00: started to bring forum back
10:00: forum looking good. trying to recover some images from news posts
12:30: front page restored including images from patch notes
12:50: all forum avatars and some news post images have been lost :( these image files weren't in the backup
14:38: working on updating wiki software. looks like a lot of image data has been lost
15:33: wiki is finally back up: a lot of images are missing though.

If you are missing items from donations please make a helpdesk ticket:;sa=newticket
I will have a look at it as soon as we have everything fully back up and running. Progress made from the rollback sadly cannot be restored. We don't have any data from the day the fire happened.

I'm sorry for things that have been lost in the fire but this was quite an unexpected incident. I'm also working on further measures to be better prepared should such an emergency happen another time.

All the best,
« Last Edit: March 12, 2021, 02:33:55 pm by Thomy »

Offline Papa Troq

  • *
  • 284
  • Playing da game since 2007.
    • View Profile
Re: Incident March 10th - Datacenter Burnt Down
« Reply #1 on: March 12, 2021, 05:49:59 pm »
Pages: 1