Moving all servers of tilllate.com in one night?
Maybe you’ve read our short announcement on the tilllate website last week: We were offline for some planned maintenance.
In detail, we moved all servers to a new cage in the datacenter. We like to give you some insight, of what happened after switching off the tilllate webservers. But let’s start at the beginning: Why did we have to move our servers? As we were growing, space in the existing racks was getting smaller and smaller. So we had to look for new space and found a solution in the datacenter we already had one rack: Now we have three racks in the same room and enough space to consolidate all servers in this room.
What exactly happened?
We met, shutdown the whole website, moved the servers that were in another datacenter to the new datacenter, re-installed everything and then brought tilllate.com back online. We don’t want to bore you with too many details and let the images speak instead:
Let’s start with Silvan, our CTO that tries to shutdown all servers (ok, the servers were shut down already, he’s sitting in front of a screen that just tells him “no signal”):
The next two images show most of our servers. We had to store them on the floor to order them before we could re-mount them. Thank god that the datacenter crew did not close the side of the new cage which would have removed all space around the racks.
During the re-instllation process, 8 guys of the IT team were on site to help out where possible. Thanks Ciprian, Thilo, Leo, Jia and Silvan! (They all volunteered for this night shift!)
Leo is mounting some equipment in one of the racks:
Steven, Mario and Stefan are the three System Engineers from the infrastructure team:
After all devices were installed into the racks, some of the services didn’t start as they should have. And so Stefan and Steven started a debugging session while sitting on the floor (Hint to the datacenter crew: If you read this, please organize some chairs and a table – sitting on the floor is unconfortable).
After going offline at 23:00 as planned we were 30min before schedule when we arrived in the datacenter. Re-installing the servers took much more time than expected due to many different types of rails, screws and other small stuff that didn’t match.
At the end, we’d like to provide you with some statistical information about this night shift:
- Team members involved: 8
- Total work hours: 66
- Accidents: 0
- Times someone cursed out loud: uncounted, many!
- Servers that did not survive the transport: 1 (unverified)
- Meters of cables used: more than 300
- RedBull: 2 (drinking not allowed in the datacenter)
- Servers unmounted and re-mounted: 51
I hope we could give you a look into our work.