28. January 2008, 16:55, by Mario Rimann

Night shift: Server move

Moving all servers of tilllate.com in one night?

Maybe you’ve read our short announcement on the tilllate website last week: We were offline for some planned maintenance.

Lots of servers…

In detail, we moved all servers to a new cage in the datacenter. We like to give you some insight, of what happened after switching off the tilllate webservers. But let’s start at the beginning: Why did we have to move our servers? As we were growing, space in the existing racks was getting smaller and smaller. So we had to look for new space and found a solution in the datacenter we already had one rack: Now we have three racks in the same room and enough space to consolidate all servers in this room.

What exactly happened?

We met, shutdown the whole website, moved the servers that were in another datacenter to the new datacenter, re-installed everything and then brought tilllate.com back online. We don’t want to bore you with too many details and let the images speak instead:

Let’s start with Silvan, our CTO that tries to shutdown all servers (ok, the servers were shut down already, he’s sitting in front of a screen that just tells him “no signal”):

The CTO hacker

The next two images show most of our servers. We had to store them on the floor to order them before we could re-mount them. Thank god that the datacenter crew did not close the side of the new cage which would have removed all space around the racks.

Chaos

Lots of servers…

During the re-instllation process, 8 guys of the IT team were on site to help out where possible. Thanks Ciprian, Thilo, Leo, Jia and Silvan! (They all volunteered for this night shift!)

Leo is mounting some equipment in one of the racks:

Leo in the box

Steven, Mario and Stefan are the three System Engineers from the infrastructure team:

System Engineers

After all devices were installed into the racks, some of the services didn’t start as they should have. And so Stefan and Steven started a debugging session while sitting on the floor (Hint to the datacenter crew: If you read this, please organize some chairs and a table – sitting on the floor is unconfortable).

Debugging

After going offline at 23:00 as planned we were 30min before schedule when we arrived in the datacenter. Re-installing the servers took much more time than expected due to many different types of rails, screws and other small stuff that didn’t match.

Statistical information

At the end, we’d like to provide you with some statistical information about this night shift:

  • Team members involved: 8
  • Total work hours: 66
  • Accidents: 0
  • Times someone cursed out loud: uncounted, many!
  • Servers that did not survive the transport: 1 (unverified)
  • Meters of cables used: more than 300
  • RedBull: 2 (drinking not allowed in the datacenter)
  • Servers unmounted and re-mounted: 51

I hope we could give you a look into our work.

Filed under: IT Infrastructure

5 Comments

  1. Can you tell us more about your Hardware for the Database?

    Comment by Jeanot Bruchmann — 31. January 2008 @ 21:09

  2. Thanks for your interest in our systems. We’re planning to re-organize the whole MySQL Cluster – and I’m sure the project leader will write a blog post on this topic :-)

    Comment by Mario Rimann — 1. February 2008 @ 08:19

  3. You do seem to have an aweful lot of servers for the size of the site you run, is it cheaper to add more hardware than optimize code/apache/etc?

    Comment by Ben — 2. February 2008 @ 21:22

  4. Hi Ben. Our site is bigger then it looks. We try to improve our code and caching strategie on a daily basis and I admit we still have room for improvment. But for short term grow it’s cheaper (and faster) to buy more hardware.

    Comment by Leo Büttiker — 4. February 2008 @ 08:57

  5. Uniflair blowers, white wall with no outside light and minkels racks, this is Interxion! :)

    Comment by Gregory — 21. February 2010 @ 17:58

RSS feed for comments on this post.

Sorry, the comment form is closed at this time.

© 2017 tilllate Schweiz AG - Powered by WordPress