Thinking outside the rack (literally)

(Reference, yesterday’s blog about moving servers from one row of racks to another):

OK, so I had several servers to move to a new rack in another row. As of this morning, they were ALL done, except one. The person who we wrote the application for on that server (the “owner”), had not responded to my emails or phone calls asking for permission to shut down the server. He probably had meetings, whatever. Yet, I was being pressured to finish my move.

So, I decided to do what any good sysadmin would do in this predicament: Move it anyway. Oh yeah, and with ZERO downtime.

First, this works only if the server has multiple power supples (it did). Unplugged power, and re-hooked new power fed from the other rack (using standard rack power cords, a few strung together). Repeated for second connection.

As far as network, I simply used a really long patch cable (25′), and did the patch panel switch (BOTH ends) within a few seconds. (This one piece required the help of my assistant, Pedro). As long as the ethernet connection was switched, QUICKLY, there were no issues (note: SAME actual switch port, that did not move).

Once the new connections were in place, it was a simple exercise to remove the server, lay it on a cart- move the rails, then put the server in its new home. Finally, unhooked and re-hooked the power and network connections again using proper length cables, correctly routed in the channels in the rack.

I have to note, while the above seems extreme (and is admittedly just a little awesome that it worked) – it by far pales in comparison to a similar (but definately trickier) situation I was told about (move server, ACROSS TOWN). In that setup, the server running with the jobs executing needed to continue, uninterrupted. The loss of a network connection for a while wasn’t important. The solution involved the obvious two-power-supply setup, but then also a portable UPS and a taxicab. Ya gotta love it.

A note about this process: It’s still quite risky, mostly from the disk perspective. I took care to remove the server from the rails verrrrrrryyyyy sloowwwwwwly. You really don’t want to disturb 7200 revolutions per minute. I was in a controlled environment, and the destination rack was 5 feet away. I can only imagine what that cab ride must have been like during each bump in the road. It’s amazing THAT server survived the trip!

A lot of people have commented that being a sysadmin isn’t fun. My counter to that is, it’s what you make it. And today thus far has most certainly been a fun day.