by Or how I conquered out-of-band management with a box of serial and a power button.
In the beginning
When dinosaurs ruled the earth and small, mammalian critters scurried through the data center, the classic method of doing out-of-band management was via a crash cart and the laying of holy hands upon the ailing system. You’d wheel that old, cranky monitor around while the wheels going thumpa-thumpa-thumpa across the raised floor. You’d crawl back behind your computer shelves with a 25 foot vga and ps/2 cable, plug in, tune out, and administer. Maybe you were lucky and had in-rack KVM access. Whatever it was, it still required a physical presence in the room, often at 2AM when you were 30 miles away at a rocking party.
This is what we did. It works all right when you have a few tens of systems to worry about and don’t often get called in the middle of the night to go fix something. Then, one day, you realize you’re surrounded by a few hundred systems and you’re sleeping behind the CRAC units because you’ve been onsite fixing problems for the last 72 hours. This is the sign of your impending evolution.
Evolution One
When we switched from shelving units to racks we re-evaluated the way we did our administration. We got tired of standing in 50 degree data centers, perched over perf tiles for hours on end. We needed something different. By this point, everything we were buying could do BIOS POST redirection to the system’s onboard serial port. Man, was that sexy. We could sit up at our desks, connect away, and do everything remotely over serial. We were living the good life. And then, we figured out we still kept going into the data center to push the power button. This wasn’t a big deal, really. In our environment we didn’t need to do this often. Most systems didn’t power down so we could do fancy things like reset the system using SysReq on Linux or sending a break to the Suns. Inevitably, we figured out that systems really enjoyed misbehaving during reboots, so we still did the slow trudge down to the cold, windy room. There we went, and here we froze. Just to push the button.
Evolution Two
After our success with remote serial console access, we embarked on a way to manage the power remotely. It was now a moral imperative to get out of the data center and stick to our desks (at least there we could hear). What we found were network attached remote power distribution units: glorified power strips that we could do a virtual power fat finger on. We were in hog heaven. We deployed hundreds of these. We danced in the cube aisles. Ok, not really. But we now came to have our first lights-out data center. We no longer needed to go in there to do simple button pushes or plug a console in to find out why a system didn’t install or reboot correctly.
Revolution
Today, we rejoice. We no longer have to tie together glorified power strips and little blue boxes of serial to make our lights-out data center. Manufacturers are beginning to understand the value in providing these handy management features in the system itself. With HP providing integrated Lights-Out (iLO), Sun’s Advanced Lights Out Manager (ALOM), the Intelligent Platform Management Interface standard (IPMI), and so on, we can deep-six the first generation of hardware, much like we did with our crash carts. One of the unseen benefits to this integrated management board is monitoring; many of them provide hooks into the host hardware for temperature monitoring, fan speed, power supply and dimm failures, and other bits you probably don’t care (much) about.
There’s a few downsides to all this integration. You rememeber that evolution one and two setup? It was nice and platform agnostic. You power off a specific power receptacle. You connect to a specific serial port. You do it the same way for all your hardware. iLO and ALOM are monothiestic. They’ll only work with HP and Sun, respectively. IPMI tries to be every vendor’s best friend. It’s an interesting standard, but not every vendor has adopted it. Maybe in a few years, the great awakening will happen, companies will gather around the camp fire, sing “Kumbaya”, and put forth the effort needed to have a management platform that works everywhere.
Additionally, these integrated management boards often demand their own network interface (and don’t forget the additional IP space). Some, like iLO, can be configured to multiplex off the host’s own NIC. Think of it as a miniature two port network switch built into the host, with both the host and management board sharing the uplink. While this reduces your network costs, you have to worry about getting to your system should the uplink itself go down. It’s up to you to decide what works best for your environment. In mine, we do both depending on the importance of the server and how willing we are to make the trek into work to figure out why something went down.
My world
For now, the history lesson is over. Let’s get into some practicalities. These aren’t fast and heavy rules; they’re more like guidelines. So, if you must “keep to the code”, feel free, but don’t feel pigeon-holed by it. Most of this applies to evolution one and two of our environment, which is still in heavy use. We’re still figuring out the revolution. Also, because I’m lazy, I’m going to define a few terms for later. The box of serial is a console access server or CAS and that glorified power strip is a power distribution unit or PDU.
Be anal. Do your inventory.
Despite what people may think, it’s perfectly fine to be anal retentive when it comes to keeping track of your CAS and PDU mappings. In fact, I think it’s an outright requirement. What’s a mapping? Well, with every system you have, you need to know a few things: where’s this wire coming from and where’s it going to. This is a mapping. You need to know this for both power and serial on each system you have. This is of singular importance. If either of these is wrong or unknown, you run the risk of not getting to the right console or, even worse, power cycling the wrong system. The last thing you want to do is explain to the boss why you rebooted your main webserver when you were attempting to reboot your Halo server.
Also, don’t forget to routinely audit your mappings. During every day work, it doesn’t take much to get a cable swapped here and there. How often you do this is up to you. If you have little churn in your racks and don’t often need to touch them, maybe you only do it once a year. If you have lots of systems and you’re constantly in and out of the racks doing hardware repairs and physical work, maybe you need to do it once a month. It’s entirely up to you. Just make sure you do it.
Now that you have all your mappings scribbled down for posterity, do your self a favor and shove it all into your inventory database. Use this as your source of data for building config files, keeping track of ports, and so on. You’ll thank me later. If you don’t have one master database to store this information, you’ll just end up with duplicate data somewhere or even worse: different data for the same host. And if this happens, you may as well re-audit everything.
Physical Wiring
Label all of your cables on both ends. In my racks, every serial cable has the name of the system and the name of the CAS, as well as the CAS port. The same goes with the power cables: PDU, PDU port, and the system name. Standardize on the same naming scheme for all your racks. Make them as cookie-cutter as possible. This makes it easier to find problems with your mappings. Bundle your cables up and get them out of the way. Make sure it’s neat. Keeping them out of the way will keep them from blocking airflow out of the rack. Oh, and bundle them up with velcro. Please don’t use zip ties. I can guarantee that somewhere, someday, you’ll have to get at one of those cables in the middle. Having to cut off those ties is a pain. The velcro is easy. It just rips right off when you need it to. It smushes back together with little effort.
Console Access
Do yourself a favor and get centralized. Get conserver. Conserver makes it easy to provide one front end for accessing all your console ports, whether they hooked up via a CAS or you’re accessing the virtual serial port used in iLO or ALOM. I’ve found that one conserver can handle a couple thousand systems with a bit of tuning. Once you get beyond that, you should consider splitting them up into several conserver masters. Here again, conserver makes it easy to manage all your consoles from one centralized point. With multiple masters, you can make the configs aware of each other. The conserver client will transparently redirect to the right master. This is extremely useful when your equipment is distributed across sites, time zones, and dimensions. Ok, maybe not the last one. But, having the same front end everywhere makes it easier for your coworkers at other sites to help debug problems and cover for you when you’re not around.
Remember that inventory database you put all the port mappings into? Use that to generate your conserver config files.
Power control
If you’re looking at networked PDUs, make sure you get some that support the Simple Network Management Protocol (SNMP) for toggling the power on ports. It really is simpler than trying to do something funky like telnetting or sshing into your PDU, parsing the interface with expect, passing the power command over, waiting for a response, and then hoping nothing got in the way during the entire process. SNMP really is simple. Just set the appropriate bit and forget it. You’ll have to configure your systems to do the right thing when they lose power, of course. This is pretty straight forward. We set our systems to always come on when power is applied and then have the PDUs configured so that they only apply power when specifically ordered. Dangerous? Maybe. You could theoritically have systems bouncing up and down when power flickers. Just make sure your PDUs are configured correctly and you’ll be fine.
We have yet to find an open-source, platform agnostic method for handling power, sort of a conserver of power. It’s fairly simple to write something that looks up a host’s port on the PDU via your inventory database and then pass the appropriate SNMP set to the PDU. The same goes for power control via iLO or ALOM. In theory, they should be able to handle this in a similar manner.
Room for improvement
Our environment and lights-out methods have evolved over many, many installs of racks and systems. Each time we complete a rack we note how things could have been better and try to fit it in for the next run. Our development is far from complete. We continually look for new ways to integrate new out-of-band gizmos and tweak our methods for managing it all. It’s often a struggle. We’re still figuring out how to modify our way of thinking to get a better handle on things like iLO, ALOM, and IPMI. Some day, we’ll have our cake and eat it to. When that happens, I’ll miss suiting up in the cold-weather gear and heading into the data center to work. Maybe not.
Tools and Information
Here’s a list of things mentioned in the article and used in our environment to help make our lives easier: