A Cooling Failure Near the Top-of-Rack SwitchesA Cooling Failure Near the Top-of-Rack Switches129

The facility environment sensors raised a high-temperature alarm in one of the networking aisles: 35 degrees Celsius at the top of rack number four, beyond the safe operating limit of the top-of-rack network switches. I had to bring the temperature down quickly, lest the switches shut themselves down.

I walked the aisle and looked. A blanking panel was missing from the middle of rack number four, and through that gap the hot aisle breathed its air into the cold. One of the computer room air conditioning (CRAC) units was also running at reduced capacity. I fitted a temporary blanking panel to close the gap at once, then set a portable cooling unit in the cold aisle and aimed it squarely at the afflicted switches.

Within ten minutes the temperature at the top of the rack had fallen to 24 degrees Celsius. The switches held; network traffic continued undisturbed. The facilities team repaired the CRAC unit later that day, and I audited every rack in the aisle to be certain each blanking panel sat where it ought.