Field Notes [/ˈfild ˈnoʊts/] n - These are war stories from actually operating the equipment described elsewhere in these pages. They are deliberately small, the sort of thing a hands-on engineer can mend with a script, a recalibration, a change of procedure, or a clever bit of wiring, without a budget request or a redesign. I collect them because the best fixes are almost never the expensive ones; they come of understanding the system one layer deeper than the alarm does.

For the problems that couldn’t be fixed at this level, see Problems We Could Only Work Around. For the ideas that looked good but failed in practice, see Things That Did Not End Up Working Out.

The Record

• The Watts Hidden in Volt-Amps [/ˈwatts ˈhidden ˈin ˈvolt-amps/] n - a metering bug that was wasting real capacity, mended without touching the hardware.
• The Fault That Would Not Be Found [/ˈkhasɪŋ ˈa ˈfantom ˈground ˈfault/] n - an intermittent alarm that proved no fault at all.
• The Generator Was Not Overheating [/ˈthe ˈgenerator ˈwas ˈnot ˈoverheatɪŋ/] n - a monthly test that kept aborting, traced to something deeply unglamorous.
• On the Camera That Drifted on the Hot Aisle [/ˈthermal ˈkamera ˈdrift ˈon ˈthe ˈhot ˈaisle/] n - a monitoring system that slowly began to lie, and a self-correcting fix.
• The Right Sensor in the Wrong Air [/ˈright ˈsensor ˈwrong ˈair/] n - a sensor that looked wrong was right; the airflow was not.
• Clean Tests, a Dirty Connector [/ˈklean ˈtests ˈdirti ˈkonnektor/] n - a clean optic that failed only under the real workload.
• The Breaker Never Moved [/ˈthe ˈbreaker ˈnever ˈmovɪd/] n - a phantom trip caused by a loose control wire, not a bad breaker.

These connect back to the systems they came from: Power distribution unit, Grounding and bonding, The Standby Engine, Direct-to-chip liquid cooling, AI factory power and cooling, and Optical interconnects in AI clusters. The tooling behind a couple of them lives in my Crafts projects.

Further Entries

• On the Loop in the Switching Fabric [/ˈdiagnosɪŋ ˈa ˈnetwork ˈswitkh ˈloop/] n
• On a Transceiver Gone Bad [/ˈfixɪŋ ˈa ˈbad ˈfiber ˈoptik ˈtransseiver/] n
• Keyboard shortcuts [/ˈkeyboard ˈshortkuts/] n
• A Cooling Failure Near the Top-of-Rack Switches [/ˈmanagɪŋ ˈa ˈkoolɪŋ ˈfailure ˈnear ˈtop-of-rakk ˈswitkhes/] n
• On the Battery Behind the RAID Controller [/ˈrepairɪŋ ˈa ˈraid ˈkontroller ˈbatteri ˈfailure/] n
• On the Changing of a Motherboard [/ˈreplakɪŋ ˈa ˈfailɪd ˈserver ˈmotherboard/] n
• On Changing a Supply While the Load Ran [/ˈreplakɪŋ ˈa ˈfaulti ˈpower ˈsuppli ˈunit ˈunder ˈload/] n
• On a Server That Would Not Wake [/ˈresolvɪŋ ˈa ˈserver ˈboot ˈfailure/] n
• On the Packet That Never Arrived [/ˈtrakɪŋ ˈa ˈdroppɪd ˈnetwork ˈpakket ˈissue/] n
• On the Router That Slowly Forgot [/ˈtroubleshootɪŋ ˈa ˈkore ˈrouter ˈmemori ˈleak/] n
• On Firmware, A Thousand Servers at a Time [/ˈupdatɪŋ ˈserver ˈfirmware ˈat ˈskale/] n
• A Room Built for Air, Asked to Run Water [/ˈa ˈroom ˈbuilt ˈfor ˈair ˈaskɪd ˈto ˈrun ˈwater/] n