Clean Tests, a Dirty Connector [/ˈklin ˈtɛsts ə ˈdərti kəˈnɛktər/] n - The symptom. A 400G optical link between two GPU switches was flapping. The vendor’s diagnostics showed light levels, bit error rate, and eye diagrams all within spec. The fiber passed every trial we set before it, yet the link kept falling under one particular training workload.

The dead end. [/ðə ˈdɛd ˈɛnd/] n - The team wished to replace the transceivers, then the cable, then the whole path: costly and slow, and the new parts would likely have failed in the same manner.

The actual cause. [/ðə ˈæktʃuəl ˈkɔːz/] n - One connector face bore a film nigh invisible, not enough to slay the link, but enough to scatter light when the pattern ran dense and the modulation held one polarization angle for long runs. The standard light-level test, averaging over time, was blind to it; the training workload was not.

The fix: limited and clever. [/ðə ˈfɪks/] n - We cleaned both ends of the connector pair with the proper pen and inspected them beneath a fiber microscope. The flapping ceased. The whole downtime was the walk to the rack, not the re-cabling of the row. The incident now lives in the onboarding runbook for Optical interconnects in AI clusters.

The lesson. [/ðə ˈlɛsən/] n - A link that passes static tests may still fail beneath true traffic. When the pattern matters more than the average, examine the physical layer with the workload running, not the test set alone.

Part of Field Notes.