On Running the Root-Cause Inquiry [/ɔn ˈrənɪŋ ðə ˈruːt-kause ˌɪnkˈwaɪˌri/] n - When an incident harms a customer, the fix is but half the work. The other half is the root cause analysis: the document that records what happened, why, and what shall prevent its recurrence. In my IOC operations role I own the RCA workflow; I do not author every RCA, but I see that each is opened, written well, reviewed, delivered on time, and retained.
Of the Lifecycle
- Trigger. [/ˈtrigger/] n - Any incident above a severity bar (or any SLA-impacting event) opens an RCA task automatically in the ITSM ticketing systems, linked to the parent incident so the timeline and alerts follow of themselves.
- Draft. [/ˈdraft/] n - The owning engineer fills a standard template: impact, timeline, detection, root cause, contributing factors, and corrective actions. The template exists lest something essential be omitted under pressure.
- Review. [/ˈreview/] n - I examine it for quality before it leaves the building: is the root cause a true cause or a symptom? Do the timestamps agree with the monitoring record? Are the action items owned and dated?
- Deliver. [/ˈdeliver/] n - It goes to the customer or leadership within the committed window. Timeliness is a deliverable, not a courtesy.
- Close the loop. [/ˈklose ˈthe ˈloop/] n - Corrective actions become tracked tickets. An RCA whose actions are never done is only a story.
- Retain. [/ˈretain/] n - The finished record is kept per the Data integrity and log retention policy, so that it may be found months later.
What Quality Signifies
A useful RCA distinguishes the trigger from the cause from the contributing factors. The field notes in these pages are good worked examples of that discipline:
- The Watts Hidden in Volt-Amps [/ˈwatts ˈhidden ˈin ˈvolt-amps/] n - the symptom was stranded capacity; the root cause was a metering quirk, not a bad sensor. An RCA that halted at “bad sensor” would have prescribed the wrong remedy.
- The Generator Was Not Overheating [/ˈthe ˈgenerator ˈwas ˈnot ˈoverheatɪŋ/] n - the test failure was the trigger; the true tale lay upstream. Good RCAs resist the easy first answer.
- The Fault That Would Not Be Found [/ˈkhasɪŋ ˈa ˈfantom ˈground ˈfault/] n - a reminder that “no fault found” is itself a finding, and belongs in the record.
Why the Workflow, and Not the Document Alone
I treat RCA as a workflow because the failure mode is almost never a bad writer; it is an RCA delivered late, an action item no one owns, a record that cannot be found at audit time. Tracking timeliness, quality, and retention as first-class metrics (reported up through Operational metrics and reporting) is what keeps the program honest, and what divides an operations center that learns from one that merely keeps a diary.