On Running the Root-Cause InquiryRetain56

On Running the Root-Cause Inquiry [/ɔn ˈrənɪŋ ðə ˈruːt-kause ˌɪnkˈwaɪˌri/] n - When an incident harms a customer, the fix is but half the work. The other half is the root cause analysis: the document that records what happened, why, and what shall prevent its recurrence. In my IOC operations role I own the RCA workflow; I do not author every RCA, but I see that each is opened, written well, reviewed, delivered on time, and retained.

Of the Lifecycle

  1. Trigger. [/ˈtrigger/] n - Any incident above a severity bar (or any SLA-impacting event) opens an RCA task automatically in the ITSM ticketing systems, linked to the parent incident so the timeline and alerts follow of themselves.
  2. Draft. [/ˈdraft/] n - The owning engineer fills a standard template: impact, timeline, detection, root cause, contributing factors, and corrective actions. The template exists lest something essential be omitted under pressure.
  3. Review. [/ˈreview/] n - I examine it for quality before it leaves the building: is the root cause a true cause or a symptom? Do the timestamps agree with the monitoring record? Are the action items owned and dated?
  4. Deliver. [/ˈdeliver/] n - It goes to the customer or leadership within the committed window. Timeliness is a deliverable, not a courtesy.
  5. Close the loop. [/ˈklose ˈthe ˈloop/] n - Corrective actions become tracked tickets. An RCA whose actions are never done is only a story.
  6. Retain. [/ˈretain/] n - The finished record is kept per the Data integrity and log retention policy, so that it may be found months later.

What Quality Signifies

A useful RCA distinguishes the trigger from the cause from the contributing factors. The field notes in these pages are good worked examples of that discipline:

  • The Watts Hidden in Volt-Amps [/ˈwatts ˈhidden ˈin ˈvolt-amps/] n - the symptom was stranded capacity; the root cause was a metering quirk, not a bad sensor. An RCA that halted at “bad sensor” would have prescribed the wrong remedy.
  • The Generator Was Not Overheating [/ˈthe ˈgenerator ˈwas ˈnot ˈoverheatɪŋ/] n - the test failure was the trigger; the true tale lay upstream. Good RCAs resist the easy first answer.
  • The Fault That Would Not Be Found [/ˈkhasɪŋ ˈa ˈfantom ˈground ˈfault/] n - a reminder that “no fault found” is itself a finding, and belongs in the record.

Why the Workflow, and Not the Document Alone

I treat RCA as a workflow because the failure mode is almost never a bad writer; it is an RCA delivered late, an action item no one owns, a record that cannot be found at audit time. Tracking timeliness, quality, and retention as first-class metrics (reported up through Operational metrics and reporting) is what keeps the program honest, and what divides an operations center that learns from one that merely keeps a diary.