On Firmware, A Thousand Servers at a TimeOn Firmware, A Thousand Servers at a Time136

The hardware vendor had released a critical security patch for the management interfaces of our servers, and we kept more than 200 servers of the same model in our datacenter. To update the firmware by hand would have devoured time and invited error; I was charged with deploying the update to all 200 quickly and safely, and with seeing that none failed along the way.

I wrote a script against the application programming interface (API) of the server management software. It authenticated with each server and uploaded the new firmware image. I tried it first upon three non-critical servers; the trial succeeded. I then configured it to update the remainder in small groups of ten, waiting for each group to restart and verify the new firmware version before passing to the next.

The script completed all 200 servers in four hours. Monitoring confirmed every management interface online and running the new firmware version. No server failed; the vulnerability stood fully patched.