SRE Weekly Issue #388

This article makes a cool analogy between designing systems to operate well under unexpected load and designing socio-technical systems that operate well when the people are surprised by what the system is doing.

  Lorin Hochstein

If you need to create SLAs, this article has some solid advice on how to go about it — and what to avoid.

  incident.io

If Prometheus can’t scrape your service, an alert can get resolved incorrectly — and that can happen exactly when your service is failing!

  Chris Siebenmann

A really nifty three-part exploration of action items in the aftermath of an incidents. Rather than consider cost/benefit, this article series proposes that we think about the likelihood of an action item being completed.

  J. Paul Reed

Yes, as it turns out — and these folks have the receipts (along with some theories as to why).

  Colin Bartlett

The “wow” moment in this article is under the heading, “What can we learn from creative desperation?”

  Eric Dobbs — Learning From Incidents

Before explaining how they set up their on-call, these folks share why they avoided it in the early stages of their startup, and what made them finally take the plunge.

  Dustin Brown — DoltHub

For the good of the profession, the SRE community still needs to coalesce around more consistent job ladders, expectations, and competencies.

  Code Reliant

Honeycomb had their worst incident ever at the end of July, and in their characteristic style, they’ve posted an incredibly detailed analysis of what happened — and that’s just the blog post. Then you can click through for a 17-page PDF with lots more detail.

  Fred Hebert — Honeycomb
  Full disclosure: Honeycomb is my employer.