{"id":616,"date":"2022-08-01T02:28:34","date_gmt":"2022-08-01T02:28:34","guid":{"rendered":"https:\/\/fde.cat\/index.php\/2022\/08\/01\/sre-weekly-issue-332\/"},"modified":"2022-08-01T02:28:34","modified_gmt":"2022-08-01T02:28:34","slug":"sre-weekly-issue-332","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2022\/08\/01\/sre-weekly-issue-332\/","title":{"rendered":"SRE Weekly Issue #332"},"content":{"rendered":"

View on sreweekly.com<\/a><\/p>\n

\n

A message from our sponsor, Rootly<\/a>:<\/h2>\n

Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging and adding responders, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly lego set):
\nhttps:\/\/rootly.com\/demo\/<\/a><\/p>\n<\/div>\n

Articles<\/h2>\n
\n
\n
\n
How Razorpay\u2019s Notification Service Handles Increasing Load<\/a><\/div>\n
\n

Their notification service had complex load characteristics that made scaling up a tricky proposition.<\/p>\n

\u00a0\u00a0Anand Prakash \u2014 Razorpay<\/p>\n<\/div>\n<\/div>\n

\n
How we improved on-call life by reducing pager noise<\/a><\/div>\n
\n

Coalescing alerts and adding dependencies in AlertManager were the key to reducing this team\u2019s excessive pager load.<\/p>\n

\u00a0\u00a0steveazz \u2014 GitLab<\/p>\n<\/div>\n<\/div>\n

\n
What\u2019s allowed to count as a cause: ALERRT edition<\/a><\/div>\n
\n

Lorin Hochstein has started a series<\/a> of blog posts on what we can learn about incident response from the Uvalde school shooting tragedy in the US. This article looks at how an organization\u2019s perspective can affect their retrospective incident analysis.<\/p>\n

\u00a0\u00a0Lorin Hochstein<\/p>\n<\/div>\n<\/div>\n

\n
The fog of war in Uvalde<\/a><\/div>\n
\n

My claim here is that we should assume the officer is telling the truth and was acting reasonably if we want to understand how these types of failure modes can happen.<\/p>\n

Every retrospective ever:<\/p>\n

We must assume that a person can act reasonably and still come to the wrong conclusion in order to make progress.<\/p>\n

\u00a0\u00a0Lorin Hochstein<\/p>\n<\/div>\n<\/div>\n

\n
User settings, Lamport clocks and lightweight formal methods<\/a><\/div>\n
\n

How do you synchronize state between multiple browsers and a backend, and ensure that everyone\u2019s state will eventually converge? These folks explain how they did it, and a bug they found through testing.<\/p>\n

\u00a0\u00a0Jakub Mikians \u2014 Airspace Intelligence<\/p>\n<\/div>\n<\/div>\n

\n
MTTR: lower isn\u2019t always better<\/a><\/div>\n
\n

MTTR is a mean, so it doesn\u2019t tell you anything about the number<\/em> of incidents, among other potential pitfalls.<\/p>\n

\u00a0\u00a0Dan Slimmon<\/p>\n<\/div>\n<\/div>\n

\n
Google Cloud Platform outage report: europe-west2 cooling failure<\/a><\/div>\n
\n

Last week, I included a GCP outage in europe-west2. This week, Google posted this report about what went wrong, and it\u2019s got layers.<\/p>\n

Bonus: another GCP outage report<\/a><\/p>\n

\u00a0\u00a0Google<\/p>\n<\/div>\n<\/div>\n

\n
It\u2019s time to leave the leap second in the past<\/a><\/div>\n
\n

Meta wants to do away with leap seconds, because they make it especially difficult to create reliable systems.<\/p>\n

\u00a0\u00a0Oleg Obleukhov and Ahmad Byagowi \u2014 Meta<\/p>\n<\/div>\n<\/div>\n

\n
3 common pitfalls of post-mortems<\/a><\/div>\n
\n

If you\u2019re anywhere near incident analysis in your organization, you need to read this list.<\/p>\n

\u00a0\u00a0Milly Leadley \u2014 incident.io<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

Outages<\/h2>\n

Stack Exchange<\/a>
\nMicrosoft 365<\/a>
\nSlack<\/a>
\nTrello<\/a>
\nSRE WEEKLY<\/p>","protected":false},"excerpt":{"rendered":"

View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging and adding responders, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly lego set): https:\/\/rootly.com\/demo\/ Articles How Razorpay\u2019s Notification Service… Continue reading SRE Weekly Issue #332<\/span><\/a><\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":""},"categories":[8],"tags":[],"class_list":["post-616","post","type-post","status-publish","format-standard","hentry","category-sre","entry"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":519,"url":"https:\/\/fde.cat\/index.php\/2021\/12\/20\/sre-weekly-issue-301\/","url_meta":{"origin":616,"position":0},"title":"SRE Weekly Issue #301","date":"December 20, 2021","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo: https:\/\/rootly.com\/demo\/?utm_source=sreweekly Articles BadgerDAO Exploit Technical Post Mortem This\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":543,"url":"https:\/\/fde.cat\/index.php\/2022\/02\/21\/sre-weekly-issue-310\/","url_meta":{"origin":616,"position":1},"title":"SRE Weekly Issue #310","date":"February 21, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly shirt): https:\/\/rootly.com\/demo\/?utm_source=sreweekly Articles\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":579,"url":"https:\/\/fde.cat\/index.php\/2022\/05\/30\/sre-weekly-issue-324\/","url_meta":{"origin":616,"position":2},"title":"SRE Weekly Issue #324","date":"May 30, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging and adding responders, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly lego set): https:\/\/rootly.com\/demo\/\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":535,"url":"https:\/\/fde.cat\/index.php\/2022\/01\/24\/sre-weekly-issue-306\/","url_meta":{"origin":616,"position":3},"title":"SRE Weekly Issue #306","date":"January 24, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly shirt): https:\/\/rootly.com\/demo\/?utm_source=sreweekly Articles\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":537,"url":"https:\/\/fde.cat\/index.php\/2022\/01\/31\/sre-weekly-issue-307\/","url_meta":{"origin":616,"position":4},"title":"SRE Weekly Issue #307","date":"January 31, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly shirt): https:\/\/rootly.com\/demo\/?utm_source=sreweekly Articles\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":546,"url":"https:\/\/fde.cat\/index.php\/2022\/03\/07\/sre-weekly-issue-312\/","url_meta":{"origin":616,"position":5},"title":"SRE Weekly Issue #312","date":"March 7, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly shirt): https:\/\/rootly.com\/demo\/?utm_source=sreweekly Articles\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/616","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/comments?post=616"}],"version-history":[{"count":0,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/616\/revisions"}],"wp:attachment":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/media?parent=616"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/categories?post=616"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/tags?post=616"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}