{"id":519,"date":"2021-12-20T02:30:54","date_gmt":"2021-12-20T02:30:54","guid":{"rendered":"https:\/\/fde.cat\/index.php\/2021\/12\/20\/sre-weekly-issue-301\/"},"modified":"2021-12-20T02:30:54","modified_gmt":"2021-12-20T02:30:54","slug":"sre-weekly-issue-301","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2021\/12\/20\/sre-weekly-issue-301\/","title":{"rendered":"SRE Weekly Issue #301"},"content":{"rendered":"

View on sreweekly.com<\/a><\/p>\n

\n

A message from our sponsor, Rootly<\/a>:<\/h2>\n

Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo:<\/span><\/p>\n

https:\/\/rootly.com\/demo\/?utm_source=sreweekly<\/a><\/p>\n<\/div>\n

Articles<\/h2>\n
\n
\n
\n
BadgerDAO Exploit Technical Post Mortem <\/a><\/div>\n
\n

This one perhaps belongs in a security newsletter, but the failure mode is just so fascinating. A CDN bug led to the loss of millions of dollars worth of Bitcoin.<\/p>\n

\u00a0\u00a0Badger<\/p>\n<\/div>\n<\/div>\n

\n
Google Calendar Incident Report for December 8th Outage<\/a><\/div>\n
\n

Google posted a report for the Google Calendar outage last week.<\/p>\n

\u00a0\u00a0Google<\/p>\n<\/div>\n<\/div>\n

\n
How We Got Here: Incident Report September 2021\u2014Slow \/ Failed Loading Investigations<\/a><\/div>\n
\n

Jeli, authors of the Howie post-incident guide<\/a> has their own \u201chowie\u201d. It\u2019s a great example of a thorough incident report.<\/p>\n

\u00a0\u00a0Vanessa Huerta Granda \u2014 Jeli<\/p>\n<\/div>\n<\/div>\n

\n
A Site Reliability Engineer\u2019s Guide to the Holiday Season<\/a><\/div>\n
\n

Hopefully not too late, here are some tips as we head into the thick of it.<\/p>\n

\u00a0\u00a0JJ Tang \u2014 Rootly
\nThis article is published by my sponsor, Rootly, but their sponsorship did not influence its inclusion in this issue.<\/em><\/p>\n<\/div>\n<\/div>\n

\n
How to Write Meaningful Retrospectives<\/a><\/div>\n
\n

Using their own incident retrospective template<\/a>, Blameless shows us how to write an incident retrospective.<\/p>\n

\u00a0\u00a0Emily Arnott \u2014 Blameless<\/p>\n<\/div>\n<\/div>\n

\n
SLICK: Adopting SLOs for improved reliability<\/a><\/div>\n
\n

Meta has their own in-house-built tool for tracking and reporting on SLIs.<\/p>\n

\u00a0\u00a0A Posten, D\u00e1vid Bart\u00f3k, Filip Klepo, and Vatika Harlalka \u2014 Meta<\/p>\n<\/div>\n<\/div>\n

\n
On-call by default<\/a><\/div>\n
\n

These folks put everyone on-call by default, and also pay them extra automatically for each shift and even covering for coworkers.<\/p>\n

\u00a0\u00a0Chris Evans \u2014 incident.io<\/p>\n<\/div>\n<\/div>\n

\n
Incident Report: The Missing Trigger Notification Emails<\/a><\/div>\n
\n

Code that was deployed under a feature flag inadvertently affected all traffic, even with the flag disabled.<\/p>\n

\u00a0\u00a0Steve Lewis \u2014 Honeycomb<\/p>\n<\/div>\n<\/div>\n

\n
What made SLOs so messy (and what we can do about it)<\/a><\/div>\n
\n

By creating SLOs for microservices at various levels of the request tree, they ended up with a morass of arbitrary targets that didn\u2019t relate clearly to the user experience.<\/p>\n

\u00a0\u00a0Ben Sigelman \u2014 Lightstep<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

Outages<\/h2>\n

AWS us-west-1 and us-west-2<\/a><\/p>\n

Hot on the heels of last week\u2019s us-east-1 outage, AWS had a shorter outage in us-west-1 and us-west-2.<\/p>\n

PagerDuty<\/a><\/p>\n

PagerDuty alert notifications were affected by the AWS us-west-2 outage, and the impact lasted about twice as long as AWS\u2019s.<\/p>\n

Slack<\/a>
\nCloudflare<\/a>
\nSolana<\/a>
\nSRE WEEKLY<\/p>","protected":false},"excerpt":{"rendered":"

View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo: https:\/\/rootly.com\/demo\/?utm_source=sreweekly Articles BadgerDAO Exploit Technical Post Mortem This one perhaps belongs in a… Continue reading SRE Weekly Issue #301<\/span><\/a><\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":""},"categories":[8],"tags":[],"class_list":["post-519","post","type-post","status-publish","format-standard","hentry","category-sre","entry"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":543,"url":"https:\/\/fde.cat\/index.php\/2022\/02\/21\/sre-weekly-issue-310\/","url_meta":{"origin":519,"position":0},"title":"SRE Weekly Issue #310","date":"February 21, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly shirt): https:\/\/rootly.com\/demo\/?utm_source=sreweekly Articles\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":579,"url":"https:\/\/fde.cat\/index.php\/2022\/05\/30\/sre-weekly-issue-324\/","url_meta":{"origin":519,"position":1},"title":"SRE Weekly Issue #324","date":"May 30, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging and adding responders, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly lego set): https:\/\/rootly.com\/demo\/\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":535,"url":"https:\/\/fde.cat\/index.php\/2022\/01\/24\/sre-weekly-issue-306\/","url_meta":{"origin":519,"position":2},"title":"SRE Weekly Issue #306","date":"January 24, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly shirt): https:\/\/rootly.com\/demo\/?utm_source=sreweekly Articles\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":540,"url":"https:\/\/fde.cat\/index.php\/2022\/02\/07\/sre-weekly-issue-308\/","url_meta":{"origin":519,"position":3},"title":"SRE Weekly Issue #308","date":"February 7, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly shirt): https:\/\/rootly.com\/demo\/?utm_source=sreweekly Articles\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":537,"url":"https:\/\/fde.cat\/index.php\/2022\/01\/31\/sre-weekly-issue-307\/","url_meta":{"origin":519,"position":4},"title":"SRE Weekly Issue #307","date":"January 31, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly shirt): https:\/\/rootly.com\/demo\/?utm_source=sreweekly Articles\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":546,"url":"https:\/\/fde.cat\/index.php\/2022\/03\/07\/sre-weekly-issue-312\/","url_meta":{"origin":519,"position":5},"title":"SRE Weekly Issue #312","date":"March 7, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly shirt): https:\/\/rootly.com\/demo\/?utm_source=sreweekly Articles\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/519","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/comments?post=519"}],"version-history":[{"count":0,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/519\/revisions"}],"wp:attachment":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/media?parent=519"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/categories?post=519"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/tags?post=519"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}