{"id":574,"date":"2022-05-09T00:53:13","date_gmt":"2022-05-09T00:53:13","guid":{"rendered":"https:\/\/fde.cat\/index.php\/2022\/05\/09\/sre-weekly-issue-321\/"},"modified":"2022-05-09T00:53:13","modified_gmt":"2022-05-09T00:53:13","slug":"sre-weekly-issue-321","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2022\/05\/09\/sre-weekly-issue-321\/","title":{"rendered":"SRE Weekly Issue #321"},"content":{"rendered":"

View on sreweekly.com<\/a><\/p>\n

\n

A message from our sponsor, Rootly<\/a>:<\/h2>\n

Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging and adding responders, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly lego set):
\nhttps:\/\/rootly.com\/demo\/<\/a><\/p>\n<\/div>\n

Articles<\/h2>\n
\n
\n
\n
Using Fault Injection Testing to Improve DoorDash Reliability\u00a0<\/a><\/div>\n
\n

A researcher explains how they implemented their microservice failure testing tool at DoorDash. The tool, Fillibuster, automatically discovers microservice dependencies and injects faults, avoiding the need to design specific individual failure scenarios.<\/p>\n

\u00a0\u00a0Christopher Meiklejohn \u2014 DoorDash<\/p>\n<\/div>\n<\/div>\n

\n
Twitter: @ReinH on Atlassian\u2019s incident write-up<\/a><\/div>\n
\n

Last week, I shared Atlassian\u2019s outage write-up. This link is a Twitter thread with a critique.<\/p>\n

I feel like it is perhaps not a \u201cgood look\u201d to repeatedly try to sell your product in your writeup about your product\u2019s catastrophic outage<\/p>\n

\u00a0\u00a0@ReinH<\/p>\n<\/div>\n<\/div>\n

\n
usefulness of error<\/a><\/div>\n
\n

\u201cError\u201d serves a number of functions for an organization: as a defense against entanglement, the illusion of control, as a means for distancing, and as a marker for a failed investigation.<\/p>\n

\u00a0\u00a0Eric Dobbs<\/p>\n<\/div>\n<\/div>\n

\n
Incident Report for Enom (January 15, 2022)<\/a><\/div>\n
\n

This is a write-up posted in January for an incident that occurred during an infrastructure migration. I feel like I can relate to every one of the learnings.<\/p>\n

\u00a0\u00a0Enom (Tucows)<\/p>\n<\/div>\n<\/div>\n

\n
On-Call: Leave It Better Than You Found It<\/a><\/div>\n
\n

In the past two years, I\u2019ve been participating in on-call rotations as a Site Reliability Engineer at Vinted. Here are some of the practical lessons I\u2019ve learned about the process.<\/p>\n

\u00a0\u00a0Ernestas Narmontas<\/p>\n<\/div>\n<\/div>\n

\n
How SREs analyze risks to evaluate SLOs<\/a><\/div>\n
\n

This article is all about finding out what risks exist that may impact your ability to meet your SLOs. Once you\u2019ve done that, you can determine whether your SLOs are realistic.<\/p>\n

\u00a0\u00a0Ayelet Sachto \u2014 Google<\/p>\n<\/div>\n<\/div>\n

\n
How we aligned 200 teams to monitor services with SLOs <\/a><\/div>\n
\n

When your organization chooses to implement SLOs, how do you get everyone on board? This two-part series has an in-depth look at how Klarna did it.<\/p>\n

\u00a0\u00a0Andrew Cartine \u2014 Klarna<\/p>\n<\/div>\n<\/div>\n

\n
What is an SRE Product Manager?<\/a><\/div>\n
\n

Subtitle: And why do SRE teams need PMs?<\/em><\/p>\n

After laying out the reasons why SREs need PMs, this article goes into detail about what a PM can bring to an SRE team.<\/p>\n

\u00a0\u00a0Ant\u00f3nio Ara\u00fajo \u2014 detech.ai<\/p>\n<\/div>\n<\/div>\n

\n
BellJar: A new framework for testing system recoverability at scale<\/a><\/div>\n
\n

BellJar helps users find cyclic dependencies in their services, by running totally isolated VMs and requiring users to explicitly enable every external dependency they need in order to bootstrap each service. It has a really neat feature of automatically generating runbooks based on these test cases.<\/p>\n

\u00a0\u00a0Christopher Bunn and Jie Huang \u2014 Meta<\/p>\n<\/div>\n<\/div>\n

\n
Meltdown: Three Mile Island<\/a><\/div>\n
\n

This week, I watched Netflix\u2019s Meltdown: Three Mile Island<\/em>, a documentary about the nuclear accident in the US in 1979. It\u2019s not exactly a post-incident write-up, but there\u2019s a lot in there about normalization of deviance, situational awareness, and risk-taking (both in and out of incidents).<\/p>\n

\u00a0\u00a0Netflix<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

Outages<\/h2>\n

Slack<\/a><\/p>\n

and this one<\/a><\/p>\n

Heroku<\/a><\/p>\n

Heroku\u2019s been dealing with a security incident since April 13. They performed a mass password reset of all accounts and their GitHub integration has been disabled for days.<\/p>\n

Roblox<\/a>
\nSRE WEEKLY<\/p>","protected":false},"excerpt":{"rendered":"

View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging and adding responders, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly lego set): https:\/\/rootly.com\/demo\/ Articles Using Fault Injection Testing… Continue reading SRE Weekly Issue #321<\/span><\/a><\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":""},"categories":[8],"tags":[],"class_list":["post-574","post","type-post","status-publish","format-standard","hentry","category-sre","entry"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":543,"url":"https:\/\/fde.cat\/index.php\/2022\/02\/21\/sre-weekly-issue-310\/","url_meta":{"origin":574,"position":0},"title":"SRE Weekly Issue #310","date":"February 21, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly shirt): https:\/\/rootly.com\/demo\/?utm_source=sreweekly Articles\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":579,"url":"https:\/\/fde.cat\/index.php\/2022\/05\/30\/sre-weekly-issue-324\/","url_meta":{"origin":574,"position":1},"title":"SRE Weekly Issue #324","date":"May 30, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging and adding responders, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly lego set): https:\/\/rootly.com\/demo\/\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":535,"url":"https:\/\/fde.cat\/index.php\/2022\/01\/24\/sre-weekly-issue-306\/","url_meta":{"origin":574,"position":2},"title":"SRE Weekly Issue #306","date":"January 24, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly shirt): https:\/\/rootly.com\/demo\/?utm_source=sreweekly Articles\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":537,"url":"https:\/\/fde.cat\/index.php\/2022\/01\/31\/sre-weekly-issue-307\/","url_meta":{"origin":574,"position":3},"title":"SRE Weekly Issue #307","date":"January 31, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly shirt): https:\/\/rootly.com\/demo\/?utm_source=sreweekly Articles\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":546,"url":"https:\/\/fde.cat\/index.php\/2022\/03\/07\/sre-weekly-issue-312\/","url_meta":{"origin":574,"position":4},"title":"SRE Weekly Issue #312","date":"March 7, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly shirt): https:\/\/rootly.com\/demo\/?utm_source=sreweekly Articles\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":603,"url":"https:\/\/fde.cat\/index.php\/2022\/07\/04\/sre-weekly-issue-329\/","url_meta":{"origin":574,"position":5},"title":"SRE Weekly Issue #329","date":"July 4, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging and adding responders, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly lego set): https:\/\/rootly.com\/demo\/\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/574","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/comments?post=574"}],"version-history":[{"count":0,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/574\/revisions"}],"wp:attachment":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/media?parent=574"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/categories?post=574"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/tags?post=574"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}