{"id":537,"date":"2022-01-31T02:01:11","date_gmt":"2022-01-31T02:01:11","guid":{"rendered":"https:\/\/fde.cat\/index.php\/2022\/01\/31\/sre-weekly-issue-307\/"},"modified":"2022-01-31T02:01:11","modified_gmt":"2022-01-31T02:01:11","slug":"sre-weekly-issue-307","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2022\/01\/31\/sre-weekly-issue-307\/","title":{"rendered":"SRE Weekly Issue #307"},"content":{"rendered":"

View on sreweekly.com<\/a><\/p>\n

\n

A message from our sponsor, Rootly<\/a>:<\/h2>\n

Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly shirt):
\nhttps:\/\/rootly.com\/demo\/?utm_source=sreweekly<\/a><\/p>\n<\/div>\n

Articles<\/h2>\n
\n
\n
\n
Roblox Return to Service 10\/28-10\/31 2021<\/a><\/div>\n
\n

This followup to their initial incident report<\/a> has a lot to learn from, especially if you run Consul at scale.<\/p>\n

\u00a0\u00a0Daniel Sturman and others \u2014 Roblox<\/p>\n<\/div>\n<\/div>\n

\n
Byford Dolphin Incident<\/a><\/div>\n
\n

This week, I came across the Byford Dolphin diving bell incident. This accident seems at face value to be \u201chuman error\u201d, but there\u2019s so much to it. Content warning: the accident was quite grisly.<\/p>\n

\u00a0\u00a0Wikipedia<\/p>\n<\/div>\n<\/div>\n

\n
What is Canary Analysis?<\/a><\/div>\n
\n

Canary testing is more than just deploying your code to a small part of your fleet. You need a plan for how you\u2019re going to spot problems.<\/p>\n

\u00a0\u00a0Jyoti Sahoo \u2014 OpsMx<\/p>\n<\/div>\n<\/div>\n

\n
Fixing Performance Regressions Before they Happen<\/a><\/div>\n
\n

My favorite part is how they look for changes in performance, rather than using a static threshold.<\/p>\n

\u00a0\u00a0Angus Croll \u2014 Netflix<\/p>\n<\/div>\n<\/div>\n

\n
How to Answer Network Outage Questions<\/a><\/div>\n
\n

It pays to think ahead about how you\u2019ll answer questions from execs during an incident.<\/p>\n

\u00a0\u00a0Chris Fenning \u2014 DZone<\/p>\n<\/div>\n<\/div>\n

\n
Incorrect proxying of 24 hostnames on January 24, 2022<\/a><\/div>\n
\n

On January 24, 2022, as a result of an internal Cloudflare product migration, 24 hostnames (including www.cloudflare.com) that were actively proxied through the Cloudflare global network were mistakenly redirected to the wrong origin.<\/p>\n

\u00a0\u00a0Jeremy Hartman \u2014 Cloudflare<\/p>\n<\/div>\n<\/div>\n

\n
Analyzing SRE Job Postings \u2013 From Amazon to Microsoft<\/a><\/div>\n
\n

An analysis of SRE job descriptions from 4 companies highlights what businesses actually expect SREs to do.<\/p>\n

\u00a0\u00a0JP Cheung \u2014 Rootly
\nThis article is published by my sponsor, Rootly, but their sponsorship did not influence its inclusion in this issue.<\/em><\/p>\n<\/div>\n<\/div>\n

\n
Google Team That Keeps Services Online Rocked by Mental Health Crisis<\/a><\/div>\n
\n

Members of the search giant\u2019s site reliability group say managers fostered a toxic environment. Google says a \u2018safe, inclusive workplace\u2019 is a top priority.<\/p>\n

\u00a0\u00a0Nico Grant \u2014 Bloomberg<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

Outages<\/h2>\n

Solana<\/a>
\niCloud<\/a>
\nTicketmaster<\/a>
\nAmazon Alexa<\/a>
\nReddit<\/a>
\nDiscord<\/a>
\nSRE WEEKLY<\/p>","protected":false},"excerpt":{"rendered":"

View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly shirt): https:\/\/rootly.com\/demo\/?utm_source=sreweekly Articles Roblox Return to Service 10\/28-10\/31… Continue reading SRE Weekly Issue #307<\/span><\/a><\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":""},"categories":[8],"tags":[],"class_list":["post-537","post","type-post","status-publish","format-standard","hentry","category-sre","entry"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":543,"url":"https:\/\/fde.cat\/index.php\/2022\/02\/21\/sre-weekly-issue-310\/","url_meta":{"origin":537,"position":0},"title":"SRE Weekly Issue #310","date":"February 21, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly shirt): https:\/\/rootly.com\/demo\/?utm_source=sreweekly Articles\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":579,"url":"https:\/\/fde.cat\/index.php\/2022\/05\/30\/sre-weekly-issue-324\/","url_meta":{"origin":537,"position":1},"title":"SRE Weekly Issue #324","date":"May 30, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging and adding responders, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly lego set): https:\/\/rootly.com\/demo\/\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":535,"url":"https:\/\/fde.cat\/index.php\/2022\/01\/24\/sre-weekly-issue-306\/","url_meta":{"origin":537,"position":2},"title":"SRE Weekly Issue #306","date":"January 24, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly shirt): https:\/\/rootly.com\/demo\/?utm_source=sreweekly Articles\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":546,"url":"https:\/\/fde.cat\/index.php\/2022\/03\/07\/sre-weekly-issue-312\/","url_meta":{"origin":537,"position":3},"title":"SRE Weekly Issue #312","date":"March 7, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly shirt): https:\/\/rootly.com\/demo\/?utm_source=sreweekly Articles\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":603,"url":"https:\/\/fde.cat\/index.php\/2022\/07\/04\/sre-weekly-issue-329\/","url_meta":{"origin":537,"position":4},"title":"SRE Weekly Issue #329","date":"July 4, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging and adding responders, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly lego set): https:\/\/rootly.com\/demo\/\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":540,"url":"https:\/\/fde.cat\/index.php\/2022\/02\/07\/sre-weekly-issue-308\/","url_meta":{"origin":537,"position":5},"title":"SRE Weekly Issue #308","date":"February 7, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly shirt): https:\/\/rootly.com\/demo\/?utm_source=sreweekly Articles\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/537","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/comments?post=537"}],"version-history":[{"count":0,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/537\/revisions"}],"wp:attachment":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/media?parent=537"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/categories?post=537"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/tags?post=537"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}