{"id":611,"date":"2022-07-25T01:10:03","date_gmt":"2022-07-25T01:10:03","guid":{"rendered":"https:\/\/fde.cat\/index.php\/2022\/07\/25\/sre-weekly-issue-331\/"},"modified":"2022-07-25T01:10:03","modified_gmt":"2022-07-25T01:10:03","slug":"sre-weekly-issue-331","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2022\/07\/25\/sre-weekly-issue-331\/","title":{"rendered":"SRE Weekly Issue #331"},"content":{"rendered":"

View on sreweekly.com<\/a><\/p>\n

\n

A message from our sponsor, Rootly<\/a>:<\/h2>\n

Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging and adding responders, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly lego set):
\nhttps:\/\/rootly.com\/demo\/<\/a><\/p>\n<\/div>\n

Articles<\/h2>\n
\n
\n
\n
DisasterCast \u2013 A podcast about scary things and how to stop them happening<\/a><\/div>\n
\n

I\u2019ve been listening to this podcast this week and I love it! Each episode covers a disaster, safety theory, and other topics \u2014 with no ads. Their site is down right now, but the podcast is available on the usual platforms.<\/p>\n

\u00a0\u00a0Drew Rae \u2014 DisasterCast<\/p>\n<\/div>\n<\/div>\n

\n
An 8 Step Guide to Go From a Clueless to a Production-aware Software Engineer<\/a><\/div>\n
\n

If we want to get folks to own their code in production, we need to teach them how to think like an SRE.<\/p>\n

\u00a0\u00a0Boris Cherkasky<\/p>\n<\/div>\n<\/div>\n

\n
3 mistakes I\u2019ve made at the beginning of an incident (and how not to make them)<\/a><\/div>\n
\n

Let\u2019s look at three mistakes I\u2019ve made during those stressful moments during the beginning of an incident \u2014 and discuss how you can avoid making them.<\/p>\n

The mistakes are:<\/p>\n

Mistake 1: We didn\u2019t have a plan.
\nMistake 2: We weren\u2019t production ready.
\nMistake 3: We fell down a cognitive tunnel. <\/p>\n

\u00a0\u00a0Robert Ross \u2014 FireHydrant<\/p>\n<\/div>\n<\/div>\n

\n
When to kill the canary<\/a><\/div>\n
\n

At what point does your canary test indicate failure? Should the criteria be the same as your normal production alerting?<\/p>\n

\u00a0\u00a0\u00d8ystein Blixhavn<\/p>\n<\/div>\n<\/div>\n

\n
On Counting Alerts<\/a><\/div>\n
\n

This is a followup to a previous article<\/a> about on-call health. In this one, the author shares metrics about the number of alerts and discusses whether this number is useful.<\/p>\n

\u00a0\u00a0Fred Hebert \u2014 Honeycomb<\/p>\n<\/div>\n<\/div>\n

\n
High Availability on Razorpay Payments Dashboard<\/a><\/div>\n
\n

Their dashboard crashed for 50% of user sessions, so they had a lot of work ahead of them. Find out how they got crash-free sessions to 99.9% and improved their time to respond to incidents.<\/p>\n

\u00a0\u00a0Sandesh Damkondwar \u2014 Razorpay<\/p>\n<\/div>\n<\/div>\n

\n
@atoonk on Twitter summarizing the Rogers Communications outage<\/a><\/div>\n
\n

Rogers Communications, a major telecom in Canada, had a country-wide outage earlier this month. I don\u2019t normally include telecom outages in the Outages section because they rarely share information that we can learn from.<\/p>\n

This time, Rogers released a (redacted) report on their outage, and this Twitter thread summarizes the key points.<\/p>\n

\u00a0\u00a0@atoonk on Twitter<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

Outages<\/h2>\n

Microsoft Teams and Office 365<\/a>
\nMicrosoft blames storage error for Teams outage<\/a>
\nGoogle Cloud Storage<\/a>
\nGoogle Cloud europe-west2 region<\/a><\/p>\n

Preliminary root cause has been identified as multiple concurrent failures to our redundant cooling systems within one of the buildings that hosts the europe-west2-a zone for the europe-west2 region.<\/p>\n

SRE WEEKLY<\/p>","protected":false},"excerpt":{"rendered":"

View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging and adding responders, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly lego set): https:\/\/rootly.com\/demo\/ Articles DisasterCast \u2013 A podcast… Continue reading SRE Weekly Issue #331<\/span><\/a><\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":""},"categories":[8],"tags":[],"class_list":["post-611","post","type-post","status-publish","format-standard","hentry","category-sre","entry"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":556,"url":"https:\/\/fde.cat\/index.php\/2022\/03\/21\/sre-weekly-issue-314\/","url_meta":{"origin":611,"position":0},"title":"SRE Weekly Issue #314","date":"March 21, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging and adding responders, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly lego set): https:\/\/rootly.com\/demo\/\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":543,"url":"https:\/\/fde.cat\/index.php\/2022\/02\/21\/sre-weekly-issue-310\/","url_meta":{"origin":611,"position":1},"title":"SRE Weekly Issue #310","date":"February 21, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly shirt): https:\/\/rootly.com\/demo\/?utm_source=sreweekly Articles\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":659,"url":"https:\/\/fde.cat\/index.php\/2022\/12\/05\/sre-weekly-issue-350\/","url_meta":{"origin":611,"position":2},"title":"SRE Weekly Issue #350","date":"December 5, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly\u00a0\ud83d\ude92. Rootly automates manual tasks like creating an incident channel, Jira ticket and Zoom rooms, inviting responders, creating statuspage updates, postmortem timelines and more. Want to see why companies like Canva and Grammarly love us?:\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":579,"url":"https:\/\/fde.cat\/index.php\/2022\/05\/30\/sre-weekly-issue-324\/","url_meta":{"origin":611,"position":3},"title":"SRE Weekly Issue #324","date":"May 30, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging and adding responders, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly lego set): https:\/\/rootly.com\/demo\/\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":535,"url":"https:\/\/fde.cat\/index.php\/2022\/01\/24\/sre-weekly-issue-306\/","url_meta":{"origin":611,"position":4},"title":"SRE Weekly Issue #306","date":"January 24, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly shirt): https:\/\/rootly.com\/demo\/?utm_source=sreweekly Articles\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":537,"url":"https:\/\/fde.cat\/index.php\/2022\/01\/31\/sre-weekly-issue-307\/","url_meta":{"origin":611,"position":5},"title":"SRE Weekly Issue #307","date":"January 31, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly shirt): https:\/\/rootly.com\/demo\/?utm_source=sreweekly Articles\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/611","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/comments?post=611"}],"version-history":[{"count":0,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/611\/revisions"}],"wp:attachment":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/media?parent=611"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/categories?post=611"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/tags?post=611"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}