{"id":723,"date":"2023-06-12T00:41:02","date_gmt":"2023-06-12T00:41:02","guid":{"rendered":"https:\/\/fde.cat\/index.php\/2023\/06\/12\/sre-weekly-issue-376\/"},"modified":"2023-06-12T00:41:02","modified_gmt":"2023-06-12T00:41:02","slug":"sre-weekly-issue-376","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2023\/06\/12\/sre-weekly-issue-376\/","title":{"rendered":"SRE Weekly Issue #376"},"content":{"rendered":"

View on sreweekly.com<\/a><\/p>\n

\n

A message from our sponsor, Rootly<\/a>:<\/h2>\n

Curious how companies like Figma, Tripadvisor, and 100s of others leverage Rootly to manage incidents in Slack and unlock instant best practices? Check out this lightning demo:
\nhttps:\/\/www.loom.com\/share\/051c4be0425a436e888dc0c3690855ad<\/a><\/p>\n<\/div>\n

Articles<\/h2>\n
\n
\n
\n
2023 03 08 Incident: A Deep Dive into Our Incident Response<\/a><\/div>\n
\n

With 100 workstreams and over 500 engineers engaged, this was the biggest incident response I\u2019ve read about in years.<\/p>\n

We had to force ourselves to identify the facts on the ground instead of \u201cwhat ought to be,\u201d and overrule our instincts to look for data in the places we normally looked (since our own monitoring was impacted).<\/p>\n

\u00a0\u00a0Laura de Vesine \u2014 Datadog<\/small><\/p>\n<\/div>\n<\/div>\n

\n
How the \u20183 Pillars of Observability\u2019 Miss the Big Picture<\/a><\/div>\n
\n

When you unify these three \u201cpillars\u201d into one cohesive approach, a new ability to understand the full state of your system in several new ways also emerges.<\/p>\n

\u00a0\u00a0Danyel Fisher \u2014 The New Stack<\/small>
\n\u00a0\u00a0Full disclosure: Honeycomb, my employer, is mentioned.<\/em><\/small><\/p>\n<\/div>\n<\/div>\n

\n
Azure DevOps Outage in South Brazil<\/a><\/div>\n
\n

This report details the 10-hour incident response following the accidental deletion of live databases (rather than their snapshots, as intended).<\/p>\n

\u00a0\u00a0Eric Mattingly \u2014 Azure<\/small><\/p>\n<\/div>\n<\/div>\n

\n
Show HN: Keep \u2013 Create production alerts from plain English<\/a><\/div>\n
\n

Neat trick: write your alerts in English and get GPT to convert them to real alert configurations.<\/p>\n

\u00a0\u00a0Shahar and Tal \u2014 Keep (via HackerNews)<\/small><\/p>\n<\/div>\n<\/div>\n

\n
A potential issue with outstanding query limits in your DNS resolver<\/a><\/div>\n
\n

If your DNS resolver is responsible for handling queries for both internal and external domains, what happens when external DNS requests fail? Can internal ones still proceed?<\/p>\n

\u00a0\u00a0Chris Siebenmann<\/small><\/p>\n<\/div>\n<\/div>\n

\n
Delusion Soup: How Observability Got Here, and What We Can Do About It<\/a><\/div>\n
\n

This article explains potential pitfalls and downsides to observability tools and the ways vendors might try to get you to use them, along with tips for how to avoid the traps.<\/p>\n

\u00a0\u00a0David Caudill<\/small><\/p>\n<\/div>\n<\/div>\n

\n
Treating uncertainty as a first-class concern<\/a><\/div>\n
\n

Too often, we dismiss the anomaly we just faced in an incident as a weird, one-off occurrence. And while that specific failure mode likely will be a one-off, we\u2019ll be faced with new anomalies in the future.<\/p>\n

\u00a0\u00a0Loron Hochstein \u2014 Surfing Complexity<\/small><\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

SRE WEEKLY<\/p>","protected":false},"excerpt":{"rendered":"

View on sreweekly.com A message from our sponsor, Rootly: Curious how companies like Figma, Tripadvisor, and 100s of others leverage Rootly to manage incidents in Slack and unlock instant best practices? Check out this lightning demo: https:\/\/www.loom.com\/share\/051c4be0425a436e888dc0c3690855ad Articles 2023 03 08 Incident: A Deep Dive into Our Incident Response With 100 workstreams and over 500… Continue reading SRE Weekly Issue #376<\/span><\/a><\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":""},"categories":[8],"tags":[],"class_list":["post-723","post","type-post","status-publish","format-standard","hentry","category-sre","entry"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":543,"url":"https:\/\/fde.cat\/index.php\/2022\/02\/21\/sre-weekly-issue-310\/","url_meta":{"origin":723,"position":0},"title":"SRE Weekly Issue #310","date":"February 21, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly shirt): https:\/\/rootly.com\/demo\/?utm_source=sreweekly Articles\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":535,"url":"https:\/\/fde.cat\/index.php\/2022\/01\/24\/sre-weekly-issue-306\/","url_meta":{"origin":723,"position":1},"title":"SRE Weekly Issue #306","date":"January 24, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly shirt): https:\/\/rootly.com\/demo\/?utm_source=sreweekly Articles\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":579,"url":"https:\/\/fde.cat\/index.php\/2022\/05\/30\/sre-weekly-issue-324\/","url_meta":{"origin":723,"position":2},"title":"SRE Weekly Issue #324","date":"May 30, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging and adding responders, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly lego set): https:\/\/rootly.com\/demo\/\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":546,"url":"https:\/\/fde.cat\/index.php\/2022\/03\/07\/sre-weekly-issue-312\/","url_meta":{"origin":723,"position":3},"title":"SRE Weekly Issue #312","date":"March 7, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly shirt): https:\/\/rootly.com\/demo\/?utm_source=sreweekly Articles\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":537,"url":"https:\/\/fde.cat\/index.php\/2022\/01\/31\/sre-weekly-issue-307\/","url_meta":{"origin":723,"position":4},"title":"SRE Weekly Issue #307","date":"January 31, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly shirt): https:\/\/rootly.com\/demo\/?utm_source=sreweekly Articles\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":603,"url":"https:\/\/fde.cat\/index.php\/2022\/07\/04\/sre-weekly-issue-329\/","url_meta":{"origin":723,"position":5},"title":"SRE Weekly Issue #329","date":"July 4, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging and adding responders, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly lego set): https:\/\/rootly.com\/demo\/\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/723","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/comments?post=723"}],"version-history":[{"count":0,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/723\/revisions"}],"wp:attachment":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/media?parent=723"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/categories?post=723"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/tags?post=723"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}