{"id":717,"date":"2023-05-22T01:21:37","date_gmt":"2023-05-22T01:21:37","guid":{"rendered":"https:\/\/fde.cat\/index.php\/2023\/05\/22\/sre-weekly-issue-373\/"},"modified":"2023-05-22T01:21:37","modified_gmt":"2023-05-22T01:21:37","slug":"sre-weekly-issue-373","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2023\/05\/22\/sre-weekly-issue-373\/","title":{"rendered":"SRE Weekly Issue #373"},"content":{"rendered":"

View on sreweekly.com<\/a><\/p>\n

\n

A message from our sponsor, Rootly<\/a>:<\/h2>\n

Rootly is hiring for a Sr. Developer Relations Advocate to continue helping more world-class companies like Figma, NVIDIA, Squarespace, accelerate their incident management journey. Looking for previous on-call engineers with a passion for making the world a more reliable place. Learn more:<\/p>\n

https:\/\/rootly.com\/careers?gh_jid=4015888007<\/a><\/p>\n<\/div>\n

Articles<\/h2>\n
\n
\n
\n
2023 03 08 Incident: Infrastructure connectivity issue affecting multiple regions<\/a><\/div>\n
\n

Datadog posted a report on their major outage in March, and it\u2019s a doozy. An unattended updates system that they didn\u2019t even want, need, or know about triggered across all hosts in multiple clouds nearly simultaneously, causing a regression.<\/p>\n

\u00a0\u00a0Alexis L\u00ea-Qu\u00f4c \u2014 Datadog<\/small><\/p>\n<\/div>\n<\/div>\n

\n
Addressing GitHub\u2019s recent availability issues<\/a><\/div>\n
\n

GitHub has had a string of apparently unrelated outages recently, and they\u2019ve posted this description.<\/p>\n

\u00a0\u00a0Mike Hanley \u2014 GitHub<\/small><\/p>\n<\/div>\n<\/div>\n

\n
StanzaSystems\/awesome-load-management<\/a><\/div>\n
\n

Oh look, another awesome-* repo relevant to our interests!<\/p>\n

A repo of links to articles, papers, conference talks, and tooling related to load management in software services: loadshedding, circuitbreaking, quota management and throttling. PRs welcome.<\/p>\n

\u00a0\u00a0Laura Nolan and Niall Murphy \u2014 Stanza Systems<\/small><\/p>\n<\/div>\n<\/div>\n

\n
SRE Story with Matthew Iselin<\/a><\/div>\n
\n

This interview covers a lot of ground including looking beyond just \u201cup or down\u201d when considering reliability.<\/p>\n

\u00a0\u00a0Prathamesh Sonpatki \u2014 SRE Stories<\/small><\/p>\n<\/div>\n<\/div>\n

\n
Debugging a FUSE deadlock in the Linux kernel<\/a><\/div>\n
\n

If you\u2019re in the mood for a deep systems debugging story, you\u2019re in for a treat. The author takes you along for the ride with a wealth of detailed code snippets.<\/p>\n

\u00a0\u00a0Tycho Andersen \u2014 Netflix<\/small><\/p>\n<\/div>\n<\/div>\n

\n
Why `fsync()`: Losing Unsynced Data<\/a><\/div>\n
\n

Regardless of the replication mechanism you must fsync() your data to prevent global data loss in non-Byzantine protocols.<\/p>\n

\u00a0\u00a0Denis Rystsov and Alexander Gallego \u2014 Redpanda<\/small><\/p>\n<\/div>\n<\/div>\n

\n
Emotional Intelligence<\/a><\/div>\n
\n

Emotional intelligence is a critical skill for SREs, especially when we interact with other teams in fraught situations.<\/p>\n

\u00a0\u00a0Amin Astaneh \u2014 Certo Modo<\/small><\/p>\n<\/div>\n<\/div>\n

\n
Fleet Management at Spotify (Part 3): Fleet-wide Refactoring<\/a><\/div>\n
\n

Wow! Spotify created a set of tools to perform automated refactoring of thousands of repositories at once. This includes the ability to run tests, automatically merge pull requests without human review, and roll refactorings out gradually.<\/p>\n

\u00a0\u00a0Matt Brown \u2014 Spotify<\/small><\/p>\n<\/div>\n<\/div>\n

\n
Teach me how to Howie!<\/a><\/div>\n
\n

Jeli has published a one-page cheat-sheet for their highly-detailed Howie guide for running incident retrospectives.<\/p>\n

\u00a0\u00a0Jeli<\/small><\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

SRE WEEKLY<\/p>","protected":false},"excerpt":{"rendered":"

View on sreweekly.com A message from our sponsor, Rootly: Rootly is hiring for a Sr. Developer Relations Advocate to continue helping more world-class companies like Figma, NVIDIA, Squarespace, accelerate their incident management journey. Looking for previous on-call engineers with a passion for making the world a more reliable place. Learn more: https:\/\/rootly.com\/careers?gh_jid=4015888007 Articles 2023 03… Continue reading SRE Weekly Issue #373<\/span><\/a><\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":""},"categories":[8],"tags":[],"class_list":["post-717","post","type-post","status-publish","format-standard","hentry","category-sre","entry"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":543,"url":"https:\/\/fde.cat\/index.php\/2022\/02\/21\/sre-weekly-issue-310\/","url_meta":{"origin":717,"position":0},"title":"SRE Weekly Issue #310","date":"February 21, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly shirt): https:\/\/rootly.com\/demo\/?utm_source=sreweekly Articles\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":535,"url":"https:\/\/fde.cat\/index.php\/2022\/01\/24\/sre-weekly-issue-306\/","url_meta":{"origin":717,"position":1},"title":"SRE Weekly Issue #306","date":"January 24, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly shirt): https:\/\/rootly.com\/demo\/?utm_source=sreweekly Articles\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":579,"url":"https:\/\/fde.cat\/index.php\/2022\/05\/30\/sre-weekly-issue-324\/","url_meta":{"origin":717,"position":2},"title":"SRE Weekly Issue #324","date":"May 30, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging and adding responders, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly lego set): https:\/\/rootly.com\/demo\/\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":546,"url":"https:\/\/fde.cat\/index.php\/2022\/03\/07\/sre-weekly-issue-312\/","url_meta":{"origin":717,"position":3},"title":"SRE Weekly Issue #312","date":"March 7, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly shirt): https:\/\/rootly.com\/demo\/?utm_source=sreweekly Articles\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":537,"url":"https:\/\/fde.cat\/index.php\/2022\/01\/31\/sre-weekly-issue-307\/","url_meta":{"origin":717,"position":4},"title":"SRE Weekly Issue #307","date":"January 31, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo (+ get a snazzy Rootly shirt): https:\/\/rootly.com\/demo\/?utm_source=sreweekly Articles\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":521,"url":"https:\/\/fde.cat\/index.php\/2021\/12\/27\/sre-weekly-issue-302\/","url_meta":{"origin":717,"position":5},"title":"SRE Weekly Issue #302","date":"December 27, 2021","format":false,"excerpt":"View on sreweekly.com Happy holidays, for those that celebrate! I put this issue together in advance, so no Outages section this week. A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/717","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/comments?post=717"}],"version-history":[{"count":0,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/717\/revisions"}],"wp:attachment":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/media?parent=717"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/categories?post=717"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/tags?post=717"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}