{"id":874,"date":"2024-06-10T01:40:20","date_gmt":"2024-06-10T01:40:20","guid":{"rendered":"https:\/\/fde.cat\/index.php\/2024\/06\/10\/sre-weekly-issue-428\/"},"modified":"2024-06-10T01:40:20","modified_gmt":"2024-06-10T01:40:20","slug":"sre-weekly-issue-428","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2024\/06\/10\/sre-weekly-issue-428\/","title":{"rendered":"SRE Weekly Issue #428"},"content":{"rendered":"

View on sreweekly.com<\/a><\/p>\n

\n

A message from our sponsor, FireHydrant<\/a>:<\/h2>\n

We\u2019ve gone all out on our new integration with Microsoft Teams. If you\u2019re a MS Teams user, FireHydrant now supports the most comprehensive integration for incident management. Run the entire IM process without ever leaving the chat.<\/p>\n

https:\/\/firehydrant.com\/blog\/introducing-a-brand-new-microsoft-teams-integration\/<\/a><\/p>\n<\/div>\n

\n
\n
\n
The Reverse Red Herring<\/a><\/div>\n
\n

This article presents in incident theme that I\u2019ve lived through many times but never had such a pithy name for.<\/p>\n

\u00a0\u00a0Geoff Townsend \u2014 Blameless<\/small><\/p>\n<\/div>\n<\/div>\n

\n
Centralisation and distribution: When one node is enough<\/a><\/div>\n
\n

There are risks and downsides inherent in a distributed system, so it\u2019s worth thinking about whether you really need one.<\/p>\n

\u00a0\u00a0Pipitz \u2014 Adevinta<\/small><\/p>\n<\/div>\n<\/div>\n

\n
Not Just Scale<\/a><\/div>\n
\n

And here\u2019s a counterpoint to the previous article: deciding whether you need a distributed system isn\u2019t just about scale.<\/p>\n

\u00a0\u00a0Marc Brooker<\/small><\/p>\n<\/div>\n<\/div>\n

\n
Use Memes<\/a><\/div>\n
\n

The effectiveness of memes in availability campaigns.<\/p>\n

This short post is a pile of memes, and the video one is top notch.<\/p>\n

\u00a0\u00a0Ross Brodbeck<\/small><\/p>\n<\/div>\n<\/div>\n

\n
Flaky alerts are telling you something<\/a><\/div>\n
\n

Paraphrasing part of this article: either you didn\u2019t understand your system fully when you wrote the alert, or there really are sporadic failures.<\/p>\n

\u00a0\u00a0Chris Siebenmann<\/small><\/p>\n<\/div>\n<\/div>\n

\n
You can\u2019t judge risk in hindsight<\/a><\/div>\n
\n

If you\u2019ve ever created an action item from an incident along the lines of \u201cdon\u2019t take unnecessary risks in the future\u201d, you need to read this one.<\/p>\n

The rest of you need to read it too.<\/p>\n

\u00a0\u00a0Lorin Hochstein<\/small><\/p>\n<\/div>\n<\/div>\n

\n
Anomaly Alerting in Prometheus<\/a><\/div>\n
\n

A how-to for building anomaly detection alerting in Prometheus with specific config examples.<\/p>\n

\u00a0\u00a0Karl Stoney<\/small><\/p>\n<\/div>\n<\/div>\n

\n
r\/sre: I almost re-imaged servers that were LIVE \u2013 Caused Disruption!<\/a><\/div>\n
\n

A panicked engineer asks reddit\u2019s r\/sre<\/a> about an incident they caused: how could they have done better? Will they be fired? The comments are spot on, and this conversation is fresh enough that you could jump in too if you\u2019re interested.<\/p>\n

\u00a0\u00a0u\/console_fulcrum and others \u2014 reddit<\/small><\/p>\n<\/div>\n<\/div>\n

\n
[Honeycomb incident followup]: US Production site is down<\/a><\/div>\n
\n

Last Monday, Honeycomb had an outaged related to a schema migration involving MySQL\u2019s ENUM data type, and they posted this incident report.<\/p>\n

Bonus content: I wasn\u2019t aware of ENUMs at all<\/em>, so I had to brush up with this article: 8 Reasons Why MySQL\u2019s ENUM Data Type Is Evil<\/a>.<\/p>\n

\u00a0\u00a0Honeycomb<\/small><\/p>\n

\u00a0\u00a0Full disclosure: Honeycomb is my employer.<\/em><\/small><\/p>\n<\/div>\n<\/div>\n

\n
Cracking the SRE Interview<\/a><\/div>\n
\n

An experienced SRE discusses the skills and experiences you might be quizzed about in an interview for an SRE role.<\/p>\n

\u00a0\u00a0Krishna Vinnakota \u2014 DZone<\/small><\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

SRE WEEKLY<\/p>","protected":false},"excerpt":{"rendered":"

View on sreweekly.com A message from our sponsor, FireHydrant: We\u2019ve gone all out on our new integration with Microsoft Teams. If you\u2019re a MS Teams user, FireHydrant now supports the most comprehensive integration for incident management. Run the entire IM process without ever leaving the chat. https:\/\/firehydrant.com\/blog\/introducing-a-brand-new-microsoft-teams-integration\/ The Reverse Red Herring This article presents in… Continue reading SRE Weekly Issue #428<\/span><\/a><\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":""},"categories":[8],"tags":[],"class_list":["post-874","post","type-post","status-publish","format-standard","hentry","category-sre","entry"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":872,"url":"https:\/\/fde.cat\/index.php\/2024\/06\/03\/sre-weekly-issue-427\/","url_meta":{"origin":874,"position":0},"title":"SRE Weekly Issue #427","date":"June 3, 2024","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, FireHydrant: We\u2019ve gone all out on our new integration with Microsoft Teams. If you\u2019re a MS Teams user, FireHydrant now supports the most comprehensive integration for incident management. Run the entire IM process without ever leaving the chat. https:\/\/firehydrant.com\/blog\/introducing-a-brand-new-microsoft-teams-integration\/ Why didn\u2019t you\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":882,"url":"https:\/\/fde.cat\/index.php\/2024\/06\/17\/sre-weekly-issue-429\/","url_meta":{"origin":874,"position":1},"title":"SRE Weekly Issue #429","date":"June 17, 2024","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, FireHydrant: We\u2019ve gone all out on our new integration with Microsoft Teams. If you\u2019re a MS Teams user, FireHydrant now supports the most comprehensive integration for incident management. Run the entire IM process without ever leaving the chat. https:\/\/firehydrant.com\/blog\/introducing-a-brand-new-microsoft-teams-integration\/ Virtualizing Our Storage\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":885,"url":"https:\/\/fde.cat\/index.php\/2024\/06\/24\/sre-weekly-issue-430\/","url_meta":{"origin":874,"position":2},"title":"SRE Weekly Issue #430","date":"June 24, 2024","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, FireHydrant: We\u2019ve gone all out on our new integration with Microsoft Teams. If you\u2019re a MS Teams user, FireHydrant now supports the most comprehensive integration for incident management. Run the entire IM process without ever leaving the chat. https:\/\/firehydrant.com\/blog\/introducing-a-brand-new-microsoft-teams-integration\/ r\/sre: Senior SRE\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":899,"url":"https:\/\/fde.cat\/index.php\/2024\/07\/22\/sre-weekly-issue-434\/","url_meta":{"origin":874,"position":3},"title":"SRE Weekly Issue #434","date":"July 22, 2024","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, FireHydrant: We\u2019ve gone all out on our new integration with Microsoft Teams. If you\u2019re a MS Teams user, FireHydrant now supports the most comprehensive integration for incident management. Run the entire IM process without ever leaving the chat. https:\/\/firehydrant.com\/blog\/introducing-a-brand-new-microsoft-teams-integration\/ Technical Details: Falcon\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":889,"url":"https:\/\/fde.cat\/index.php\/2024\/07\/01\/sre-weekly-issue-431\/","url_meta":{"origin":874,"position":4},"title":"SRE Weekly Issue #431","date":"July 1, 2024","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, FireHydrant: We\u2019ve gone all out on our new integration with Microsoft Teams. If you\u2019re a MS Teams user, FireHydrant now supports the most comprehensive integration for incident management. Run the entire IM process without ever leaving the chat. https:\/\/firehydrant.com\/blog\/introducing-a-brand-new-microsoft-teams-integration\/ Cloudflare incident on\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":891,"url":"https:\/\/fde.cat\/index.php\/2024\/07\/08\/sre-weekly-issue-432\/","url_meta":{"origin":874,"position":5},"title":"SRE Weekly Issue #432","date":"July 8, 2024","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, FireHydrant: We\u2019ve gone all out on our new integration with Microsoft Teams. If you\u2019re a MS Teams user, FireHydrant now supports the most comprehensive integration for incident management. Run the entire IM process without ever leaving the chat. https:\/\/firehydrant.com\/blog\/introducing-a-brand-new-microsoft-teams-integration\/ Investigating Mysterious Kafka\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/874","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/comments?post=874"}],"version-history":[{"count":0,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/874\/revisions"}],"wp:attachment":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/media?parent=874"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/categories?post=874"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/tags?post=874"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}