How can you make your services observable and embrace service ownership? This article presents a variety of universally applicable design patterns for the developer to consider.<\/p>\n
Design patterns in software development are repeatable solutions and best practices for solving commonly occurring problems. Even in the case of service monitoring, design patterns, when used appropriately, can help teams embrace service ownership and troubleshoot their services in production. You can think of service monitoring design patterns in three categories:<\/p>\n
Health Checks Let\u2019s look at the patterns (and a few anti-patterns!) for each of these categories in\u00a0turn.<\/p>\n The two patterns for health checks are the outside-in health check<\/strong>, which verifies that your service is running and determines response time \/ latency from your service, and the inside-out health check<\/strong>, which keeps tabs on app and system metrics so that you can detect potential problems (including performance problems) before they cause an incident.<\/p>\n In this pattern, you ping your service endpoint using using a health-check service or a synthetic testing tool. We use a tool we built in-house, but other options include NewRelic, Gomez, and DataDog. Your service responds to the ping, which records the output of the check as a metric in a time-series metric system such as Argus<\/a> or whatever you\u2019re using for such a service. Once the data is available, you can visualize your service\u2019s health and other key metrics over a period of time and\/or opt for it to send you alerts when specific conditions are\u00a0met.<\/p>\n The high-level pattern looks like\u00a0this.<\/p>\n Diagram outlining the high-level pattern for outside-in health\u00a0checks.<\/p>\n This pattern can be used to check two main groups of\u00a0metrics:<\/p>\n Uptime<\/strong>\u200a\u2014\u200aThis metric answers the time-honored question, \u201cis my service running, and is it doing what it is supposed to do?\u201d After the health-check invoker pings your service, if your service responds, it is up; if it doesn\u2019t respond, it\u2019s down, and you need to start remediation effortsUser-perceived latency<\/strong>\u200a\u2014\u200aYou must check service latency from multiple locations around the globe. User-perceived latency might be different for a customer accessing your service from Japan, for example, than it is for customers accessing it from Spain and the United States. You can use the latency of the health-check API call as a proxy for user-perceived latency.<\/p>\n The combination of the two groups above can be used to determine a synthetic-based availability signal for your service as\u00a0well.<\/p>\n In this pattern, you use system and application metrics to detect potential issues before they result in service interruptions\u200a\u2014\u200afor your service and your customers!<\/p>\n To determine whether there are underlying issues that might affect your service\u2019s performance or availability, collect metrics on application and the underlying infrastructure on which it is\u00a0running.<\/p>\n Diagram outlining the high-level pattern of inside-out health\u00a0checks.<\/p>\n So which application metrics should you be concerned with? At a minimum, collect these four\u00a0signals.<\/p>\n Request rate<\/strong>\u200a\u2014\u200aHow busy is your\u00a0service?Error rate<\/strong>\u200a\u2014\u200aAre there any errors in your service? If there are, how many, and how often do they\u00a0occur?Duration (of requests)<\/strong>\u200a\u2014\u200aHow long is it taking for your service to respond to requests?Saturation<\/strong>\u200a\u2014\u200aHow overloaded is your service? How much room does it have to\u00a0grow?<\/p>\n In addition, gather these system metrics for each resource type (e.g. CPU, memory, disk space, IOPS) that your service\u00a0uses.<\/p>\n Utilization<\/strong>\u200a\u2014\u200aHow busy is this resource?Saturation<\/strong>\u200a\u2014\u200aHow utilized is the most constrained resource of the application which is impacting the application\u2019s performance \/ ability to function normally?Errors<\/strong>\u200a\u2014\u200aWhat is the resource\u2019s count of\u00a0errors?<\/p>\n Collecting these metrics allows you to also compute the availability metric of the service, which answers the critical<\/em> question\u200a\u2014\u200aIs my service or feature considered available to my customers? In most cases a combination of Uptime, Duration (latency) and Error Rate metrics can be used to compute the availability of the service in a manner that is representative of the customer\u2019s experience with the service. E.g. if you compute the availability of the service solely based upon uptime, then you will still consider yourself available even if queries take a long time to respond \/ web pages take a long time to load. Hence, taking additional metrics such as Error Rates and Duration are critical for an appropriate definition of Availability.<\/p>\n Log data is great for troubleshooting, which we\u2019ll talk about soon. However, given the volume of log data that most applications generate these days, you really don\u2019t want to emit metrics using log data. Modeling metrics using log data is computationally expensive (which means expensive in monetary terms) and increases time to ingest, process and react which increases MTTD (mean time to detect). Neither of these outcomes are desirable. At Salesforce, we have teams that have built out efficient metrics-gathering tools for our developers to\u00a0use.<\/p>\n Alerts and notifications are the primary ways in which you as a service owner would want to get notified when something is wrong with your service. You should configure alerts when key health and performance metrics of your service and infrastructure go above or below a specified threshold which would indicate an issue. E.g. when response time of the service suddenly increases beyond an acceptable threshold, when the availability of your service goes below the specified threshold such that it breaches the SLA of the\u00a0service.<\/p>\n The goal of alerting is to notify a human or a remediation process to fix the issue and bring the service back to a healthy status so that it can continue to serve its customers. Although auto-remediation of an issue by kicking in an automated workflow upon getting alerted is desirable, not all issues (specially complex service issues) can be addressed as such. Typical scenarios where auto-remediation can be helpful include restarting the service to fix an issue, increase \/ decrease compute capacity in cloud environments, terminate an instance if there are any unauthorized ports open (security vulnerability).<\/p>\n By using more sophisticated techniques (including machine learning), you can make the experience better for the service owners by alerting proactively and detecting anomalies. Proactive alerting allows you to take the remedial action before the issue impacts service health and the customers. E.g. it is better to get alerted that based upon current usage trends you will run out of disk capacity in a week as opposed to getting alerted when you are 90% disk full! Proactive alerting provides for more lead time to fix the issue and the alert need not result in a page at 3am in the morning to fix the issue. It can be a ticket that can be logged for the service owner to look at the issue the following morning. Detecting anomalies reduces alert noise and saves service owners from configuring static thresholds on metrics to get alerted on. The system will alert you when it detects an anomaly in the pattern for a given metrics. It will take seasonality into account and thus will reduce false positive alerts (noise) and automatically adjust the thresholds over a period of time and its definition of what constitutes an\u00a0anomaly.<\/p>\n The general pattern for alerting involves determining whether a problem can be auto-remediated, or whether some action needs to be taken by a human to avoid breaching your company\u2019s Service Level Agreement (SLA) with your customers. This process flow diagram incorporates both auto-remediation and remediation requiring human intervention.<\/p>\n Flow diagram showing the decision process for determining whether to initiate auto remediation or invoke human\u00a0action.<\/p>\n Don\u2019t do glass watching. <\/strong>As systems scale and complexity increases, we cannot rely on humans to stare at monitors 24\/7 to look for service health trends and call someone to fix an issue if the threshold is breached. We need to rely on machines and algorithms notifying us when something goes wrong. This also takes human errors out of the equation. Automate as much of the process as possible.All alerts are not created equal. <\/strong>As such, don\u2019t treat all alerts the same way. Sending every small issue with the service as an alert to email\/Slack channel results in nothing more than spam and greatly reduces the signal-to-noise ratio. For every service issue, that may potentially result in a customer issue \/ SLA breach (critical issue), page the on-call engineer (using something like PagerDuty). Everything else should be logged as a ticket (if action is needed) or should be logged as a log\u00a0entry.<\/p>\n When something does go wrong with your service, you need information on what went wrong, when it went wrong, and where it went wrong. Code your service so that this kind of information is available to troubleshoot issues. There are two key ways to make that information available.<\/p>\n Logging error conditions and related informationEnabling distributed tracing in your service (especially in microservice environments)<\/p>\n Logs are your best friends for finding out what went wrong with your service and when, so be sure to log your service\u2019s error conditions. Include a logging library in your service to capture app logs and send those logs to a logging service. At Salesforce, we use Splunk<\/a>, but some other options might be DataDog, NewRelic, etc.<\/p>\n When you troubleshoot, you can use both app and infrastructure logs to help you determine what caused an incident and how to reduce the chance of it recurring.<\/p>\n The following diagram captures the high-level architecture of a logging\u00a0system.<\/p>\n Diagram showing the high-level architecture of a logging\u00a0system.<\/p>\n In a microservices architecture, when an incident or performance degradation occurs, you need to know more than just what went wrong. You also need to know which of your microservices caused or contributed to the\u00a0issue.<\/p>\n Distributed tracing<\/strong> allows you to get that info by identifying each request with a requestID. When an issue occurs, you can pinpoint where it occurred in the request stream and whether the issue was associated with your service or with a dependent service. At Salesforce, we\u2019ve integrated all of our various applications with homegrown distributed tracing service (build on top of Zipkin), Tracer. Generated spans are sent to Tracer, and context is propagated to the downstream applications using B3 headers<\/a>. For instrumentation, we use either Zipkin<\/a> or OpenTelemetry<\/a> libraries.<\/p>\n By applying design patterns around health checks, alerting, and troubleshooting, you can build observable services from the ground up. There are many tools and additional resources out there to help you get started. One that we highly recommend is Google\u2019s Site Reliability Engineering (SRE)<\/em> book<\/a>, particularly the \u201cMonitoring\u201d<\/strong> section in the Introduction<\/em><\/a> and Chapter 6: Monitoring Distributed Systems<\/em><\/a>. Also check out our Observability 101 series<\/a> for even more on monitoring microservices.<\/p>\n 5 Design Patterns for Building Observable Services<\/a> was originally published in Salesforce Engineering<\/a> on Medium, where people are continuing the conversation by highlighting and responding to this story.<\/p>\n Read More<\/a><\/p>","protected":false},"excerpt":{"rendered":"
<\/strong>How do you know that your service is running\u200a\u2014\u200aand, if it is\u200a\u2014\u200aalso doing what it\u2019s supposed to be doing? Is it responding in a timely manner? Are there potential service issues that you can address before they affect customers?Real time Alerting
<\/strong>When something does go wrong\u200a\u2014\u200asuch as your service becoming unresponsive, slowing to a crawl, or using too many resources\u200a\u2014\u200ado you have alerts that you configured to notify you of that\u00a0issue?Troubleshooting
<\/strong>Has something gone wrong with your service? If it has, you probably need to know three things: when it happened, where it happened, and what caused it to happen. Use logs and traces to diagnose issues after they occur, and update your service so that those issues do not have lasting customer\u00a0impacts.<\/p>\nHealth Checks<\/h3>\n
1. The Outside-In Health\u00a0Check<\/h3>\n
2. The Inside-Out Health\u00a0Check<\/h3>\n
An Anti-Pattern: Modeling Metrics Using\u00a0Logs<\/h3>\n
Real time\u00a0Alerting<\/h3>\n
3. Remediating Issues<\/h3>\n
Alerting Anti-Patterns<\/h3>\n
Troubleshooting<\/h3>\n
4. Logging Error Conditions<\/h3>\n
5. Distributed Tracing for Microservices<\/h3>\n
So now\u00a0what?<\/h3>\n