{"id":593,"date":"2022-03-29T20:53:00","date_gmt":"2022-03-29T20:53:00","guid":{"rendered":"https:\/\/fde.cat\/index.php\/2022\/03\/29\/investigate-issues-with-ease-by-adding-a-correlation-id-to-your-api-2\/"},"modified":"2022-03-29T20:53:00","modified_gmt":"2022-03-29T20:53:00","slug":"investigate-issues-with-ease-by-adding-a-correlation-id-to-your-api-2","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2022\/03\/29\/investigate-issues-with-ease-by-adding-a-correlation-id-to-your-api-2\/","title":{"rendered":"Investigate Issues with Ease by Adding a Correlation ID to your API"},"content":{"rendered":"

With APIs becoming more complex and distributed, developers sometimes struggle to find the relevant logs when they need to investigate a specific issue. In the new\u00a0Salesforce Commerce APIs<\/a>\u00a0(SCAPI), we created such an architecture of distributed systems and recognized this problem early. Our approach to mitigate it was the introduction of a correlation ID.<\/p>\n

This ID is a unique identifier of the API request across systems and is commonly used for these scenarios. It has proven to be very helpful for internal developers as well as external users.<\/p>\n

This blog post describes how a correlation ID can be leveraged to easily find logs for further investigation.<\/p>\n

A quick overview of the SCAPI ecosystem<\/h1>\n

In order to understand the necessity of a correlation ID, let\u2019s have a look at the architecture in place.<\/p>\n

Diagram showing the architecture of the Salesforce Commerce API ecosystem<\/p>\n

Every HTTP request to these new REST APIs initially comes in through the CDN layer. Among other things, this layer is responsible for routing the request to the correct instance of the API layer.The API layer (e.g. Mulesoft) acts as an API middleware or gateway with applications handling the API requests. During processing, a request is made to the Commerce Cloud backend.The Commerce Cloud backend consists of services that generate and return a response.<\/p>\n

Each component has access to the monitoring layer where the log events are aggregated.<\/p>\n

The benefit of a correlation ID<\/h1>\n

When looking at the architecture diagram above, we can spot a few issues that a developer might face. The applications on the API layer and the Commerce Cloud backend both process the request, which involves writing a number of events into their respective log files. Since this processing can result in errors in each of the systems, it is often necessary to look at the different files in order to understand where an issue is located.<\/p>\n

This is why the CDN layer contains a module that injects the correlation ID into the request. Since every request passes through this layer, this is the perfect place to handle this task. Although the CDN system already provides a header to identify requests, it is not guaranteed to be unique for this purpose. As a consequence, the module generates a new UUID and sends it with the HTTP request header x-correlation-id.<\/p>\n

For the same requirement of uniqueness, the correlation ID cannot be set by clients. Any existing value in the header x-correlation-id will be overwritten with the newly generated ID. However, clients are of course free to use their own identifiers for internal purposes by sending them in a different custom header.<\/p>\n

After the injection of the correlation ID, the request is routed to the API layer. The name x-correlation-id is beneficial here. It is commonly used in descriptions of correlation IDs and is, therefore, a de-facto industry standard. By following this convention, the Mulesoft API layer automatically uses the value from this header for its log events without requiring any changes to the deployed applications.<\/p>\n

Eventually, the request is sent to the Commerce Cloud backend, which writes the identifier into its log files as well. When the response is generated, the correlation ID is returned to the client as an HTTP response header.<\/p>\n

Different use cases for the same tool<\/h1>\n

The correlation ID has considerable benefits for both internal and external use cases.<\/p>\n

Internally, developers may need to track a request chain in order to identify the nature of a specific issue. Without the same request ID, correlating the individual transactions or events must be done with other properties like timestamps, client identifiers, or a combination of those. However, due to processing time, timestamps might slightly differ across systems. Also, different timezones could be configured and shown in the log events. This makes such investigations lengthy and cumbersome, especially when the request volume is very high.<\/p>\n

Externally, clients might notice irregularities in their requests and ask Commerce Cloud Support for help and advice. Without supplying a correlation ID in the support case, the support team would have to take the same difficult steps described above. Instead, they can now use the correlation ID and query their monitoring systems for all relevant logs. This tremendously accelerates the issue resolution, hence increasing client satisfaction.<\/p>\n

How does it work?<\/h1>\n

Let\u2019s have a look at an example. We try to update a product via the Products API and get a response with the 500 Internal Server Error status code. This is a server-side error that should not expose implementation details like a stack trace in the response body. We need to inspect the logs to find the cause of it.<\/p>\n

The header section that is returned with the response will look like this:<\/p>\n

HTTP\/2 500
date: Mon, 29 Nov 2021 11:47:05 GMT
content-type: application\/json; charset=UTF-8
content-length: 177
strict-transport-security: max-age=31536000; includeSubdomains;
x-correlation-id: 002df95f-3e2b-4553-97ac-bc241a26d8ca<\/p>\n

It contains the correlation ID under x-correlation-id. This value can now be used to find the related log events.<\/p>\n

For instance, the logs from the applications on the API layer and from the Commerce Cloud backend are pushed to the monitoring layer of Salesforce. Because the correlation ID is used in different fields across systems, a query needs to combine those with the value from the request:<\/p>\n

index=commercecloud-secure “message.activity.external”=002df95f-3e2b-4553-97ac-bc241a26d8ca OR “@metadata.correlationId”=002df95f-3e2b-4553-97ac-bc241a26d8ca<\/p>\n

Running this query will show results from all involved systems:<\/p>\n

Screenshot of results from a correlation ID query<\/p>\n

The field sourcetype shows two entries. The value cc-application:umon refers to the event on the Commerce Cloud backend, whereas cc.mercury:applogs is the event on the API layer.<\/p>\n

Screenshot of a detailed view of the sourcetype field in the query results<\/p>\n

Depending on the nature of the error, there could be more than two events. Opening an event will show the details to identify the cause:<\/p>\n

Detailed screenshot of event logs<\/p>\n

Conclusion<\/h1>\n

As shown, the benefits of using a correlation ID are substantial. Keep in mind that the higher the number of connected systems is, the greater the value of such an identifier is. We encourage every reader to use the correlation ID as a tool in their own architecture. The effort of implementing it in new or existing systems is often justified by the help it provides.<\/p>\n

The post Investigate Issues with Ease by Adding a Correlation ID to your API<\/a> appeared first on Salesforce Engineering Blog<\/a>.<\/p>\n

Read More<\/a><\/p>","protected":false},"excerpt":{"rendered":"

With APIs becoming more complex and distributed, developers sometimes struggle to find the relevant logs when they need to investigate a specific issue. In the new\u00a0Salesforce Commerce APIs\u00a0(SCAPI), we created such an architecture of distributed systems and recognized this problem early. Our approach to mitigate it was the introduction of a correlation ID. This ID… Continue reading Investigate Issues with Ease by Adding a Correlation ID to your API<\/span><\/a><\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":""},"categories":[7],"tags":[],"class_list":["post-593","post","type-post","status-publish","format-standard","hentry","category-technology","entry"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":558,"url":"https:\/\/fde.cat\/index.php\/2022\/03\/29\/investigate-issues-with-ease-by-adding-a-correlation-id-to-your-api\/","url_meta":{"origin":593,"position":0},"title":"Investigate Issues with Ease by Adding a Correlation ID to your API","date":"March 29, 2022","format":false,"excerpt":"With APIs becoming more complex and distributed, developers sometimes struggle to find the relevant logs when they need to investigate a specific issue. In the new Salesforce Commerce APIs (SCAPI), we created such an architecture of distributed systems and recognized this problem early. Our approach to mitigate it was the\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":839,"url":"https:\/\/fde.cat\/index.php\/2024\/03\/18\/logarithm-a-logging-engine-for-ai-training-workflows-and-services\/","url_meta":{"origin":593,"position":1},"title":"Logarithm: A logging engine for AI training workflows and services","date":"March 18, 2024","format":false,"excerpt":"Systems and application logs play a key role in operations, observability, and debugging workflows at Meta. Logarithm is a hosted, serverless, multitenant service, used only internally at Meta, that consumes and indexes these logs and provides an interactive query interface to retrieve and view logs. In this post, we present\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":726,"url":"https:\/\/fde.cat\/index.php\/2023\/06\/21\/tackling-cyber-threats-with-automation-inside-salesforces-cutting-edge-security-strategy\/","url_meta":{"origin":593,"position":2},"title":"Tackling Cyber Threats with Automation: Inside Salesforce\u2019s Cutting-edge Security Strategy","date":"June 21, 2023","format":false,"excerpt":"In our \u201cEngineering Energizers\u201d Q&A series, we examine the professional journeys that have shaped Salesforce Engineering leaders. In this special edition, we meet Avinash Reddy Thumma, lead threat detection engineer for Salesforce. Based in Hyderabad, India, Avinash\u2019s threat detection team specializes in protecting Salesforce\u2019s network by thwarting malicious threats. Read\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":227,"url":"https:\/\/fde.cat\/index.php\/2021\/02\/02\/heroku-ci-and-github-checks-integration\/","url_meta":{"origin":593,"position":3},"title":"Heroku CI and Github Checks Integration","date":"February 2, 2021","format":false,"excerpt":"Heroku CI and GitHub Checks IntegrationBefore we dive into the crux of this article, let\u2019s first get an understanding of what GitHub Checks is and how it will be useful for you when you use any external Continuous Integration (CI) tool like CircleCI, Heroku CI, or any local tool. The\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":221,"url":"https:\/\/fde.cat\/index.php\/2021\/02\/02\/asyncapi-and-openapi-an-api-modeling-approach\/","url_meta":{"origin":593,"position":4},"title":"AsyncAPI and OpenAPI: an API Modeling Approach","date":"February 2, 2021","format":false,"excerpt":"AsyncAPI is gaining traction in the ecosystem of API tools. It solves an important problem: it provides a convenient way of describing the interface of event-driven systems independently of the underlying technology. With AsyncAPI, evented systems can be treated as any other API product: a productizable and reusable, self-describing building\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":499,"url":"https:\/\/fde.cat\/index.php\/2021\/11\/01\/the-journey-of-building-a-scalable-api\/","url_meta":{"origin":593,"position":5},"title":"The Journey of Building a Scalable API","date":"November 1, 2021","format":false,"excerpt":"APIs are an essential tool to allow partners, developers, and applications to consume, communicate\u00a0,or build on top of the various capabilities your microservices provide. Building a high quality API that can scale and perform with the business ecosystem is not easy and requires putting thought and planning into everything, from\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/593","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/comments?post=593"}],"version-history":[{"count":0,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/593\/revisions"}],"wp:attachment":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/media?parent=593"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/categories?post=593"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/tags?post=593"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}