{"id":188,"date":"2021-02-02T20:01:07","date_gmt":"2021-02-02T20:01:07","guid":{"rendered":"https:\/\/fde.cat\/?p=188"},"modified":"2021-02-02T20:01:07","modified_gmt":"2021-02-02T20:01:07","slug":"pegasus-data-language-evolving-schema-definitions-for-data-modeling","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2021\/02\/02\/pegasus-data-language-evolving-schema-definitions-for-data-modeling\/","title":{"rendered":"Pegasus Data Language: Evolving schema definitions for data modeling"},"content":{"rendered":"<div class=\"resourceParagraph section\">\n<div class=\"component-anchor-container\">\n  <a class=\"component-anchor\" name=\"post_par_resourceparagraph_859137906\"><\/a>\n <\/div>\n<div class=\"resource-text-section\">\n<div class=\"resource-paragraph rich-text\">\n<p>Pegasus Data Schema (<a href=\"https:\/\/linkedin.github.io\/rest.li\/pdsc_syntax\" target=\"_blank\" rel=\"noopener\">PDSC<\/a>) is a Pegasus schema definition language that has been used for data modeling with <a href=\"https:\/\/linkedin.github.io\/rest.li\/\" target=\"_blank\" rel=\"noopener\">Rest.li<\/a> services for years. It&#8217;s the underlying language that helps define data models, describe the data returned by REST endpoints, and generate derivative schemas for other uses, such as XML schemas and various database schemas. However, writing PDSC files is hard and often error-prone because they can be verbose and lack integrated developer environment (IDE) support.\u00a0<\/p>\n<p>With this in mind, we are excited to announce the general availability of Pegasus Data Language (<a href=\"https:\/\/linkedin.github.io\/rest.li\/pdl_schema\" target=\"_blank\" rel=\"noopener\">PDL<\/a>) as a new Pegasus schema definition language to replace PDSC. In open sourcing PDL, we hope to help Rest.li developers bring in a new standard for defining schemas in a more user-friendly format. In this blog, we discuss the differences between PDL and PDSC, the process of migrating existing PDSC to PDL, and future plans for PDL.<\/p>\n<h2>Why Pegasus Data Language?<\/h2>\n<p>As the successor to PDSC, PDL was developed to address user experience issues with PDSC and decrease the number of errors while writing schemas. PDL boasts a multitude of improvements, including a Java-like syntax that makes it more readable and rich editor support bundled with LinkedIn\u2019s IntelliJ plugin.\u00a0<\/p>\n<h2>Core features<\/h2>\n<p>Although PDSC has a syntax that is a subset of <a href=\"https:\/\/www.json.org\/json-en.html\" target=\"_blank\" rel=\"noopener\">JSON<\/a>, it lacks in readability. Therefore, in developing PDL, we knew a priority would be to design it to be easier to read. PDL also features extra shorthand that developers can leverage to write less lines of code and make their schemas easier to understand.<\/p>\n<p><b>Java-like syntax<br \/> <\/b>PDL and PDSC are fully compatible. Developers will write less for the same schema than they would if they used PDL instead of PDSC.<\/p>\n<\/p><\/div>\n<\/p><\/div>\n<\/div>\n<div class=\"resourceImageBlock section\">\n<div class=\"component-anchor-container\">\n  <a class=\"component-anchor\" name=\"post_par_resourceimageblock_966329898\"><\/a>\n <\/div>\n<ul class=\"resource-image-block single\">\n<li class=\"resource-image\"> <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/i0.wp.com\/content.linkedin.com\/content\/dam\/engineering\/site-assets\/images\/blog\/posts\/2020\/11\/pegasus-1.png?resize=550%2C434&#038;ssl=1\" alt=\"comparing-p-d-s-c-and-p-d-l-by-schemas-written\" height=\"434\" width=\"550\"  data-recalc-dims=\"1\"> <\/li>\n<\/ul>\n<\/div>\n<div class=\"resourceParagraph section\">\n<div class=\"component-anchor-container\">\n  <a class=\"component-anchor\" name=\"post_par_resourceparagraph_1365608544\"><\/a>\n <\/div>\n<div class=\"resource-text-section\">\n<div class=\"resource-paragraph rich-text\">\n<p><i>Side-by-side comparison of PDSC and PDL for the same schema<\/i><\/p>\n<p><b>Support for import statements<br \/> <\/b>In PDSC, all references to types outside of the schema\u2019s own namespace must be written as fully-qualified type names. In PDL, however, feature-import statements allow the user to specify types that can be referenced by their simple name rather than their full name. This helps to reduce the amount of redundant data written in schemas that refer to the same type numerous times.<\/p>\n<\/p><\/div>\n<\/p><\/div>\n<\/div>\n<div class=\"resourceImageBlock section\">\n<div class=\"component-anchor-container\">\n  <a class=\"component-anchor\" name=\"post_par_resourceimageblock_382806354\"><\/a>\n <\/div>\n<ul class=\"resource-image-block single\">\n<li class=\"resource-image\"> <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/i0.wp.com\/content.linkedin.com\/content\/dam\/engineering\/site-assets\/images\/blog\/posts\/2020\/11\/pegasus-2.png?resize=550%2C488&#038;ssl=1\" alt=\"showing-import-statements-in-p-d-l\" height=\"488\" width=\"550\"  data-recalc-dims=\"1\"> <\/li>\n<\/ul>\n<\/div>\n<div class=\"resourceParagraph section\">\n<div class=\"component-anchor-container\">\n  <a class=\"component-anchor\" name=\"post_par_resourceparagraph_1650891992\"><\/a>\n <\/div>\n<div class=\"resource-text-section\">\n<div class=\"resource-paragraph rich-text\">\n<p><i>Side-by-side comparison of PDSC and PDL with the latter using import statements<\/i><\/p>\n<p><b>Shorthand for custom properties<br \/> <\/b>In PDSC, the custom properties are arbitrary values keyed at anything that\u2019s not a reserved keyword. In PDL, the custom properties are cleaner and carry a more Java-like syntax.<\/p>\n<\/p><\/div>\n<\/p><\/div>\n<\/div>\n<div class=\"resourceImageBlock section\">\n<div class=\"component-anchor-container\">\n  <a class=\"component-anchor\" name=\"post_par_resourceimageblock_1753736184\"><\/a>\n <\/div>\n<ul class=\"resource-image-block single\">\n<li class=\"resource-image\"> <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/i0.wp.com\/content.linkedin.com\/content\/dam\/engineering\/site-assets\/images\/blog\/posts\/2020\/11\/pegasus-3.png?resize=550%2C146&#038;ssl=1\" alt=\"showing-custom-properties-in-p-d-l\" height=\"146\" width=\"550\"  data-recalc-dims=\"1\"> <\/li>\n<\/ul>\n<\/div>\n<div class=\"resourceParagraph section\">\n<div class=\"component-anchor-container\">\n  <a class=\"component-anchor\" name=\"post_par_resourceparagraph_998014595\"><\/a>\n <\/div>\n<div class=\"resource-text-section\">\n<div class=\"resource-paragraph rich-text\">\n<p><b>Cleaner enum declarations<br \/> <\/b>In PDSC, the metadata of enum must be specified in individual mappings that are separate from the main symbol list. In this way, defining complex enums in PDSC is unintuitive and can be hard to read and maintain. In contrast,defining enum symbol metadata in PDL is quite intuitive. Each doc string, custom property, and deprecation annotation can be placed right alongside the symbol.<\/p>\n<\/p><\/div>\n<\/p><\/div>\n<\/div>\n<div class=\"resourceImageBlock section\">\n<div class=\"component-anchor-container\">\n  <a class=\"component-anchor\" name=\"post_par_resourceimageblock_348355934\"><\/a>\n <\/div>\n<ul class=\"resource-image-block single\">\n<li class=\"resource-image\"> <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/i0.wp.com\/content.linkedin.com\/content\/dam\/engineering\/site-assets\/images\/blog\/posts\/2020\/11\/pegasus-4.png?resize=550%2C488&#038;ssl=1\" alt=\"showing-enum-declarations-in-p-d-l\" height=\"488\" width=\"550\"  data-recalc-dims=\"1\"> <\/li>\n<\/ul>\n<\/div>\n<div class=\"resourceParagraph section\">\n<div class=\"component-anchor-container\">\n  <a class=\"component-anchor\" name=\"post_par_resourceparagraph_1674375625\"><\/a>\n <\/div>\n<div class=\"resource-text-section\">\n<div class=\"resource-paragraph rich-text\">\n<p><i>Side-by-side comparison of PDSC and PDL with enum declarations<\/i><\/p>\n<h2>Benefits of PDL<\/h2>\n<p><b>Rich editor (IntelliJ) support<br \/> <\/b>For PDSC, we have had limited IDE support. The experience of writing PDSC is similar to writing JSON text in IDE.<\/p>\n<\/p><\/div>\n<\/p><\/div>\n<\/div>\n<div class=\"resourceImageBlock section\">\n<div class=\"component-anchor-container\">\n  <a class=\"component-anchor\" name=\"post_par_resourceimageblock_1883915305\"><\/a>\n <\/div>\n<ul class=\"resource-image-block single\">\n<li class=\"resource-image\"> <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/i0.wp.com\/content.linkedin.com\/content\/dam\/engineering\/site-assets\/images\/blog\/posts\/2020\/11\/pegasus-5.png?resize=400%2C468&#038;ssl=1\" alt=\"screenshot-showing-the-limited-i-d-e-support-in-p-d-s-c\" height=\"468\" width=\"400\"  data-recalc-dims=\"1\"> <\/li>\n<\/ul>\n<\/div>\n<div class=\"resourceParagraph section\">\n<div class=\"component-anchor-container\">\n  <a class=\"component-anchor\" name=\"post_par_resourceparagraph_380998654\"><\/a>\n <\/div>\n<div class=\"resource-text-section\">\n<div class=\"resource-paragraph rich-text\">\n<p>With rich IDE (IntelliJ) support, PDL makes writing schema much easier and reduces the likelihood for human error. The IntelliJ PDL plugin features include, but are not limited to:<\/p>\n<p><b>1. Syntax highlighting, including warnings for errors and deprecation.<br \/> <\/b>Syntax highlighting makes the Pegasus schema more readable.<\/p>\n<\/p><\/div>\n<\/p><\/div>\n<\/div>\n<div class=\"resourceImageBlock section\">\n<div class=\"component-anchor-container\">\n  <a class=\"component-anchor\" name=\"post_par_resourceimageblock_880776645\"><\/a>\n <\/div>\n<ul class=\"resource-image-block single\">\n<li class=\"resource-image\"> <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/i0.wp.com\/content.linkedin.com\/content\/dam\/engineering\/site-assets\/images\/blog\/posts\/2020\/11\/pegasus-6.png?resize=400%2C263&#038;ssl=1\" alt=\"intelli-j-screenshot-showing-syntax-highlighting\" height=\"263\" width=\"400\"  data-recalc-dims=\"1\"> <\/li>\n<\/ul>\n<\/div>\n<div class=\"resourceParagraph section\">\n<div class=\"component-anchor-container\">\n  <a class=\"component-anchor\" name=\"post_par_resourceparagraph_1600702142\"><\/a>\n <\/div>\n<div class=\"resource-text-section\">\n<div class=\"resource-paragraph rich-text\">\n<p>The error highlighting feature helps developers easily detect any mistakes while writing a schema rather than finding the errors later during build time.<\/p>\n<\/p><\/div>\n<\/p><\/div>\n<\/div>\n<div class=\"resourceImageBlock section\">\n<div class=\"component-anchor-container\">\n  <a class=\"component-anchor\" name=\"post_par_resourceimageblock_1221305294\"><\/a>\n <\/div>\n<ul class=\"resource-image-block single\">\n<li class=\"resource-image\"> <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/i0.wp.com\/content.linkedin.com\/content\/dam\/engineering\/site-assets\/images\/blog\/posts\/2020\/11\/pegasus-7.png?resize=400%2C142&#038;ssl=1\" alt=\"intelli-j-screenshot-showing-error-highlighting\" height=\"142\" width=\"400\"  data-recalc-dims=\"1\"> <\/li>\n<\/ul>\n<\/div>\n<div class=\"resourceParagraph section\">\n<div class=\"component-anchor-container\">\n  <a class=\"component-anchor\" name=\"post_par_resourceparagraph_1359524846\"><\/a>\n <\/div>\n<div class=\"resource-text-section\">\n<div class=\"resource-paragraph rich-text\">\n<p><b>2. Autocomplete suggestions for keywords and schema names<\/b><\/p>\n<p>Autocomplete helps developers easily find all the available keywords or schema names while writing a schema by prompting suggestions.<\/p>\n<\/p><\/div>\n<\/p><\/div>\n<\/div>\n<div class=\"resourceImageBlock section\">\n<div class=\"component-anchor-container\">\n  <a class=\"component-anchor\" name=\"post_par_resourceimageblock_1313823018\"><\/a>\n <\/div>\n<ul class=\"resource-image-block single\">\n<li class=\"resource-image\"> <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/i0.wp.com\/content.linkedin.com\/content\/dam\/engineering\/site-assets\/images\/blog\/posts\/2020\/11\/pegasus-8.png?resize=377%2C245&#038;ssl=1\" alt=\"intelli-j-screenshot-showing-auto-complete\" height=\"245\" width=\"377\"  data-recalc-dims=\"1\"> <\/li>\n<\/ul>\n<\/div>\n<div class=\"resourceParagraph section\">\n<div class=\"component-anchor-container\">\n  <a class=\"component-anchor\" name=\"post_par_resourceparagraph_243196699\"><\/a>\n <\/div>\n<div class=\"resource-text-section\">\n<div class=\"resource-paragraph rich-text\">\n<p><b>3. Auto-suggestions to fix common errors like file or schema name mismatch<\/b><\/p>\n<\/p><\/div>\n<\/p><\/div>\n<\/div>\n<div class=\"resourceImageBlock section\">\n<div class=\"component-anchor-container\">\n  <a class=\"component-anchor\" name=\"post_par_resourceimageblock_1866788672\"><\/a>\n <\/div>\n<ul class=\"resource-image-block single\">\n<li class=\"resource-image\"> <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/i0.wp.com\/content.linkedin.com\/content\/dam\/engineering\/site-assets\/images\/blog\/posts\/2020\/11\/pegasus-9.png?resize=300%2C236&#038;ssl=1\" alt=\"intelli-j-screenshot-showing-auto-suggestions\" height=\"236\" width=\"300\"  data-recalc-dims=\"1\"> <\/li>\n<\/ul>\n<\/div>\n<div class=\"resourceParagraph section\">\n<div class=\"component-anchor-container\">\n  <a class=\"component-anchor\" name=\"post_par_resourceparagraph_1421383768\"><\/a>\n <\/div>\n<div class=\"resource-text-section\">\n<div class=\"resource-paragraph rich-text\">\n<p>IDE support helps developers reduce possible human errors while writing Pegasus schemas and overall, improves the developer experience.<\/p>\n<p><b>4. Readable syntax and rich editor support for PDL to reduce build time errors.<\/b><\/p>\n<\/p><\/div>\n<\/p><\/div>\n<\/div>\n<div class=\"resourceImageBlock section\">\n<div class=\"component-anchor-container\">\n  <a class=\"component-anchor\" name=\"post_par_resourceimageblock_1706212806\"><\/a>\n <\/div>\n<ul class=\"resource-image-block single\">\n<li class=\"resource-image\"> <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/i0.wp.com\/content.linkedin.com\/content\/dam\/engineering\/site-assets\/images\/blog\/posts\/2020\/11\/pegasus-10.png?resize=700%2C166&#038;ssl=1\" alt=\"graph-showing-failure-counts-before-and-after-the-move-to-p-d-l\" height=\"166\" width=\"700\"  data-recalc-dims=\"1\"> <\/li>\n<\/ul>\n<\/div>\n<div class=\"resourceParagraph section\">\n<div class=\"component-anchor-container\">\n  <a class=\"component-anchor\" name=\"post_par_resourceparagraph_605455061\"><\/a>\n <\/div>\n<div class=\"resource-text-section\">\n<div class=\"resource-paragraph rich-text\">\n<p>As seen in the chart above, the move to PDL was significant in reducing GenerateDataTemplate failure counts. The orange line indicates the failure counts over 2019, while the blue line indicates the failure counts in 2020. The GenerateDataTemplate is a gradle task used to generate Pegasus schemas\u2019 Java classes based on the Pegasus schema. When compared year over year, you can clearly see a drop in GenerateDataTemplate failure counts since beginning the PDL migration in the spring of 2020.<\/p>\n<h2>Migrating existing PDSC to PDL<\/h2>\n<p>Because PDL is fully compatible with PDSC, the process of migrating PDSC to PDL can be done through an automated job. We provided a command line tool to help developers easily migrate existing PDSC files to PDL format. Developers can run <span class=\"monospace\">.\/gradlew convertToPdl<\/span> to convert all existing schemas from PDSC to PDL. Considering most repositories within LinkedIn used PDSC, it was important that the migration process was seamless in adoption. In addition to the command line tool, we also introduced an automated migration tool to convert all the existing PDSC files to the PDL format and to generate code reviews for the schema changes across all code repositories in LinkedIn. The schema owners only needed to review the Review Boards and give the greenlight for the schema changes. Once the code review is approved, the change is automatically committed to the mainline. By leveraging the automated migration tool, we were able to migrate 90% of LinkedIn\u2019s PDSC schemas to PDL in one quarter.<\/p>\n<\/p><\/div>\n<\/p><\/div>\n<\/div>\n<div class=\"resourceImageBlock section\">\n<div class=\"component-anchor-container\">\n  <a class=\"component-anchor\" name=\"post_par_resourceimageblock\"><\/a>\n <\/div>\n<ul class=\"resource-image-block single\">\n<li class=\"resource-image\"> <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/i0.wp.com\/content.linkedin.com\/content\/dam\/engineering\/site-assets\/images\/blog\/posts\/2020\/11\/pegasus-11.jpg?resize=521%2C341&#038;ssl=1\" alt=\"diagram-showing-the-migration-workflow\" height=\"341\" width=\"521\"  data-recalc-dims=\"1\"> <\/li>\n<\/ul>\n<\/div>\n<div class=\"resourceParagraph section\">\n<div class=\"component-anchor-container\">\n  <a class=\"component-anchor\" name=\"post_par_resourceparagraph\"><\/a>\n <\/div>\n<div class=\"resource-text-section\">\n<div class=\"resource-paragraph rich-text\">\n<h2>Future plans<\/h2>\n<p>Our biggest focus in continuing to expand PDL will be to provide more comprehensive IntelliJ plugin support. The PDL IntelliJ plugin will be updated to provide extension points that will allow other teams to implement context-specific auto-complete suggestions for custom schema annotations. Moreover, since the PDL schema format is more developer friendly and comes with comprehensive IntelliJ plugin support, we plan to completely replace all other schema formats (e.g., Avro) at LinkedIn with PDL. In the future, developers at Linkedin will be able to use PDL to define any kind of schema with support to be translated from PDL using the infrastructure we have built. Future updates can be found on the <a href=\"https:\/\/linkedin.github.io\/rest.li\/pdsc_syntax\" target=\"_blank\" rel=\"noopener\">Github<\/a>.<\/p>\n<h2>Acknowledgements<\/h2>\n<p>A special thanks to the Rest.li development team for working on this project: <a href=\"https:\/\/www.linkedin.com\/in\/kb000\/\" target=\"_blank\" rel=\"noopener\">Karthik Balasubramanian<\/a>, <a href=\"https:\/\/www.linkedin.com\/in\/evan-williams-5a4201108\/\" target=\"_blank\" rel=\"noopener\">Evan Williams<\/a>, <a href=\"https:\/\/www.linkedin.com\/in\/junchuan-wang-68906b59\/\" target=\"_blank\" rel=\"noopener\">Junchuan Wang<\/a>, and <a href=\"https:\/\/www.linkedin.com\/in\/aman1309\/\" target=\"_blank\" rel=\"noopener\">Aman Gupta<\/a>. Thanks to the management team for supporting this project: <a href=\"https:\/\/www.linkedin.com\/in\/heather-mckelvey-4b31441\/\" target=\"_blank\" rel=\"noopener\">Heather McKelvey<\/a>, <a href=\"https:\/\/www.linkedin.com\/in\/goksel-genc-b654802\/\" target=\"_blank\" rel=\"noopener\">Goksel Genc<\/a>, <a href=\"https:\/\/www.linkedin.com\/in\/maximelamure\/\" target=\"_blank\" rel=\"noopener\">Maxime Lamure<\/a>, and <a href=\"https:\/\/www.linkedin.com\/in\/qunzeng\/\" target=\"_blank\" rel=\"noopener\">Qunzeng Liu<\/a>.<\/p>\n<\/p><\/div>\n<\/p><\/div>\n<\/div>\n<p><a href=\"https:\/\/engineering.linkedin.com\/blog\/2020\/pegasus-data-language\" target=\"_blank\" rel=\"noopener\">Read More<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Pegasus Data Schema (PDSC) is a Pegasus schema definition language that has been used for data modeling with Rest.li services for years. It&#8217;s the underlying language that helps define data models, describe the data returned by REST endpoints, and generate derivative schemas for other uses, such as XML schemas and various database schemas. However, writing&hellip; <a class=\"more-link\" href=\"https:\/\/fde.cat\/index.php\/2021\/02\/02\/pegasus-data-language-evolving-schema-definitions-for-data-modeling\/\">Continue reading <span class=\"screen-reader-text\">Pegasus Data Language: Evolving schema definitions for data modeling<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":""},"categories":[1,7],"tags":[],"class_list":["post-188","post","type-post","status-publish","format-standard","hentry","category-external","category-technology","entry"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":669,"url":"https:\/\/fde.cat\/index.php\/2023\/01\/26\/tulip-modernizing-metas-data-platform\/","url_meta":{"origin":188,"position":0},"title":"Tulip: Modernizing Meta\u2019s data platform","date":"January 26, 2023","format":false,"excerpt":"The technical journey discusses the motivations, challenges, and technical solutions employed for warehouse schematization, especially a change to the wire serialization format employed in Meta\u2019s data platform for data interchange related to Warehouse Analytics Logging. Here, we discuss the engineering, scaling, and nontechnical challenges of modernizing\u00a0 Meta\u2019s exabyte-scale data platform\u2026","rel":"","context":"In &quot;Technology&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":649,"url":"https:\/\/fde.cat\/index.php\/2022\/11\/09\/tulip-schematizing-metas-data-platform\/","url_meta":{"origin":188,"position":1},"title":"Tulip: Schematizing Meta\u2019s data platform","date":"November 9, 2022","format":false,"excerpt":"We\u2019re sharing Tulip, a binary serialization protocol supporting schema evolution.\u00a0 Tulip assists with data schematization by addressing protocol reliability and other issues simultaneously.\u00a0 It replaces multiple legacy formats used in Meta\u2019s data platform and has achieved significant performance and efficiency gains. There are numerous heterogeneous services, such as warehouse data\u2026","rel":"","context":"In &quot;Technology&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":181,"url":"https:\/\/fde.cat\/index.php\/2020\/12\/10\/coral-a-sql-translation-analysis-and-rewrite-engine-for-modern-data-lakehouses\/","url_meta":{"origin":188,"position":2},"title":"Coral: A SQL translation, analysis, and rewrite engine for modern data lakehouses","date":"December 10, 2020","format":false,"excerpt":"Co-authors: Walaa Eldin Moustafa, Wenye Zhang, Sushant Raikar, Raymond Lam, Ron Hu, Shardul Mahadik, Laura Chen, Khai Tran, Chris Chen, and Nagarathnam Muthusamy Introduction At LinkedIn, our big data compute infrastructure continually grows over time, not only to keep pace with the growth in the number of data applications, or\u2026","rel":"","context":"In &quot;External&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":221,"url":"https:\/\/fde.cat\/index.php\/2021\/02\/02\/asyncapi-and-openapi-an-api-modeling-approach\/","url_meta":{"origin":188,"position":3},"title":"AsyncAPI and OpenAPI: an API Modeling Approach","date":"February 2, 2021","format":false,"excerpt":"AsyncAPI is gaining traction in the ecosystem of API tools. It solves an important problem: it provides a convenient way of describing the interface of event-driven systems independently of the underlying technology. With AsyncAPI, evented systems can be treated as any other API product: a productizable and reusable, self-describing building\u2026","rel":"","context":"In &quot;Technology&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":315,"url":"https:\/\/fde.cat\/index.php\/2021\/08\/31\/api-federation-growing-scalable-api-landscapes\/","url_meta":{"origin":188,"position":4},"title":"API Federation: growing scalable API landscapes","date":"August 31, 2021","format":false,"excerpt":"Organizations embrace micro-services and event-driven APIs in their technology platforms to try to achieve the promise of greater agility, increased innovation, and more autonomy for their development teams. However, after the initial success, it is not unusual for organizations to face difficulties when they try to scale their distributed platforms.\u2026","rel":"","context":"In &quot;Technology&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":283,"url":"https:\/\/fde.cat\/index.php\/2021\/08\/31\/ai-research-to-production-with-einstein-reply-recommendations\/","url_meta":{"origin":188,"position":5},"title":"AI Research to Production with Einstein Reply Recommendations","date":"August 31, 2021","format":false,"excerpt":"We all know that AI is here and it\u2019s quickly changing our lives. However, the impacts of AI are unevenly distributed and it favors those with \u201cmore data,\u201d leaving those with \u201cfew data\u201d behind. This runs counter to our Salesforce core values of Customer Success and Equality, so we set\u2026","rel":"","context":"In &quot;Technology&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/188","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/comments?post=188"}],"version-history":[{"count":1,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/188\/revisions"}],"predecessor-version":[{"id":220,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/188\/revisions\/220"}],"wp:attachment":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/media?parent=188"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/categories?post=188"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/tags?post=188"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}