{"id":176,"date":"2021-01-25T18:32:00","date_gmt":"2021-01-25T18:32:00","guid":{"rendered":"https:\/\/fde.cat\/index.php\/2021\/01\/25\/smart-argument-suite-seamlessly-connecting-python-jobs\/"},"modified":"2021-02-02T13:46:55","modified_gmt":"2021-02-02T13:46:55","slug":"smart-argument-suite-seamlessly-connecting-python-jobs","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2021\/01\/25\/smart-argument-suite-seamlessly-connecting-python-jobs\/","title":{"rendered":"Smart Argument Suite: Seamlessly connecting Python jobs"},"content":{"rendered":"
\n
\n <\/a>\n <\/div>\n
\n
\n

Co-authors: Jun Jia<\/a> and Alice Wu<\/a><\/i><\/p>\n

Introduction<\/h2>\n

It\u2019s a very common scenario that an AI solution involves composing different jobs, such as data processing and model training or evaluation, into workflows and then submitting them to an orchestration engine for execution. At large companies such as LinkedIn, there may be hundreds of thousands of such executions per day, submitted and executed by multiple teams and engineers. Any improvements in the tools used by machine learning engineers lead to significant improvements in productivity, which highlights the need for robust productivity infrastructure to support machine learning engineers.<\/p>\n

In most cases, these jobs are launched via the command line interface (CLI). Passing the arguments through the CLI becomes a producer and consumer problem: on the workflow generation side, you need to produce a set of arguments which are passed to the CLI to launch the jobs; on the other side, the launched jobs would consume the arguments passed from the CLI. We built Smart Argument Suite (smart-arg<\/a>) to make this process standard, smooth, and safe while also being human-friendly.<\/p>\n<\/p><\/div>\n<\/p><\/div>\n<\/div>\n

\n
\n <\/a>\n <\/div>\n