{"id":643,"date":"2022-10-24T16:00:50","date_gmt":"2022-10-24T16:00:50","guid":{"rendered":"https:\/\/fde.cat\/index.php\/2022\/10\/24\/from-zero-to-10-million-lines-of-kotlin\/"},"modified":"2022-10-24T16:00:50","modified_gmt":"2022-10-24T16:00:50","slug":"from-zero-to-10-million-lines-of-kotlin","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2022\/10\/24\/from-zero-to-10-million-lines-of-kotlin\/","title":{"rendered":"From zero to 10 million lines of Kotlin"},"content":{"rendered":"

We\u2019re sharing lessons learned from shifting our Android development from Java to Kotlin.<\/span>
\nKotlin is a popular language for Android development and offers some key advantages over Java.\u00a0<\/span>
\nAs of today, our Android codebase contains over 10 million lines of Kotlin code.<\/span>
\nWe\u2019re open sourcing various examples and utilities<\/a> we used to manipulate Kotlin code as part of this migration<\/p>\n

In recent years, Kotlin has become a popular language for Android development. So it only makes sense that we would shift our Android development at Meta to Kotlin as we work to make our development workflows more efficient.\u00a0<\/span><\/p>\n

Meta\u2019s Android repository is very large and reaches across our family of apps and technologies, including Facebook, Instagram, Messenger, Portal, and the Quest. Shifting away from Java, which we currently use for Android development, and over to Kotlin is not a trivial task.<\/span><\/p>\n

Why we\u2019re converting our codebase to Kotlin<\/span><\/h2>\n

Kotlin is generally regarded as a better language than Java, with higher favorability ratings than Java in the<\/span> yearly Stack Overflow developer survey<\/span><\/a>. We also compared the latest Kotlin version with Java 11, which is the latest version that can be used for Android development.<\/span><\/p>\n

Aside from its popularity, Kotlin holds some major advantages:<\/span>
\n<\/span><\/p>\n

Nullability: Null pointer exceptions are a common problem at Meta, as everywhere else. We are very good at fixing them before releasing our apps, but dealing with those issues is still time-consuming. We use internal tools to detect null safety issues earlier, and we rigorously annotate our code as part of our work to <\/span>detect such issues in Java earlier<\/span><\/a>. But even with that, Kotlin\u2019s built-in nullability handling is more robust and easier to work with.<\/span>
\nFunctional programming: Kotlin\u2019s support for inline functions and lambda expressions allows us to use a functional programming style without compromising execution speed. Although Java 8 adds support for lambdas and is available for Android, it comes at the cost of more anonymous objects, which affect performance negatively on low-end Android devices. Meta\u2019s home-brewed <\/span> Redex<\/span><\/a> minimizes these issues, but they still exist, making Kotlin a better alternative.<\/span>
\nShorter code: Kotlin\u2019s modern design makes its code shorter. Kotlin allows for dropping explicit types (as does Java 11), and together with the standard library, which is based on the functional style mentioned above, it shortens many repetitive loops into simpler statements. This shorter code is also more explicit, which can make it easier to follow.<\/span>
\nDomain-specific language (DSL) \/ Type-safe builders<\/a>: Kotlin\u2019s various features come together and let us define a DSL. Basically, a way to move definitions such as Android XMLs to be implemented directly in Kotlin code. But this tool should be wielded carefully because implementing DSLs in Kotlin can either be useful or turn into overengineering.<\/span><\/p>\n

\u00a0<\/span>However, adopting Kotlin also has a few disadvantages that we could not ignore:<\/span>
\n<\/span><\/p>\n

Adopting another language could mean we\u2019ll have to deal with a mixed codebase of two languages for a long time. Kotlin is very good at interacting with Java, but quirks do pop up at times.<\/span>
\nKotlin is a popular language, but compared with Java, the popularity gap is clear. Java is the world\u2019s second or third most popular language (depending on how one measures this). This means fewer tools are available. Worse than that, all the Kotlin tools need to account for Kotlin and Java interoperability, which complicates their implementation.<\/span><\/p>\n

Lastly, our biggest worry was build times. We knew from the start that Kotlin\u2019s build times would be longer than Java\u2019s. The language and its ecosystem are more complicated, and Java had two decades of a head start to optimize its compiler. Since we own several large apps, the consequences of longer build times could negatively impact our developers\u2019 experience. Hearing anecdotes such as <\/span>OkHttp\u2019s experience migrating to Kotlin<\/span><\/a> painted a less-than-ideal picture.<\/span><\/p>\n

How we\u2019re approaching the migration<\/span><\/h2>\n

Migrating to Kotlin is both surprisingly easy and very complicated. It\u2019s easy because Kotlin\u2019s design allows simple conversion from Java with well-thought-out interoperability. This design made it possible for JetBrains to supply the developer community with J2K, the Java to Kotlin converter that comes with IntelliJ\/Android Studio.<\/span><\/p>\n

But even with J2K, the migration is still complicated. J2K doesn\u2019t always get things correct, and the interoperability of Java and Kotlin exposes us to several edge cases. These run the gamut from style fixes to make the code cleaner all the way to tricky runtime behavior changes (which we will discuss later).<\/span><\/p>\n

Going into this migration, we had two options:<\/span>
\n<\/span><\/p>\n

We could make it possible to write new code at Meta using Kotlin but leave most of the existing code in Java.<\/span>
\nWe could attempt to convert almost all our in-house code into Kotlin.<\/span><\/p>\n

The advantage of the first option is clear \u2014 it\u2019s much less work. But there are two notable disadvantages to this approach. First, enabling interoperability between Kotlin and Java code introduces the use of platform types in Kotlin. Platform types give rise to runtime null pointer dereferences that result in crashes instead of the static safety offered by pure Kotlin code. In some complicated cases, Kotlin\u2019s null check elision can let nulls through and create surprising null pointer exceptions later. This could happen if, for example, Kotlin code calls a Kotlin interface implemented by a Java interface.<\/span><\/p>\n

Other issues include Java\u2019s inability to tag type parameters as nullable (until recently), and Kotlin\u2019s overloading rules taking nullability into account, while Java\u2019s overloading rules do not.<\/span><\/p>\n

The second disadvantage comes when considering that most software development at Meta \u2014 as with anywhere else \u2014 entails modifying existing code. If most of our code is in Java, we aren\u2019t allowing our developers to fully enjoy Kotlin. Since the migration is a long process, expecting every engineer to convert a file to Kotlin before they touch it is exhausting and inefficient.<\/span><\/p>\n

How we\u2019re migrating to Kotlin<\/span><\/h2>\n

We considered these two options and decided our goal would be to convert almost all our code into Kotlin. After a slow start, where we had to fix a few blockers, we were able to begin converting a lot of code at bulk. Today, our Android apps for Facebook, Messenger, and Instagram each have more than 1 million lines of Kotlin code, and the rate of conversion is increasing. In total, our Android codebase has more than 10 millions lines of Kotlin code.<\/span><\/p>\n

Unblocking<\/span><\/h3>\n

As soon as we started trying to use Kotlin in our existing apps we hit some issues. For example, we needed to<\/span> update Redex<\/span><\/a> to support bytecode patterns that Java did not generate. In addition, some internal libraries we use depend on transforming bytecode during compilation to achieve better performance. This code did not work when run as part of a Kotlin compilation.<\/span><\/p>\n

We built workarounds for our tools to solve these issues. If you migrate your code to Kotlin and have a bunch of in-house optimizations, you should expect similar problems. However, we expect most people will not face such issues.<\/span><\/p>\n

We also identified various gaps with existing tooling. For example, Kotlin syntax highlighting in our code review or wiki was lacking. We <\/span> updated Pygments<\/span><\/a>, the library we are using, to bring the experience to par with Java. We updated some of our internal code-modding tools to be able to handle Kotlin. We also built <\/span>Ktfmt<\/span><\/a>, a deterministic Kotlin formatter based on the code and philosophy of <\/span> google-java-format<\/span><\/a>.<\/span><\/p>\n

Accelerating the migration<\/span><\/h3>\n

With our tools ready, we could now convert any part of our code to Kotlin. But each migration required a bunch of boilerplate work that had to be done manually. J2K is a general tool and, as such, avoids understanding the code it is converting. This creates many cases that require manual work.<\/span><\/p>\n

One popular example is the usage of <\/span> JUnit testing rules<\/span><\/a>, which are commonly used in tests.<\/span><\/p>\n

For example, you may want to verify the correct exceptions are thrown using the ExpectedException rule:<\/span><\/p>\n

@Rule public<\/strong> ExpectedException expectedException = ExpectedException.none();<\/p>\n

When J2K converts this code to Kotlin, we get:<\/span>\u00a0<\/span><\/p>\n

@Rule var<\/strong> expectedException = ExpectedException.none()<\/p>\n

This code looks equivalent at first to the original Java, but due to Kotlin\u2019s use site annotations, it is actually equivalent to:<\/span><\/p>\n

@Rule private ExpectedException expectedException = ExpectedException.none();<\/p>\n

public ExpectedException getExpectedException() {
\n return expectedException
\n}<\/p>\n

Trying to run this test will fail and return an error: \u201cThe @Rule expectedException must be public\u201d since JUnit will see a private field annotated with @Rule. This is a common problem that has been answered many times in forums and can be fixed in one of two ways: either add `@JvmField` to the field or add an <\/span> annotation use-site<\/span><\/a> to the annotation so it is `@get:Rule`:<\/span><\/p>\n

\/\/ solution 1: use `get` as the use-site for the annotation
\n@get:Rule var expectedException = ExpectedException.none()<\/p>\n

\/\/ solution 2: generate JVM code only for a Java field without a getter
\n@JvmField @Rule var expectedException = ExpectedException.none()<\/p>\n

Since J2K does not (and probably should not) know the intricacies of JUnit, it cannot do the right thing. Even if we thought JUnit were popular enough that it might warrant having J2K know about it, we would still have this same problem with many niche frameworks.<\/span><\/p>\n

For example, a lot of Android Java code will use the utility methods from android.text.TextUtils, such as isEmpty to simplify the check of some strings. In Kotlin, however, we have the built-in standard library method String.isNullOrEmpty. This method is preferable not only because it\u2019s in the standard library, but also because it has a contract that tells the Kotlin compiler that if it returns false, the object being tested can no longer be null and can be smart-cast to a String.<\/span><\/p>\n

Java code has many other similar helper methods, and many libraries implement these same basic methods. All of these should be replaced with the standard Kotlin methods to simplify the code and allow the compiler to properly detect nonnullable types.<\/span><\/p>\n

We have found many instances of these small fixes. Some are easy to do (such as replacing isEmpty), some require research to figure out the first time (as in the case of JUnit rules), and a few are workarounds for actual J2K bugs that can result in anything from a build error to <\/span> different runtime behavior<\/span><\/a>.<\/span><\/p>\n

To solve these issues, we put J2K in the middle of a three-step pipeline:<\/span>
\n<\/span><\/p>\n

In the first step, we take one Java package and prepare it to be converted to Kotlin. This step mostly works around bugs and does conversions needed for our internal tools.<\/span>
\nThe second step is running J2K. We have been able to run Android Studio in a headless mode and invoke J2K, which allows us to run the entire pipeline as a script.<\/span>
\nIn the last step, we postprocess the new Kotlin files. This step contains the majority of our automated refactors and fixes steps such as tagging a JUnit rule as a @JvmField. As part of this step, we also apply our autocorrecting linters and apply various Android Studio suggestions in headless mode.<\/span><\/p>\n

These automations do not resolve all the problems, but we are able to prioritize the most common ones. We run our conversion script (aptly named Kotlinator) on modules, prioritizing active and simpler modules first. We then observe the resulting commit: Does it compile? Does it pass our continuous integration smoothly? If it does, we commit it. And if not, we look at the issues and devise new automatic refactors to fix them. For issues that don\u2019t seem systematic or new, we simply fix them manually and commit the change.<\/span><\/p>\n

For the Java refactors, we use JavaASTParser, which allows us to resolve some types, along with other internal tools.<\/span><\/p>\n

For the Kotlin side, we don\u2019t yet have a good solution that can resolve types, so we opt to use the Kotlin compiler APIs. Loading a Kotlin code into its <\/span> PSI AST<\/span><\/a> is simple and, in practice, gives us all the power we need to continuously improve Kotlinator.<\/span><\/p>\n

Since we started this process, we\u2019ve learned a bit while using the Kotlin compiler APIs, so we\u2019re also <\/span>releasing a limited set of some of the automated refactorings<\/span><\/a> in the hope that it will help more developers use the Kotlin compiler parser to their advantage.<\/span><\/p>\n

Here is a quick example of using a template-matching utility we built to handle the Android TextUtils.isEmpty case mentioned above:\u00a0<\/span><\/p>\n

val ktFile = load(path)
\n\/\/ make sure the correct class is imported
\nif (ktFile.imports.none {
\n it.importedReference?.text == “android.text.TextUtils”
\n }) {
\n return
\n}
\nval newContent = ktFile.replaceAll<KtExpression>(
\n matcher = template {
\n val a by match<KtExpression> {}
\n “TextUtils.isEmpty($a)”
\n },
\n replaceWith = {
\n val a by it.variables
\n “$a.isNullOrEmpty()”
\n })
\nwrite(path, newContent)<\/p>\n

If you have an adversarial mind, you can probably see a bunch of ways to break this refactor. In practice, we find that they don\u2019t show up in our code, and that this is enough for us to move forward.<\/span><\/p>\n

What we\u2019ve learned from our Kotlin migration<\/span><\/h2>\n

With our tooling improvements, we were already able to convert a sizable chunk of our code into Kotlin. We already have more than 10 million lines of Kotlin code in our codebase, and the majority of Android developers at Meta are now writing Kotlin code. <\/span>\u00a0<\/span><\/p>\n

This scale has led us to a few conclusions:<\/span><\/p>\n

Reduced code length<\/span><\/h3>\n

We expected Kotlin code to be shorter going into this migration. Some files were indeed cut in half (and even more), especially when the Java code had to null-check many fields, or when simple repetitive loops could be replaced with standard Kotlin methods that accept a lambda, such as \u201cfirst,\u201d \u201csingle,\u201d \u201cany,\u201d \u201cetc.\u201d<\/span><\/p>\n

However, a lot of our code is simply about passing values around. For example, a Litho class, which defines UI and its styling, stays about the same length regardless of whether it\u2019s in Java or Kotlin.<\/span><\/p>\n

On average, we\u2019ve seen a reduction of 11 percent in the number of lines of code from this migration. We have seen much higher numbers quoted online, but we suspect these numbers are derived from specific examples.<\/span><\/p>\n

We are still happy about this number, as the lines removed are usually boilerplate code, which is less implicit than its shorter Kotlin counterpart.<\/span><\/p>\n

Maintaining execution speed<\/span><\/h3>\n

Since Kotlin compiles to the same JVM bytecode, we did not expect to see any execution speed performance regressions from this migration.<\/span><\/p>\n

To verify this, we ran multiple A\/B tests comparing a Java implementation with a Kotlin implementation, using Kotlin features such as lambdas, nullability, and more. We found that Kotlin matched the performance of Java, as we expected.<\/span><\/p>\n

Build size is not an issue<\/span><\/h3>\n

The Kotlin standard library is pretty small, and since all of our releases use Proguard and Redex, only some of it even makes it into a release APK. Therefore, size hasn\u2019t proved to be a problem except in a situation where a few KBs of extra code matter. In those cases, we found that by avoiding Kotlin\u2019s standard library and using the already available Java methods, the problem can be solved. For example, using CharSequence.split from kotlin.text would add a few classes and constants when compared with using Java\u2019s String.split.<\/span><\/p>\n

Addressing longer build times<\/span><\/h3>\n

We expected build times would be longer with Kotlin since it\u2019s a relatively new language, compared with Java. We guessed right, and our developers noticed that build times increased as we used more Kotlin in our codebase.<\/span><\/p>\n

While the Kotlin compiler keeps improving, we looked at ways we can improve build times on our end. One of them is source-only ABI support in our Buck build system, which can generate ABI jars for dependencies in the build graph without actually compiling them. This is <\/span> already supported for Java<\/span><\/a>, and we\u2019re working on a Kotlin version, which we believe will flatten the build graph and vastly improve incremental build speeds.<\/span><\/p>\n

The other area we investigated is annotation processing, a known pain point for build speed. Kotlin supports annotation processors using KAPT, which is currently in <\/span> maintenance mode<\/span><\/a>. KAPT works by generating a Java code stub for the existing Java annotation processor code to run. It\u2019s nice since it lets your existing code work without modifications, but it\u2019s slow due to the generation of the Java stub.<\/span><\/p>\n

The solution is to use <\/span> KSP<\/span><\/a>, the new, recommended way to handle annotation processing. We added support for it in Buck and are working on porting our existing processors to KSP using an adapter we developed. This does minimize the cost of running annotation processors, but only if no KAPT-based processors remain. The downside is that this requires a lot of work to update all the annotation processors. We found the <\/span> interop library<\/span><\/a> by the Room developers as another option to reuse existing code, but there\u2019s still necessary migration work needed for each processor.<\/span><\/p>\n

What\u2019s next for Kotlin at Meta?<\/span><\/h2>\n

Our migration to Kotlin is still ongoing and accelerating. We have been able to allow any Android developer at Meta who wants to use Kotlin to do so and have supplied them with tools to easily migrate existing code to Kotlin.<\/span><\/p>\n

Kotlin still lacks some of the tools and optimizations that we have grown used to from working with Java. But we\u2019re working to close those gaps. As we make progress and these tools and libraries mature, we will also work to release them back to the community.<\/span><\/p>\n

The post From zero to 10 million lines of Kotlin<\/a> appeared first on Engineering at Meta<\/a>.<\/p>\n

Engineering at Meta<\/p>","protected":false},"excerpt":{"rendered":"

We\u2019re sharing lessons learned from shifting our Android development from Java to Kotlin. Kotlin is a popular language for Android development and offers some key advantages over Java.\u00a0 As of today, our Android codebase contains over 10 million lines of Kotlin code. We\u2019re open sourcing various examples and utilities we used to manipulate Kotlin code… Continue reading From zero to 10 million lines of Kotlin<\/span><\/a><\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":""},"categories":[7],"tags":[],"class_list":["post-643","post","type-post","status-publish","format-standard","hentry","category-technology","entry"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":334,"url":"https:\/\/fde.cat\/index.php\/2021\/08\/31\/building-data-pipelines-using-kotlin\/","url_meta":{"origin":643,"position":0},"title":"Building Data Pipelines Using Kotlin","date":"August 31, 2021","format":false,"excerpt":"Co-written by Alex\u00a0OscherovUp until recently, we, like many companies, built our data pipelines in any one of a handful of technologies using Java or Scala, including Apache Spark, Storm, and Kafka. But Java is a very verbose language, so writing these pipelines in Java involves a lot of boilerplate code.\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":656,"url":"https:\/\/fde.cat\/index.php\/2022\/11\/22\/retrofitting-null-safety-onto-java-at-meta\/","url_meta":{"origin":643,"position":1},"title":"Retrofitting null-safety onto Java at Meta","date":"November 22, 2022","format":false,"excerpt":"We developed a new static analysis tool called Nullsafe that is used at Meta to detect NullPointerException (NPE) errors in Java code. Interoperability with legacy code and gradual deployment model were key to Nullsafe\u2019s wide adoption and allowed us to recover some null-safety properties in the context of an otherwise\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":253,"url":"https:\/\/fde.cat\/index.php\/2021\/08\/31\/sre-weekly-issue-254\/","url_meta":{"origin":643,"position":2},"title":"SRE Weekly Issue #254","date":"August 31, 2021","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, StackHawk: Need to run a standalone Kotlin app as a fat jar in a Gradle project? Check out how we handled that! http:\/\/sthwk.com\/kotlin-with-gradle Articles Coinbase Incident Post Mortem: January 6\u20137, 2021 This one\u2019s juicy. At one point, the front-end was blocked up,\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":548,"url":"https:\/\/fde.cat\/index.php\/2022\/03\/08\/an-open-source-compositional-deadlock-detector-for-android-java\/","url_meta":{"origin":643,"position":3},"title":"An open source compositional deadlock detector for Android Java","date":"March 8, 2022","format":false,"excerpt":"What the research is: We\u2019ve developed a new static analyzer that catches deadlocks in Java code for Android without ever running the code. What distinguishes our analyzer from past research is its ability to analyze revisions in codebases with hundreds of millions of lines of code. We have deployed our\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":499,"url":"https:\/\/fde.cat\/index.php\/2021\/11\/01\/the-journey-of-building-a-scalable-api\/","url_meta":{"origin":643,"position":4},"title":"The Journey of Building a Scalable API","date":"November 1, 2021","format":false,"excerpt":"APIs are an essential tool to allow partners, developers, and applications to consume, communicate\u00a0,or build on top of the various capabilities your microservices provide. Building a high quality API that can scale and perform with the business ecosystem is not easy and requires putting thought and planning into everything, from\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":480,"url":"https:\/\/fde.cat\/index.php\/2021\/09\/29\/open-sourcing-mariana-trench-analyzing-android-and-java-app-security-in-depth\/","url_meta":{"origin":643,"position":5},"title":"Open-sourcing Mariana Trench: Analyzing Android and Java app security in depth","date":"September 29, 2021","format":false,"excerpt":"We\u2019re sharing details about Mariana Trench (MT), a tool we use to spot and prevent security and privacy bugs in Android and Java applications. As part of our effort to help scale security through building automation, we recently open-sourced MT to support security engineers at Facebook and across the industry.\u00a0\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/643","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/comments?post=643"}],"version-history":[{"count":0,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/643\/revisions"}],"wp:attachment":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/media?parent=643"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/categories?post=643"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/tags?post=643"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}