Tips for Safely Migrating From Gson to Moshi

Several people have already written about their journey migrating from Gson to Moshi. So why write this post? While migrating a fairly large project to Moshi, my team learned a few things (and came up with a few simple tools) that made it safer and easier to do this migration. Some of these points were either mentioned in passing in the aforementioned articles, or were skipped over altogether, and they are the focus of this post. Consequently, I’ll skip things like the motivation of moving to Moshi, since some of the aforementioned articles cover that pretty well.

A Quick Note on Gson’s field naming policy

One of the earliest concerns about migration is how to deal with Gson’s .setFieldNamingPolicy method, which allows the automatic mapping of something like created_at in json to createdAt in Java/Kotlin. One of the articles on this matter in specific that resonated with me when I read it long ago is Jesse Wilson’s article on naming conventions, which specifically addresses this point. Due to the arguments in that article, we decided to explicitly add annotations (@Json(name = "")) to the fields. While manually adding them is a lot of work, it can be done incrementally, and, as we discovered during the migration, it can help identify existing bugs that no one noticed before.

Overall Strategy

Moshi-Gson-Interop

Slack’s moshi-gson-interop is a useful tool for slowly migrating from Gson to Moshi. It tries to intelligently decide whether a particular model should be deserialized using Gson or Moshi, based on some heuristics (which one can optionally tweak if needed). One such heuristic is that classes that are annotated with Moshi’s @JsonClass are given to Moshi to deserialize.

There’s also a logger parameter that lets us tell the library how to log various events.With this, we can easily watch logcat (for example) to know which classes are deserialized by Gson and which by Moshi and annotate more classes until the entire payload of a particular screen is all deserialized via Moshi.

To set this up, we do something like:

@Provides
fun provideGson(): Gson {
  val originalGson = // ...
  val originalMoshi =  // ...
  
  return if (ENABLE_MOSHI_INTEROP) {
    val (moshi, gson) = originalMoshi.interopBuilder(originalGson)
      .logger { Timber.d("moshi-gson: $it") }
      .build()
    gson
  } else { originalGson }
}

Notice the nice property allowing us to gate this feature to a subset of people and safely continue using Gson while we build trust in our migration. This helps us stage our migration in a (mostly) safe and iterative manner.

Migration Strategy

At a high level, the migration strategy we used went a bit like this:

• choose a screen to migrate.
• enable moshi-gson-interop.
• run the app, go to the screen, and see what is still being deserialized by Gson.
• iteratively add annotations to models until the entire screen is deserialized via Moshi.
• run some runtime validations to ensure results are properly deserialized.
• fix issues found from validations.
• repeat.

As mentioned above, moshi-gson-interop makes it easy to toggle this feature to make this migration incremental. Initially, for example, only a single developer might enable interop. Once a critical mass of screens has been migrated, we might enable this for all engineers, and only start rolling it out to employees after we’ve gained enough confidence on it.

Validation and Other Useful Tools

Since most of our network calls use Retrofit, we wanted to build something for testing purposes to help us validate the correctness of the json data. Retrofit allows us to add a Converter.Factory while initializing it. Typically, this is the in-built GsonConverterFactory.create(gson) instance, for example. We can choose to either replace (or wrap) this instance to do some neat things.

For validation, for instance, we use a Converter.Factory that deserializes the data twice - once in Gson, and once in Moshi, and compares the outputs. This should only be used for development, since it’s very inefficient (both due to the double deserialization of the payload, and the one-shot reading of the entire response body). This looks something like this:

class GsonMoshiValidationConverterFactory(
  private val gson: Gson,
  private val moshi: Moshi
) : Converter.Factory() {
  override fun responseBodyConverter(
      type: Type,
      annotations: Array<out Annotation>,
      retrofit: Retrofit
    ): Converter<ResponseBody, *>? {
    return InteropConverter<Any>(type, gson, moshi)
  }

  // TODO: override requestBodyConverter here if needed for migration
  // in a similar way to what we did for the response body above. We'll
  // need to write another Converter that takes the actual value and
  // converts it to json, once for Moshi and once for Gson. Because the
  // field ordering maybe different, we can do validation by converting
  // the json strings back to objects and comparing the objects.
}

class InteropConverter<T>(
  private val type: Type,
  private val gson: Gson,
  private val moshi: Moshi
): Converter<ResponseBody, T> {

  override fun convert(value: ResponseBody): T? {
    // WARNING: This is very inefficient, do NOT use this outside of development
    val bodyAsString = value.string()
    val moshiResult = try {
      val adapter = moshi.adapter<T>(type)
      adapter.fromJson(bodyAsString)
    } catch (exception: Exception) {
      // handle parsing exception via Moshi
    }

    val gsonResult = gson.fromJson<T>(bodyAsString, type)
    if (moshiResult != gsonResult) {
      // flag mismatch between Moshi and Gson
    }
    return gsonResult
  }
}

Using this, we can clearly flag cases where Moshi and Gson deserialize something differently and can work to fix it. In order to work, however, the caveat is that the types being compared need to properly have an implemented .equals methods. One nice side effect of this is that whenever a particular item mismatches, we can also get the body as a string, and write a small validation unit test to iron out the cases.

This same trick of rolling our own Converter.Factory is useful for other things also, such as for very roughly measuring the performance of deserialization in Moshi versus Gson, and for surfacing exceptions at parsing time that are swallowed somewhere in the upstream code.

Note that if json is being sent, we should also override requestBodyConverter in the Converter.Factory similar to what we did for the rsponseBodyConverter. In this method, we can then convert the object to json for both Moshi and Gson. Note that if we compare them at this time, we’ll get a lot of noise due to the ordering of the fields being different. To work around this, we can re-serialize the json back to the type again, and check equality after the round trip.

Gotchas

While developing, we ran into several interesting problems and issues that we’ll go over here. Many of these are arguably working around bugs on the backend, and often times, the right solution to these problems is to reach out to the backend team to fix the result instead of working around it on mobile.

Sealed Classes

For sealed class hierarchies, we used moshi-sealed. Under the hood, moshi-sealed will create a PolymorphicJsonAdapterFactory, which will decide the flavor to create based on the parameter type. In case of an unknown type, a fallbackJsonAdapter can be passed in. One known issue is that if the backend sends no type (i.e. the field is absent), Moshi will throw an exception. This can likely be worked around with a custom JsonAdapter if necessary, but it would make more sense to ask backend to properly send a type in this case instead.

Handling Alternate Keys

There are some scenarios where a payload will have a field coming back with any one of several json fields, such as in this example:

data class Person(
  @SerializedName(value = "name", alternate = ["full_name"])
  val actualName: String
)

In this case, the actualName field in our Person object should contain whichever the server sent back of name or full_name. One approach for solving this is mentioned in one of the articles mentioned earlier, but that approach doesn’t work well when the model has many fields (since it would otherwise result in a lot of duplication just for mapping).

When it wasn’t easy to use the above approach, we opted to take what we thought would be an easier approach to this. Note that the below code has a bug, we’ll get to that after:

data class Person(
  val name: String?,
  @Json(name = "full_name")
  val fullName: String?
) {
  // WARNING: This is actually wrong when using Gson and will result in a null
  // value of actualName, even if one of name or fullName are passed in.
  @Transient val actualName = (name ?: fullName)!!
}

We figured we’d make all the different names as nullable variables, and knowing that the backend will always return one, we force our result to be whichever one of those fields is non-null. Note that, without the @Transient, Gson will try to look for an actual_name in the json to set this to.

While this works well for Moshi with codegen, this doesn’t actually work for Gson (when the type doesn’t use a type adapter), where actualName will return null, irrespective of the value of name or fullName. This happens to be a side-effect of how Gson makes these values via reflection.

If we look at the decompiled bytecode for our Kotlin data class, we can see that this variable gets set as part of the constructor:

  public Person(@Nullable String name, @Json(name = "full_name") @Nullable String fullName) {
    this.name = name;
    this.fullName = fullName;
    String var10001 = this.name;
    if (var10001 == null) {
       var10001 = this.fullName;
    }

    Intrinsics.checkNotNull(var10001);
    this.actualName = var10001;
  }

Looking through these posts on StackOverflow, we find out that the reason is that without a custom deserializer, Gson uses Unsafe.allocateInstance to make an instance reflectively, bypassing the constructor. It reflectively sets the properties afterwards, resulting in our actualName never being set. To fix this, we can just change actualName to a getter instead:

data class Person(
  val name: String?,
  @Json(name = "full_name")
  val fullName: String?
) {
  val actualName: String
    get() = (name ?: fullName)!!
}

Handling non-nullable Primitives

Suppose we had a model with:

// case A
data class Widget(identifier: Int)

// case B
data class Widget(identifier: Int = 0)

In case A, if no identifier is set in the json, the Gson result would set it to 0, whereas the Moshi result would crash due to the field being missing. We can fix this by updating to case B, where we set a default value.

What if the json contained an identifier set to null? We’d expect the value to be defaulted to 0, and, in Gson, it is. However, in reality, this throws an error instead. While there are some suggested solutions in this thread about working around this, we opted for a similar approach to the above:

data class Widget(
  @Json(name = "identifier")
  @SerializedName("identifier")
  internal val internalIdentifier: Int?
) {
  val identifier: Int
    get() = identifier ?: 0
}

One interesting note is that if the default in case B is not 0, Gson will still default to 0, which should make sense considering the note in the alternate keys section above.

Summary

This article offered a set of suggestions to make the migration from Gson to Moshi easier. By making the upgrade incremental, toggleable, and able to be validated, we make the migration a lot more achievable.