{"id":18119,"date":"2020-04-09T16:04:53","date_gmt":"2020-04-09T16:04:53","guid":{"rendered":"https:\/\/www.fullcontact.com\/?p=18119"},"modified":"2023-02-07T03:33:42","modified_gmt":"2023-02-07T10:33:42","slug":"serializers-for-classes-in-datasets","status":"publish","type":"post","link":"https:\/\/www.fullcontact.com\/blog\/engineering\/serializers-for-classes-in-datasets\/","title":{"rendered":"Serializers for Classes in Datasets"},"content":{"rendered":"

Apache Spark powers a lot of modern big data processing pipelines at software companies today. At FullContact, we\u2019ve found its Dataset API to be particularly useful, since it combines the type-safe, expressive, functional style of the older RDD API with the efficiency of Spark SQL and its Catalyst optimizer.<\/p>\n

However, it has a major limitation on the types it\u2019s most easily usable with — primitive types, tuples, and case classes.<\/p>\n

I was faced with this exact limitation when a colleague’s DM slid into view with a familiar brush-knock<\/b><\/i> sound:<\/p>\n

Question for you — I’m trying to get kryo working in zeppelin by setting “spark.serializer” and “spark.kryo.registrator” on the spark interpreter.<\/i>I should NOT have to do this:<\/i>
\nimplicit val myObjEncoder = org.apache.spark.sql.Encoders.kryo[CustomMessage]<\/i><\/code>
\nif kryo is working correctly right?<\/i>The error I get is:<\/i>
\njava.lang.UnsupportedOperationException: No Encoder found for com.fullcontact.publish.CustomMessage
\n- field (class: \"com.fullcontact.publish.CustomMessage\", name: \"_1\")
\n- root class: \"scala.Tuple2\"<\/span><\/i><\/code><\/div>\n

 <\/p>\n

His actual code looked a little like this:<\/p>\n

implicit val myObjEncoder =\r\n  org.apache.spark.sql.Encoders.kryo[CustomMessage]\r\n\r\nval inputs: Dataset[InputRecord] =\r\n  spark.read.csv(\u201cs3:\/\/...\").as[InputRecord]\r\nval messages: Dataset[(String, CustomMessage)] =\r\n  inputs.map(keyMsgById)\r\n\r\nmessages.forEach(publishMsgToKafka)\r\n<\/pre>\n
 <\/p>\n
By defining a kryo encoder in that way, his intended effect was to allow the Dataset<\/code> class to accept CustomMessage (a java type) as a type parameter.<\/p>\n
So why was his code still throwing that exception when he had clearly provided an Encoder<\/code>? Normally, Spark uses several features of Scala\u2019s robust type system to make the Encoder<\/code> system invisible to the programmer, but that invisibility comes with the drawback of the type limitations I mentioned earlier. Trying to make it work outside those preferred types requires the programmer to be very particular with how they set up their Encoder<\/code> to ensure Spark uses it properly.<\/p>\n
This post will cover:<\/b><\/p>\n
    \n
  1. How Spark uses Scala\u2019s implicits and type system to construct Encoder<\/code>s<\/li>\n
  2. How Spark exposes an “escape hatch” for custom type Encoder<\/code>s<\/li>\n
  3. Why custom type Encoder<\/code>s don\u2019t work well with the automatic Encoder<\/code>s systems, which led to my coworker\u2019s problem<\/li>\n<\/ol>\n
    How Spark Finds An Encoder For Your Class<\/h2>\n
    By default, Spark reads records in a Dataset<\/code> as Row objects (Dataframe<\/code> is an alias for the Dataset[Row]<\/code> type in recent Spark releases). But Row objects are unwieldy. They cannot know the structure of your data at compile time, so you have to interact via either untyped access of indexed fields or inspect the carried schema at runtime to find the proper type casts.<\/p>\n
    It is possible to get a statically-typed Dataset<\/code> via the method Dataset.as[_]<\/code>. This lets you have typed access to the data in your records. Rather than interacting through the spark-sql API:<\/p>\n
    val df: Dataframe =\r\n  spark.read.csv(\u201cs3:\/\/my-bucket\/my-csv.csv\u201d)\r\ndf.select($\u201demail\u201d, explode($\u201dcontacts\u201d) as \u201ccontact\u201d)\r\n  .where($\u201dsent\u201d > 1000)\r\n  .where($\u201dreceived\u201d < 10)\r\n<\/pre>\n
     <\/p>\n
    You can use maps, flatmaps, reduces, folds, etc., just like in the RDD API:<\/p>\n
    val ds: Dataset[EmailAcc] =\r\n  spark.read.csv(\u201cs3:\/\/my-bucket\/my-csv.csv\u201d).as[EmailAcc]\r\nds.filter(acc => acc.sent > 1000 && acc.received < 10)\r\n  .flatMap(acc => acc.contacts.map((acc.email, _)))\r\n<\/pre>\n
     <\/p>\n
    You lose some performance because the Catalyst optimizer has to assume you need every column of your rows to deserialize your objects, but you can often gain readability compared to complex spark-sql calls or UDFs.<\/p>\n
    Let\u2019s take a look at the definition of .as[_]<\/code>:<\/p>\n
     \/**\r\n   * :: Experimental ::\r\n   * Returns a new Dataset where each record has been mapped on to the specified type. The\r\n   * method used to map columns depend on the type of `U`:\r\n   *  - When `U` is a class, fields for the class will be mapped to columns of the same name\r\n   *    (case sensitivity is determined by `spark.sql.caseSensitive`).\r\n   *  - When `U` is a tuple, the columns will be mapped by ordinal (i.e. the first column will\r\n   *    be assigned to `_1`).\r\n   *  - When `U` is a primitive type (i.e. String, Int, etc), then the first column of the\r\n   *    `DataFrame` will be used.\r\n   *\r\n   * If the schema of the Dataset does not match the desired `U` type, you can use `select`\r\n   * along with `alias` or `as` to rearrange or rename as required.\r\n   *\r\n   * @group basic\r\n   * @since 1.6.0\r\n   *\/\r\n  @Experimental\r\n  @InterfaceStability.Evolving\r\n  def as[U : Encoder]: Dataset[U] =\r\n    Dataset[U](sparkSession, logicalPlan)\r\n<\/pre>\n
     <\/p>\n
    The method appears<\/em> to take no arguments, but the syntax in the type parameters list is actually hiding an implicit parameter list. This syntax is called a context bound<\/a> and during compilation is expanded to:<\/p>\n
    def as[U](implicit evidence: Encoder[U]): Dataset[U]<\/pre>\n
     <\/p>\n
    By including this implicit parameter to its signature, .as[_]<\/code> is doing two things:<\/p>\n
      \n
    1. Requiring that there exist an object of type Encoder[U]<\/code><\/li>\n
    2. Accepting that object via the implicit scope at its call site<\/li>\n<\/ol>\n
      This is an implementation of the “type class<\/a>” pattern in Scala. Type classes enable ad-hoc polymorphism, meaning methods on Dataset<\/code> can use different code depending on the type they contain, but the choice of which code to use is deferred to some time after the Dataset class itself is implemented. In fact, the necessary code path is not chosen until the programmer\u2019s code is compiled!<\/p>\n
      The necessary code does<\/em> exist, though. It is chosen in such a way that if the programmer is using the basic supported types, they never need to mention the Encoder<\/code> type by name:<\/p>\n
      val spark = SparkSession.builder.createOrGet()\r\nimport spark.implicits._\r\n\r\n\/\/ A single-column CSV with integer values\r\nval numbers: Dataframe =\r\n  spark.read.csv(\u201cs3:\/\/my-bucket\/numbers.csv\u201d)\r\n\r\nval numbersDS: Dataset[Int] = numbers.as[Int]\r\n<\/pre>\n
       <\/p>\n
      As we are about to see, importing spark.implicits._<\/code>\u00a0makes several implicit values available to the compiler. These implicit values make it possible for the Dataset<\/code> to have the necessary Encoder<\/code> for a type the Dataset<\/code> did not know existed until the code is compiled.<\/p>\n
      spark.implicits<\/code> is an object which extends SQLImplicits, which contains the actual definitions we\u2019re interested in:<\/p>\n
      \/\/ Primitives\r\n\r\n  \/** @since 1.6.0 *\/\r\n  implicit def newIntEncoder: Encoder[Int] =\r\n    Encoders.scalaInt\r\n\r\n  \/** @since 1.6.0 *\/\r\n  implicit def newLongEncoder: Encoder[Long] =\r\n    Encoders.scalaLong\r\n\r\n  \/** @since 1.6.0 *\/\r\n  implicit def newDoubleEncoder: Encoder[Double] =\r\n    Encoders.scalaDouble\r\n\r\n... and so on ...\r\n<\/pre>\n
       <\/p>\n
      These implicit methods provide values of their return type into implicit scope wherever they are visible. The import statement from before takes all of them, thereby making Encoder<\/code>s for all the listed types: Int, Long, Double, and so on; there is also a provider for any Scala Product<\/a> types to cover tuples and case classes. Examining the definitions for the examples above, we see:<\/p>\n
        def scalaInt: Encoder[Int] = ExpressionEncoder()\r\n\r\n  def scalaLong: Encoder[Long] = ExpressionEncoder()\r\n\r\n  def scalaDouble: Encoder[Double] = ExpressionEncoder()\r\n\r\n... and so on ...\r\n\r\n<\/pre>\n
       <\/p>\n
      They are all using the same ExpressionEncoder.apply<\/code> method, and relying on the compiler\u2019s type inference to sort out what type the apply call should be parameterized with. The type parameter list for apply once again uses a context bound, but for a different purpose this time:<\/p>\n
      def apply[T : TypeTag](): ExpressionEncoder[T] = {\r\n  \/\/ We convert the not-serializable TypeTag into StructType and ClassTag.\r\n  val mirror = ScalaReflection.mirror\r\n  val tpe = typeTag[T].in(mirror).tpe\r\n  (... method implementation ...)\r\n}<\/pre>\n
       <\/p>\n
      The context bound generates a TypeTag<\/a> to circumvent the usual erasure restriction on the JVM. That way, the Encoder<\/code> implementation can leverage Scala\u2019s Reflection APIs<\/a> to inspect the type the Encoder<\/code> was built for to know how to convert between JVM values and Catalyst expressions for Spark\u2019s internal Row format. These conversions are taken from a large static mapping between Scala types and Catalyst expressions; primitive types are supported directly while tuple and case class types are supported via recursive definitions.<\/p>\n
      The actual implementation of ExpressionEncoder<\/code> that combines the Catalyst expressions is an impressive stack of scala reflections code, too complex to review in-depth here, yet still relevant to the investigation of why the custom Encoder<\/code> wasn\u2019t working. But first, a brief look at how Spark lets you create an Encoder<\/code> for non-tuple, non-case class types.<\/p>\n
      How You Can Create An Encoder For \u201cNon-Supported\u201d Types<\/h2>\n
      In addition to definitions of Encoder<\/code>s for the supported types, the Encoders object has methods to create Encoder<\/code>s using other Encoder<\/code>s (for tuples), using java serialization, using kryo serialization, and using reflection on Java beans.<\/p>\n
      In addition to definitions of Encoder<\/code>s for the supported types, the Encoders<\/code> objects has methods to create Encoder<\/code>s using java serialization, kryo serialization, reflection on Java beans, and tuples of other Encoder<\/code>s.<\/p>\n
      The java and kryo serializers work very similarly. They serialize the entire object to bytes, and then store that in a single column of data of type binary:<\/p>\n
      \/\/ Assume we have a Bean class with fields 's' and 'i'\r\nval beans = Seq(\r\n  new Bean(\"a\", 0),\r\n  new Bean(\"b\", 1),\r\n  new Bean(\"c\", 2)\r\n)\r\n\r\nimplicit val kryoEncoder: Encoder[Bean] = Encoders.kryo[Bean]\r\n\r\nval beanRDD: RDD[Bean] = sc.parallelize(beans)\r\nval beanDS: Dataset[Bean] = beanRdd.toDS\r\n\/\/ beanDS: Dataset[Bean] = [value: binary]\r\n<\/pre>\n
       <\/p>\n
      This is generally not what you want, because you now have to pay the overhead cost of going through a full serialization layer instead of letting the Catalyst engine optimize the plan for your data. And, you can\u2019t do anything with your data except to deserialize it into JVM objects — even if you wanted to start interacting with it in the SparkSQL API, you couldn\u2019t.<\/p>\n
      The Encoder<\/code> specialized for Java beans makes the data columns more available:<\/p>\n
      \/\/ Assume we have a Bean class with fields 's' and 'i'\r\nval beans = Seq(\r\n  new Bean(\"a\", 0),\r\n  new Bean(\"b\", 1),\r\n  new Bean(\"c\", 2)\r\n)\r\n\r\nimplicit val beanEncoder: Encoder[Bean] = Encoders.bean[Bean]\r\n\r\nval beanRdd: RDD[Bean] = sc.parallelize(beans)\r\nval beanDS: Dataset[Bean] = beanRdd.toDS\r\n\/\/ beanDS: Dataset[Bean] = [i: int, s: string]\r\n<\/pre>\n
       <\/p>\n
      But also comes with the major drawback of requiring your data type to adhere to the bean spec: a no-args constructor, with getters and setters for every data field.<\/p>\n
      The tuple methods are interesting because they do something we haven\u2019t seen in other Encoders yet: they accept pre-existing Encoder<\/code>s:<\/p>\n
      def tuple[T1, T2](\r\n  e1: ExpressionEncoder[T1],\r\n  e2: ExpressionEncoder[T2]): ExpressionEncoder[(T1, T2)] =\r\n  tuple(Seq(e1, e2)).asInstanceOf[ExpressionEncoder[(T1, T2)]]\r\n<\/pre>\n
       <\/p>\n
      The tuple method pulls existing<\/em> serializers from the supplied Encoder<\/code>s and composes them together without reflection. These tuple methods are not called when Spark sets up Encoder<\/code>s for a tuple of supported types, though. This leads to precisely the problem my colleague was running into when he Slacked me for help.<\/p>\n
      Why Custom Encoders Don\u2019t Work With Tuple Datasets<\/h2>\n
      If we look at the definition of of Encoder.apply<\/code> again, we can see that it does not accept any pre-existing Encoder<\/code> objects:<\/p>\n
      def apply[T : TypeTag](): ExpressionEncoder[T] = {\r\n  \/\/ We convert the not-serializable TypeTag into StructType and ClassTag.\r\n  val mirror = ScalaReflection.mirror\r\n  val tpe = typeTag[T].in(mirror).tpe\r\n  ... method implementation ...\r\n}<\/pre>\n
       <\/p>\n
      In addition,\u00a0 the tuple methods we saw earlier are not called when generating Encoders for tuple-typed Dataset<\/code>. Instead, these methods are how we reach the necessary ExpressionEncoder constructor:<\/p>\n
      \/\/ org.apache.spark.LowPrioritySQLImplicits\r\nimplicit def newProductEncoder[T <: Product : TypeTag]: Encoder[T] =\r\n  Encoders.product[T]\r\n\r\n\/\/org.apache.spark.Encoders\r\ndef product[T <: Product : TypeTag]: Encoder[T] =\r\n  ExpressionEncoder()\r\n<\/pre>\n
       <\/p>\n
      If T is a tuple type, and one of its fields is not one of the default supported types, the ExpressionEncoder<\/code> generated has no way of knowing whether an Encoder<\/code> already exists for that field, much less use that Encoder<\/code>. Instead, what will happen is that at runtime the reflection code inside ExpressionEncoder<\/code> will run, try to generate Catalyst expressions for the unsupported field, and fail because the type is not supported in its mapping of supported expressions! It then throws exactly the exception my coworker was seeing!<\/p>\n
      The solution my colleague went with was to fall back to the older RDD<\/code> API. He needed most of the data fields from the objects he was working with anyway, which made it more practical to accept a performance hit from using the API rather than attempt to build up a mapping to an analogous case class. Had he wanted to keep using his custom models in tuples in a Dataset<\/code>, the solution would have been to supply evidence of an Encoder<\/code> for the tuple type himself rather than rely on the type-inferred Encoder<\/code> system:<\/p>\n
      implicit val myObjEncoder =\r\n  Encoders.kryo[CustomMessage]\r\nimplicit val tupleEncoder =\r\n  Encoders.tuple(Encoders.STRING, myObjEncoder)\r\n\r\nval inputs: Dataset[InputRecord] =\r\n  spark.read.csv(\"s3:\/\/...\").as[InputRecord]\r\nval messages: Dataset[(String, CustomMessage)] =\r\n  inputs.map(keyMsgById)\r\n\r\nmessages.forEach(publishMsgToKafka)\r\n\r\n<\/pre>\n
       <\/p>\n
      Key Points<\/h2>\n
      After digging into the implementation of Spark\u2019s Encoder<\/code> system, I understood it well enough that I could answer my colleague\u2019s need for a way to use his custom data type in Spark. The process of unraveling the system led me to several valuable concepts:<\/p>\n