CatalystSerde Utility¶

Creating Deserializer¶

deserialize[T : Encoder](
  child: LogicalPlan): DeserializeToObject

deserialize creates a DeserializeToObject logical operator for the given child logical plan.

Internally, deserialize creates an UnresolvedDeserializer for the deserializer for the type T first and passes it on to a DeserializeToObject with an AttributeReference (being the result of generateObjAttr).

deserialize is used when:

• Dataset is requested for a QueryExecution
• ExpressionEncoder is requested to resolveAndBind
• MapPartitions utility is used to apply
• MapElements utility is used to apply
• (Catalyst DSL) deserialize operator is used

Creating Serializer¶

serialize[T : Encoder](
  child: LogicalPlan): SerializeFromObject

serialize tries to find the ExpressionEncoder for the type T and requests it for serializer expressions.

In the end, creates a SerializeFromObject logical operator with the serializer and the given child logical operator.