SparkConnectPlanner¶

Creating Instance¶

SparkConnectPlanner takes the following to be created:

• SparkSession

SparkConnectPlanner is created when:

• SparkConnectAnalyzeHandler is requested to process a request
• SparkConnectStreamHandler is requested to handlePlan and handleCommand

transformRelation¶

transformRelation(
  rel: proto.Relation): LogicalPlan

transformRelation...FIXME

transformRelation is used when:

• SparkConnectAnalyzeHandler is requested to process a request
• SparkConnectStreamHandler is requested to handlePlan

transformRelationPlugin¶

transformRelationPlugin(
  extension: ProtoAny): LogicalPlan

transformRelationPlugin...FIXME

transformMapPartitions¶

transformMapPartitions(
  rel: proto.MapPartitions): LogicalPlan

transformMapPartitions...FIXME

transformPythonUDF¶

transformPythonUDF(
  fun: proto.CommonInlineUserDefinedFunction): PythonUDF

transformPythonUDF creates a PythonUDF (based on the given fun and transformPythonFunction).

transformPythonUDF is used when:

• SparkConnectPlanner is requested to transformMapPartitions, transformGroupMap, transformCoGroupMap, transformCommonInlineUserDefinedFunction

transformPythonFunction¶

transformPythonFunction(
  fun: proto.PythonUDF): SimplePythonFunction

transformPythonFunction...FIXME

transformPythonFunction is used when:

• SparkConnectPlanner is requested to transformPythonUDF and handleRegisterPythonUDF