Skip to content

SparkConnectPlanner

Creating Instance

SparkConnectPlanner takes the following to be created:

SparkConnectPlanner is created when:

transformRelation

transformRelation(
  rel: proto.Relation): LogicalPlan

transformRelation...FIXME

transformRelation is used when:

  • SparkConnectPlanExecution is requested to handlePlan
  • SparkConnectPlanner is requested to do many things
  • SparkConnectAnalyzeHandler is requested to process a request

Handle Plugins for Spark Connect Relation Types

transformRelationPlugin(
  extension: ProtoAny): LogicalPlan

transformRelationPlugin...FIXME

transformMapPartitions

transformMapPartitions(
  rel: proto.MapPartitions): LogicalPlan

transformMapPartitions...FIXME

transformPythonUDF

transformPythonUDF(
  fun: proto.CommonInlineUserDefinedFunction): PythonUDF

transformPythonUDF creates a PythonUDF (based on the given fun and transformPythonFunction).

transformPythonUDF is used when:

transformPythonFunction

transformPythonFunction(
  fun: proto.PythonUDF): SimplePythonFunction

transformPythonFunction...FIXME

transformPythonFunction is used when:

Process Command

process(
  command: proto.Command,
  responseObserver: StreamObserver[ExecutePlanResponse],
  executeHolder: ExecuteHolder): Unit

process handles the input Command based on its type.

Command Type Handler
REGISTER_FUNCTION handleRegisterUserDefinedFunction
REGISTER_TABLE_FUNCTION handleRegisterUserDefinedTableFunction
WRITE_OPERATION handleWriteOperation
CREATE_DATAFRAME_VIEW handleCreateViewCommand
WRITE_OPERATION_V2 handleWriteOperationV2
EXTENSION handleCommandPlugin
SQL_COMMAND handleSqlCommand
WRITE_STREAM_OPERATION_START handleWriteStreamOperationStart
STREAMING_QUERY_COMMAND handleStreamingQueryCommand
STREAMING_QUERY_MANAGER_COMMAND handleStreamingQueryManagerCommand
GET_RESOURCES_COMMAND handleGetResourcesCommand

process is used when:

handleCommandPlugin

handleCommandPlugin(
  extension: ProtoAny,
  executeHolder: ExecuteHolder): Unit

handleCommandPlugin...FIXME

handleMlCommand

handleMlCommand(
  command: proto.MlCommand,
  responseObserver: StreamObserver[proto.ExecutePlanResponse]): Unit

handleMlCommand handles the ML command (with this SessionHolder).

handleMlCommand requests this ExecuteHolder for the ExecuteEventsManager to postFinished.

In the end, handleMlCommand registers a response with the result of executing the ML command.

handleMlCommand is used when:

  • SparkConnectPlanner is requested to process a ML_COMMAND command