ResourceProfileBuilder¶

ResourceProfileBuilder is a fluent API for Spark developers to build ResourceProfiles (to associate with an RDD).

Creating Instance¶

ResourceProfileBuilder takes no arguments to be created.

Building ResourceProfile¶

build: ResourceProfile

build creates a ResourceProfile:

• TaskResourceProfile when _executorResources are undefined
• ResourceProfile with the executorResources and the taskResources

Executor Resources¶

executorResources: Map[String, ExecutorResourceRequest]

executorResources...FIXME

Task Resources¶

taskResources: Map[String, TaskResourceRequest]

taskResources is TaskResourceRequests specified by users (by their resource names)

taskResources are specified using require method.

taskResources can be removed using clearTaskResourceRequests method.

taskResources can be printed out using toString method.

taskResources is used when:

• ResourceProfileBuilder is requested to build a ResourceProfile

Demo¶

import org.apache.spark.resource.ResourceProfileBuilder
val rp1 = new ResourceProfileBuilder()

import org.apache.spark.resource.ExecutorResourceRequests
val execReqs = new ExecutorResourceRequests().cores(4).resource("gpu", 4)

import org.apache.spark.resource.ExecutorResourceRequests
val taskReqs = new TaskResourceRequests().cpus(1).resource("gpu", 1)

rp1.require(execReqs).require(taskReqs)
val rprof1 = rp1.build

val rpManager = sc.resourceProfileManager // (1)!
rpManager.addResourceProfile(rprof1)

1. NOTE: resourceProfileManager is private[spark]