ResourceProfileBuilder¶

Demo¶

import org.apache.spark.resource.ResourceProfileBuilder
val rp1 = new ResourceProfileBuilder()

import org.apache.spark.resource.ExecutorResourceRequests
val execReqs = new ExecutorResourceRequests().cores(4).resource("gpu", 4)

import org.apache.spark.resource.ExecutorResourceRequests
val taskReqs = new TaskResourceRequests().cpus(1).resource("gpu", 1)

rp1.require(execReqs).require(taskReqs)
val rprof1 = rp1.build

val rpManager = sc.resourceProfileManager // (1)!
rpManager.addResourceProfile(rprof1)

NOTE: resourceProfileManager is private[spark]

Creating Instance¶

ResourceProfileBuilder takes no arguments to be created.

Building ResourceProfile¶

build: ResourceProfile

With _executorResources empty, build creates a TaskResourceProfile (with the taskResources). Otherwise, build creates a ResourceProfile (with the executorResources and the taskResources).

Executor Resources¶

executorResources: Map[String, ExecutorResourceRequest]

executorResources...FIXME

Task Resources¶

taskResources: Map[String, TaskResourceRequest]

taskResources...FIXME