ResourceProfileBuilder¶
Demo¶
import org.apache.spark.resource.ResourceProfileBuilder
val rp1 = new ResourceProfileBuilder()
import org.apache.spark.resource.ExecutorResourceRequests
val execReqs = new ExecutorResourceRequests().cores(4).resource("gpu", 4)
import org.apache.spark.resource.ExecutorResourceRequests
val taskReqs = new TaskResourceRequests().cpus(1).resource("gpu", 1)
rp1.require(execReqs).require(taskReqs)
val rprof1 = rp1.build
val rpManager = sc.resourceProfileManager // (1)!
rpManager.addResourceProfile(rprof1)
- NOTE:
resourceProfileManager
isprivate[spark]
Creating Instance¶
ResourceProfileBuilder
takes no arguments to be created.
Building ResourceProfile¶
build: ResourceProfile
With _executorResources empty, build
creates a TaskResourceProfile (with the taskResources). Otherwise, build
creates a ResourceProfile (with the executorResources and the taskResources).
Executor Resources¶
executorResources: Map[String, ExecutorResourceRequest]
executorResources
...FIXME
Task Resources¶
taskResources: Map[String, TaskResourceRequest]
taskResources
...FIXME