Skip to content

ResourceProfileBuilder

Demo

import org.apache.spark.resource.ResourceProfileBuilder
val rp1 = new ResourceProfileBuilder()

import org.apache.spark.resource.ExecutorResourceRequests
val execReqs = new ExecutorResourceRequests().cores(4).resource("gpu", 4)

import org.apache.spark.resource.ExecutorResourceRequests
val taskReqs = new TaskResourceRequests().cpus(1).resource("gpu", 1)

rp1.require(execReqs).require(taskReqs)
val rprof1 = rp1.build
val rpManager = sc.resourceProfileManager // (1)!
rpManager.addResourceProfile(rprof1)
  1. NOTE: resourceProfileManager is private[spark]

Creating Instance

ResourceProfileBuilder takes no arguments to be created.

Building ResourceProfile

build: ResourceProfile

With _executorResources empty, build creates a TaskResourceProfile (with the taskResources). Otherwise, build creates a ResourceProfile (with the executorResources and the taskResources).

Executor Resources

executorResources: Map[String, ExecutorResourceRequest]

executorResources...FIXME

Task Resources

taskResources: Map[String, TaskResourceRequest]

taskResources...FIXME