Skip to content


ResourceProfileBuilder is a fluent API for Spark developers to build ResourceProfiles (to associate with an RDD).

Available in Scala and Python APIs

ResourceProfileBuilder is available in Scala and Python APIs.

Creating Instance

ResourceProfileBuilder takes no arguments to be created.

Building ResourceProfile

build: ResourceProfile

build creates a ResourceProfile:

Executor Resources

executorResources: Map[String, ExecutorResourceRequest]


Task Resources

taskResources: Map[String, TaskResourceRequest]

taskResources is TaskResourceRequests specified by users (by their resource names)

taskResources are specified using require method.

taskResources can be removed using clearTaskResourceRequests method.

taskResources can be printed out using toString method.

taskResources is used when:


import org.apache.spark.resource.ResourceProfileBuilder
val rp1 = new ResourceProfileBuilder()

import org.apache.spark.resource.ExecutorResourceRequests
val execReqs = new ExecutorResourceRequests().cores(4).resource("gpu", 4)

import org.apache.spark.resource.ExecutorResourceRequests
val taskReqs = new TaskResourceRequests().cpus(1).resource("gpu", 1)

val rprof1 =
val rpManager = sc.resourceProfileManager // (1)!
  1. NOTE: resourceProfileManager is private[spark]