Skip to content

ExecutorResourceRequests

ExecutorResourceRequests is a set of ExecutorResourceRequests for Spark developers to (programmatically) specify resources for an RDD to be applied at stage level:

Creating Instance

ExecutorResourceRequests takes no arguments to be created.

ExecutorResourceRequests is created when:

Serializable

ExecutorResourceRequests is a Serializable (Java).

resource

resource(
  resourceName: String,
  amount: Long,
  discoveryScript: String = "",
  vendor: String = ""): this.type

resource creates a ExecutorResourceRequest and registers it under resourceName.

resource is used when:

Text Representation

ExecutorResourceRequests presents itself as:

Executor resource requests: [_executorResources]

Demo

import org.apache.spark.resource.ExecutorResourceRequests
val executorResources = new ExecutorResourceRequests()
  .memory("2g")
  .memoryOverhead("512m")
  .cores(8)
  .resource(
    resourceName = "my-custom-resource",
    amount = 1,
    discoveryScript = "/this/is/path/to/discovery/script.sh",
    vendor = "pl.japila")

scala> println(executorResources)
Executor resource requests: {memoryOverhead=name: memoryOverhead, amount: 512, script: , vendor: , memory=name: memory, amount: 2048, script: , vendor: , cores=name: cores, amount: 8, script: , vendor: , my-custom-resource=name: my-custom-resource, amount: 1, script: /this/is/path/to/discovery/script.sh, vendor: pl.japila}