github.com/ferranbt/nomad@v0.9.3-0.20190607002617-85c449b7667c/website/source/guides/analytical-workloads/spark/resource.html.md (about)

     1  ---
     2  layout: "guides"
     3  page_title: "Apache Spark Integration - Resource Allocation"
     4  sidebar_current: "guides-analytical-workloads-spark-resource"
     5  description: |-
     6    Learn how to configure resource allocation for your Spark applications.
     7  ---
     8  
     9  # Resource Allocation
    10  
    11  Resource allocation can be configured using a job template or through
    12  configuration properties. Here is a sample template in HCL syntax (this would
    13  need to be converted to JSON):
    14  
    15  ```hcl
    16  job "template" {
    17    group "group-name" {
    18  
    19      task "executor" {
    20        meta {
    21          "spark.nomad.role" = "executor"
    22        }
    23  
    24        resources {
    25          cpu = 2000
    26          memory = 2048
    27          network {
    28            mbits = 100
    29          }
    30        }
    31      }
    32    }
    33  }
    34  ```
    35  Resource-related configuration properties are covered below.
    36  
    37  ## Memory
    38  
    39  The standard Spark memory properties will be propagated to Nomad to control
    40  task resource allocation: `spark.driver.memory` (set by `--driver-memory`) and
    41  `spark.executor.memory` (set by `--executor-memory`). You can additionally specify
    42   [spark.nomad.shuffle.memory](/guides/analytical-workloads/spark/configuration.html#spark-nomad-shuffle-memory)
    43    to control how much memory Nomad allocates to  shuffle service tasks.
    44  
    45  ## CPU
    46  
    47  Spark sizes its thread pools and allocates tasks based on the number of CPU
    48  cores available. Nomad manages CPU allocation in terms of processing speed
    49  rather than number of cores. When running Spark on Nomad, you can control how
    50  much CPU share Nomad will allocate to tasks using the
    51  [spark.nomad.driver.cpu](/guides/analytical-workloads/spark/configuration.html#spark-nomad-driver-cpu)
    52  (set by `--driver-cpu`),
    53  [spark.nomad.executor.cpu](/guides/analytical-workloads/spark/configuration.html#spark-nomad-executor-cpu)
    54  (set by `--executor-cpu`) and
    55  [spark.nomad.shuffle.cpu](/guides/analytical-workloads/spark/configuration.html#spark-nomad-shuffle-cpu)
    56  properties. When running on Nomad, executors will be configured to use one core
    57  by default, meaning they will only pull a single 1-core task at a time. You can
    58  set the `spark.executor.cores` property (set by `--executor-cores`) to allow
    59  more tasks to be executed concurrently on a single executor.
    60  
    61  ## Network
    62  
    63  Nomad does not restrict the network bandwidth of running tasks, bit it does
    64  allocate a non-zero number of Mbit/s to each task and uses this when bin packing
    65  task groups onto Nomad clients. Spark defaults to requesting the minimum of 1
    66  Mbit/s per task, but you can change this with the
    67  [spark.nomad.driver.networkMBits](/guides/analytical-workloads/spark/configuration.html#spark-nomad-driver-networkmbits),
    68  [spark.nomad.executor.networkMBits](/guides/analytical-workloads/spark/configuration.html#spark-nomad-executor-networkmbits), and
    69  [spark.nomad.shuffle.networkMBits](/guides/analytical-workloads/spark/configuration.html#spark-nomad-shuffle-networkmbits)
    70  properties.
    71  
    72  ## Log rotation
    73  
    74  Nomad performs log rotation on the `stdout` and `stderr` of its tasks. You can
    75  configure the number number and size of log files it will keep for driver and
    76  executor task groups using
    77  [spark.nomad.driver.logMaxFiles](/guides/analytical-workloads/spark/configuration.html#spark-nomad-driver-logmaxfiles)
    78  and [spark.nomad.executor.logMaxFiles](/guides/analytical-workloads/spark/configuration.html#spark-nomad-executor-logmaxfiles).
    79  
    80  ## Next Steps
    81  
    82  Learn how to [dynamically allocate Spark executors](/guides/analytical-workloads/spark/dynamic.html).