github.com/smintz/nomad@v0.8.3/website/source/guides/spark/resource.html.md (about)

     1  ---
     2  layout: "guides"
     3  page_title: "Apache Spark Integration - Resource Allocation"
     4  sidebar_current: "guides-spark-resource"
     5  description: |-
     6    Learn how to configure resource allocation for your Spark applications.
     7  ---
     8  
     9  # Resource Allocation
    10  
    11  Resource allocation can be configured using a job template or through 
    12  configuration properties. Here is a sample template in HCL syntax (this would 
    13  need to be converted to JSON):
    14  
    15  ```hcl
    16  job "template" {
    17    group "group-name" {
    18  
    19      task "executor" {
    20        meta {
    21          "spark.nomad.role" = "executor"
    22        }
    23  
    24        resources {
    25          cpu = 2000
    26          memory = 2048
    27          network {
    28            mbits = 100
    29          }
    30        }
    31      }
    32    }
    33  }
    34  ```
    35  Resource-related configuration properties are covered below.
    36  
    37  ## Memory
    38  
    39  The standard Spark memory properties will be propagated to Nomad to control 
    40  task resource allocation: `spark.driver.memory` (set by `--driver-memory`) and 
    41  `spark.executor.memory` (set by `--executor-memory`). You can additionally specify
    42   [spark.nomad.shuffle.memory](/guides/spark/configuration.html#spark-nomad-shuffle-memory)
    43    to control how much memory Nomad allocates to  shuffle service tasks.
    44  
    45  ## CPU
    46  
    47  Spark sizes its thread pools and allocates tasks based on the number of CPU 
    48  cores available. Nomad manages CPU allocation in terms of processing speed 
    49  rather than number of cores. When running Spark on Nomad, you can control how 
    50  much CPU share Nomad will allocate to tasks using the 
    51  [spark.nomad.driver.cpu](/guides/spark/configuration.html#spark-nomad-driver-cpu) 
    52  (set by `--driver-cpu`), 
    53  [spark.nomad.executor.cpu](/guides/spark/configuration.html#spark-nomad-executor-cpu) 
    54  (set by `--executor-cpu`) and 
    55  [spark.nomad.shuffle.cpu](/guides/spark/configuration.html#spark-nomad-shuffle-cpu) 
    56  properties. When running on Nomad, executors will be configured to use one core 
    57  by default, meaning they will only pull a single 1-core task at a time. You can 
    58  set the `spark.executor.cores` property (set by `--executor-cores`) to allow 
    59  more tasks to be executed concurrently on a single executor.
    60  
    61  ## Network
    62  
    63  Nomad does not restrict the network bandwidth of running tasks, bit it does 
    64  allocate a non-zero number of Mbit/s to each task and uses this when bin packing 
    65  task groups onto Nomad clients. Spark defaults to requesting the minimum of 1 
    66  Mbit/s per task, but you can change this with the 
    67  [spark.nomad.driver.networkMBits](/guides/spark/configuration.html#spark-nomad-driver-networkmbits), 
    68  [spark.nomad.executor.networkMBits](/guides/spark/configuration.html#spark-nomad-executor-networkmbits), and
    69  [spark.nomad.shuffle.networkMBits](/guides/spark/configuration.html#spark-nomad-shuffle-networkmbits) 
    70  properties.
    71  
    72  ## Log rotation
    73  
    74  Nomad performs log rotation on the `stdout` and `stderr` of its tasks. You can 
    75  configure the number number and size of log files it will keep for driver and 
    76  executor task groups using 
    77  [spark.nomad.driver.logMaxFiles](/guides/spark/configuration.html#spark-nomad-driver-logmaxfiles) 
    78  and [spark.nomad.executor.logMaxFiles](/guides/spark/configuration.html#spark-nomad-executor-logmaxfiles).
    79  
    80  ## Next Steps
    81  
    82  Learn how to [dynamically allocate Spark executors](/guides/spark/dynamic.html).