github.com/hhrutter/nomad@v0.6.0-rc2.0.20170723054333-80c4b03f0705/website/source/guides/spark/configuration.html.md (about)

     1  ---
     2  layout: "guides"
     3  page_title: "Apache Spark Integration - Configuration Properties"
     4  sidebar_current: "guides-spark-configuration"
     5  description: |-
     6    Comprehensive list of Spark configuration properties.
     7  ---
     8  
     9  # Spark Configuration Properties
    10  
    11  Spark [configuration properties](https://spark.apache.org/docs/latest/configuration.html#available-properties) 
    12  are generally applicable to the Nomad integration. The properties listed below 
    13   are specific to running Spark on Nomad. Configuration properties can be set by 
    14   adding `--conf [property]=[value]` to the `spark-submit` command.
    15  
    16  - `spark.nomad.cluster.expectImmediateScheduling` `(bool: false)` - Specifies 
    17  that `spark-submit` should fail if Nomad is not able to schedule the job 
    18  immediately.
    19  
    20  - `spark.nomad.cluster.monitorUntil` `(string: "submitted"`) - Specifies the 
    21  length of time that `spark-submit` should monitor a Spark application in cluster
    22   mode. When set to `submitted`, `spark-submit` will return as soon as the 
    23   application has been submitted to the Nomad cluster. When set to `scheduled`, 
    24   `spark-submit` will return as soon as the Nomad job has been scheduled. When 
    25   set to `complete`, `spark-submit` will tail the output from the driver process 
    26   and return when the job has completed.
    27  
    28  - `spark.nomad.datacenters` `(string: dynamic)` - Specifies a comma-separated 
    29  list of Nomad datacenters to use. This property defaults to the datacenter of 
    30  the first Nomad server contacted.
    31  
    32  - `spark.nomad.docker.email` `(string: nil)` - Specifies the email address to 
    33  use when downloading the Docker image specified by 
    34  [spark.nomad.dockerImage](#spark.nomad.dockerImage). See the 
    35  [Docker driver authentication](https://www.nomadproject.io/docs/drivers/docker.html#authentication) 
    36  docs for more information.
    37  
    38  -  `spark.nomad.docker.password` `(string: nil)` - Specifies the password to use
    39    when downloading the Docker image specified by 
    40    [spark.nomad.dockerImage](#spark.nomad.dockerImage). See the 
    41  [Docker driver authentication](https://www.nomadproject.io/docs/drivers/docker.html#authentication) 
    42  docs for more information.
    43  
    44  - `spark.nomad.docker.serverAddress` `(string: nil)` - Specifies the server 
    45  address (domain/IP without the protocol) to use when downloading the Docker 
    46  image specified by [spark.nomad.dockerImage](#spark.nomad.dockerImage). Docker 
    47  Hub is used by default. See the 
    48  [Docker driver authentication](https://www.nomadproject.io/docs/drivers/docker.html#authentication) 
    49  docs for more information.
    50  
    51  - `spark.nomad.docker.username` `(string: nil)` - Specifies the username to use
    52   when downloading the Docker image specified by 
    53   [spark.nomad.dockerImage](#spark-nomad-dockerImage). See the 
    54  [Docker driver authentication](https://www.nomadproject.io/docs/drivers/docker.html#authentication) 
    55  docs for more information.
    56  
    57  - `spark.nomad.dockerImage` `(string: nil)` - Specifies the `URL` for the 
    58  [Docker image](https://www.nomadproject.io/docs/drivers/docker.html#image) to 
    59  use to run Spark with Nomad's `docker` driver. When not specified, Nomad's 
    60  `exec` driver will be used instead.
    61  
    62  - `spark.nomad.driver.cpu` `(string: "1000")` - Specifies the CPU in MHz that 
    63  should be reserved for driver tasks.
    64  
    65  - `spark.nomad.driver.logMaxFileSize` `(string: "1m")` - Specifies the maximum 
    66  size by time that Nomad should use for driver task log files.
    67  
    68  - `spark.nomad.driver.logMaxFiles` `(string: "5")` - Specifies the number of log
    69   files that Nomad should keep for driver tasks.
    70  
    71  - `spark.nomad.driver.networkMBits` `(string: "1")` - Specifies the network 
    72  bandwidth that Nomad should allocate to driver tasks.
    73  
    74  - `spark.nomad.driver.retryAttempts` `(string: "5")` - Specifies the number of 
    75  times that Nomad should retry driver task groups upon failure.
    76  
    77  - `spark.nomad.driver.retryDelay` `(string: "15s")` - Specifies the length of 
    78  time that Nomad should wait before retrying driver task groups upon failure.
    79  
    80  - `spark.nomad.driver.retryInterval` `(string: "1d")` - Specifies Nomad's retry 
    81  interval for driver task groups.
    82  
    83  - `spark.nomad.executor.cpu` `(string: "1000")` - Specifies the CPU in MHz that 
    84  should be reserved for executor tasks.
    85  
    86  - `spark.nomad.executor.logMaxFileSize` `(string: "1m")` - Specifies the maximum
    87   size by time that Nomad should use for executor task log files.
    88  
    89  - `spark.nomad.executor.logMaxFiles` `(string: "5")` - Specifies the number of 
    90  log files that Nomad should keep for executor tasks.
    91  
    92  - `spark.nomad.executor.networkMBits` `(string: "1")` - Specifies the network 
    93  bandwidth that Nomad should allocate to executor tasks.
    94  
    95  - `spark.nomad.executor.retryAttempts` `(string: "5")` - Specifies the number of
    96   times that Nomad should retry executor task groups upon failure.
    97  
    98  - `spark.nomad.executor.retryDelay` `(string: "15s")` - Specifies the length of 
    99  time that Nomad should wait before retrying executor task groups upon failure.
   100  
   101  - `spark.nomad.executor.retryInterval` `(string: "1d")` - Specifies Nomad's retry 
   102  interval for executor task groups.
   103  
   104  - `spark.nomad.job` `(string: nil)` - Specifies the Nomad job name.
   105  
   106  - `spark.nomad.job.template` `(string: nil)` - Specifies the path to a JSON file 
   107  containing a Nomad job to use as a template. This can also be set with 
   108  `spark-submit's --nomad-template` parameter.
   109  
   110  - `spark.nomad.priority` `(string: nil)` - Specifies the priority for the 
   111  Nomad job.
   112  
   113  - `spark.nomad.region` `(string: dynamic)` - Specifies the Nomad region to use. 
   114  This property defaults to the region of the first Nomad server contacted.
   115  
   116  - `spark.nomad.shuffle.cpu` `(string: "1000")` - Specifies the CPU in MHz that 
   117  should be reserved for shuffle service tasks.
   118  
   119  - `spark.nomad.shuffle.logMaxFileSize` `(string: "1m")` - Specifies the maximum
   120   size by time that Nomad should use for shuffle service task log files..
   121  
   122  - `spark.nomad.shuffle.logMaxFiles` `(string: "5")` - Specifies the number of 
   123  log files that Nomad should keep for shuffle service tasks.
   124  
   125  - `spark.nomad.shuffle.memory` `(string: "256m")` - Specifies the memory that 
   126  Nomad should allocate for the shuffle service tasks.
   127  
   128  - `spark.nomad.shuffle.networkMBits` `(string: "1")` - Specifies the network 
   129  bandwidth that Nomad should allocate to shuffle service tasks.
   130  
   131  - `spark.nomad.sparkDistribution` `(string: nil)` - Specifies the location of 
   132  the Spark distribution archive file to use.
   133  
   134  - `spark.nomad.tls.caCert` `(string: nil)` - Specifies the path to a `.pem` file
   135   containing the certificate authority that should be used to validate the Nomad 
   136   server's TLS certificate.
   137  
   138  - `spark.nomad.tls.cert` `(string: nil)` - Specifies the path to a `.pem` file 
   139  containing the TLS certificate to present to the Nomad server.
   140  
   141  - `spark.nomad.tls.trustStorePassword` `(string: nil)` - Specifies the path to a
   142   `.pem` file containing the private key corresponding to the certificate in 
   143  [spark.nomad.tls.cert](#spark-nomad-tls-cert).
   144