github.com/smintz/nomad@v0.8.3/website/source/guides/spark/configuration.html.md (about)

     1  ---
     2  layout: "guides"
     3  page_title: "Apache Spark Integration - Configuration Properties"
     4  sidebar_current: "guides-spark-configuration"
     5  description: |-
     6    Comprehensive list of Spark configuration properties.
     7  ---
     8  
     9  # Spark Configuration Properties
    10  
    11  Spark [configuration properties](https://spark.apache.org/docs/latest/configuration.html#available-properties) 
    12  are generally applicable to the Nomad integration. The properties listed below 
    13   are specific to running Spark on Nomad. Configuration properties can be set by 
    14   adding `--conf [property]=[value]` to the `spark-submit` command.
    15  
    16  - `spark.nomad.authToken` `(string: nil)` - Specifies the secret key of the auth 
    17  token to use when accessing the API. This falls back to the NOMAD_TOKEN environment 
    18  variable. Note that if this configuration setting is set and the cluster deploy 
    19  mode is used, this setting will be propagated to the driver application in the 
    20  job spec. If it is not set and an auth token is taken from the NOMAD_TOKEN 
    21  environment variable, the token will not be propagated to the driver which will 
    22  require the driver to pick up its token from an environment variable.
    23  
    24  - `spark.nomad.cluster.expectImmediateScheduling` `(bool: false)` - Specifies 
    25  that `spark-submit` should fail if Nomad is not able to schedule the job 
    26  immediately.
    27  
    28  - `spark.nomad.cluster.monitorUntil` `(string: "submitted"`) - Specifies the 
    29  length of time that `spark-submit` should monitor a Spark application in cluster
    30   mode. When set to `submitted`, `spark-submit` will return as soon as the 
    31   application has been submitted to the Nomad cluster. When set to `scheduled`, 
    32   `spark-submit` will return as soon as the Nomad job has been scheduled. When 
    33   set to `complete`, `spark-submit` will tail the output from the driver process 
    34   and return when the job has completed.
    35  
    36  - `spark.nomad.datacenters` `(string: dynamic)` - Specifies a comma-separated 
    37  list of Nomad datacenters to use. This property defaults to the datacenter of 
    38  the first Nomad server contacted.
    39  
    40  - `spark.nomad.docker.email` `(string: nil)` - Specifies the email address to 
    41  use when downloading the Docker image specified by 
    42  [spark.nomad.dockerImage](#spark.nomad.dockerImage). See the 
    43  [Docker driver authentication](https://www.nomadproject.io/docs/drivers/docker.html#authentication) 
    44  docs for more information.
    45  
    46  -  `spark.nomad.docker.password` `(string: nil)` - Specifies the password to use
    47    when downloading the Docker image specified by 
    48    [spark.nomad.dockerImage](#spark.nomad.dockerImage). See the 
    49  [Docker driver authentication](https://www.nomadproject.io/docs/drivers/docker.html#authentication) 
    50  docs for more information.
    51  
    52  - `spark.nomad.docker.serverAddress` `(string: nil)` - Specifies the server 
    53  address (domain/IP without the protocol) to use when downloading the Docker 
    54  image specified by [spark.nomad.dockerImage](#spark.nomad.dockerImage). Docker 
    55  Hub is used by default. See the 
    56  [Docker driver authentication](https://www.nomadproject.io/docs/drivers/docker.html#authentication) 
    57  docs for more information.
    58  
    59  - `spark.nomad.docker.username` `(string: nil)` - Specifies the username to use
    60   when downloading the Docker image specified by 
    61   [spark.nomad.dockerImage](#spark-nomad-dockerImage). See the 
    62  [Docker driver authentication](https://www.nomadproject.io/docs/drivers/docker.html#authentication) 
    63  docs for more information.
    64  
    65  - `spark.nomad.dockerImage` `(string: nil)` - Specifies the `URL` for the 
    66  [Docker image](https://www.nomadproject.io/docs/drivers/docker.html#image) to 
    67  use to run Spark with Nomad's `docker` driver. When not specified, Nomad's 
    68  `exec` driver will be used instead.
    69  
    70  - `spark.nomad.driver.cpu` `(string: "1000")` - Specifies the CPU in MHz that 
    71  should be reserved for driver tasks.
    72  
    73  - `spark.nomad.driver.logMaxFileSize` `(string: "1m")` - Specifies the maximum 
    74  size by time that Nomad should use for driver task log files.
    75  
    76  - `spark.nomad.driver.logMaxFiles` `(string: "5")` - Specifies the number of log
    77   files that Nomad should keep for driver tasks.
    78  
    79  - `spark.nomad.driver.networkMBits` `(string: "1")` - Specifies the network 
    80  bandwidth that Nomad should allocate to driver tasks.
    81  
    82  - `spark.nomad.driver.retryAttempts` `(string: "5")` - Specifies the number of 
    83  times that Nomad should retry driver task groups upon failure.
    84  
    85  - `spark.nomad.driver.retryDelay` `(string: "15s")` - Specifies the length of 
    86  time that Nomad should wait before retrying driver task groups upon failure.
    87  
    88  - `spark.nomad.driver.retryInterval` `(string: "1d")` - Specifies Nomad's retry 
    89  interval for driver task groups.
    90  
    91  - `spark.nomad.executor.cpu` `(string: "1000")` - Specifies the CPU in MHz that 
    92  should be reserved for executor tasks.
    93  
    94  - `spark.nomad.executor.logMaxFileSize` `(string: "1m")` - Specifies the maximum
    95   size by time that Nomad should use for executor task log files.
    96  
    97  - `spark.nomad.executor.logMaxFiles` `(string: "5")` - Specifies the number of 
    98  log files that Nomad should keep for executor tasks.
    99  
   100  - `spark.nomad.executor.networkMBits` `(string: "1")` - Specifies the network 
   101  bandwidth that Nomad should allocate to executor tasks.
   102  
   103  - `spark.nomad.executor.retryAttempts` `(string: "5")` - Specifies the number of
   104   times that Nomad should retry executor task groups upon failure.
   105  
   106  - `spark.nomad.executor.retryDelay` `(string: "15s")` - Specifies the length of 
   107  time that Nomad should wait before retrying executor task groups upon failure.
   108  
   109  - `spark.nomad.executor.retryInterval` `(string: "1d")` - Specifies Nomad's retry 
   110  interval for executor task groups.
   111  
   112  - `spark.nomad.job.template` `(string: nil)` - Specifies the path to a JSON file 
   113  containing a Nomad job to use as a template. This can also be set with 
   114  `spark-submit's --nomad-template` parameter.
   115  
   116  - `spark.nomad.namespace` `(string: nil)` - Specifies the namespace to use. This 
   117  falls back first to the NOMAD_NAMESPACE environment variable and then to Nomad's 
   118  default namespace. 
   119  
   120  - `spark.nomad.priority` `(string: nil)` - Specifies the priority for the 
   121  Nomad job.
   122  
   123  - `spark.nomad.region` `(string: dynamic)` - Specifies the Nomad region to use. 
   124  This property defaults to the region of the first Nomad server contacted.
   125  
   126  - `spark.nomad.shuffle.cpu` `(string: "1000")` - Specifies the CPU in MHz that 
   127  should be reserved for shuffle service tasks.
   128  
   129  - `spark.nomad.shuffle.logMaxFileSize` `(string: "1m")` - Specifies the maximum
   130   size by time that Nomad should use for shuffle service task log files..
   131  
   132  - `spark.nomad.shuffle.logMaxFiles` `(string: "5")` - Specifies the number of 
   133  log files that Nomad should keep for shuffle service tasks.
   134  
   135  - `spark.nomad.shuffle.memory` `(string: "256m")` - Specifies the memory that 
   136  Nomad should allocate for the shuffle service tasks.
   137  
   138  - `spark.nomad.shuffle.networkMBits` `(string: "1")` - Specifies the network 
   139  bandwidth that Nomad should allocate to shuffle service tasks.
   140  
   141  - `spark.nomad.sparkDistribution` `(string: nil)` - Specifies the location of 
   142  the Spark distribution archive file to use.
   143  
   144  - `spark.nomad.tls.caCert` `(string: nil)` - Specifies the path to a `.pem` file
   145   containing the certificate authority that should be used to validate the Nomad 
   146   server's TLS certificate.
   147  
   148  - `spark.nomad.tls.cert` `(string: nil)` - Specifies the path to a `.pem` file 
   149  containing the TLS certificate to present to the Nomad server.
   150  
   151  - `spark.nomad.tls.trustStorePassword` `(string: nil)` - Specifies the path to a
   152   `.pem` file containing the private key corresponding to the certificate in 
   153  [spark.nomad.tls.cert](#spark-nomad-tls-cert).
   154