github.com/smintz/nomad@v0.8.3/website/source/guides/spark/configuration.html.md (about) 1 --- 2 layout: "guides" 3 page_title: "Apache Spark Integration - Configuration Properties" 4 sidebar_current: "guides-spark-configuration" 5 description: |- 6 Comprehensive list of Spark configuration properties. 7 --- 8 9 # Spark Configuration Properties 10 11 Spark [configuration properties](https://spark.apache.org/docs/latest/configuration.html#available-properties) 12 are generally applicable to the Nomad integration. The properties listed below 13 are specific to running Spark on Nomad. Configuration properties can be set by 14 adding `--conf [property]=[value]` to the `spark-submit` command. 15 16 - `spark.nomad.authToken` `(string: nil)` - Specifies the secret key of the auth 17 token to use when accessing the API. This falls back to the NOMAD_TOKEN environment 18 variable. Note that if this configuration setting is set and the cluster deploy 19 mode is used, this setting will be propagated to the driver application in the 20 job spec. If it is not set and an auth token is taken from the NOMAD_TOKEN 21 environment variable, the token will not be propagated to the driver which will 22 require the driver to pick up its token from an environment variable. 23 24 - `spark.nomad.cluster.expectImmediateScheduling` `(bool: false)` - Specifies 25 that `spark-submit` should fail if Nomad is not able to schedule the job 26 immediately. 27 28 - `spark.nomad.cluster.monitorUntil` `(string: "submitted"`) - Specifies the 29 length of time that `spark-submit` should monitor a Spark application in cluster 30 mode. When set to `submitted`, `spark-submit` will return as soon as the 31 application has been submitted to the Nomad cluster. When set to `scheduled`, 32 `spark-submit` will return as soon as the Nomad job has been scheduled. When 33 set to `complete`, `spark-submit` will tail the output from the driver process 34 and return when the job has completed. 35 36 - `spark.nomad.datacenters` `(string: dynamic)` - Specifies a comma-separated 37 list of Nomad datacenters to use. This property defaults to the datacenter of 38 the first Nomad server contacted. 39 40 - `spark.nomad.docker.email` `(string: nil)` - Specifies the email address to 41 use when downloading the Docker image specified by 42 [spark.nomad.dockerImage](#spark.nomad.dockerImage). See the 43 [Docker driver authentication](https://www.nomadproject.io/docs/drivers/docker.html#authentication) 44 docs for more information. 45 46 - `spark.nomad.docker.password` `(string: nil)` - Specifies the password to use 47 when downloading the Docker image specified by 48 [spark.nomad.dockerImage](#spark.nomad.dockerImage). See the 49 [Docker driver authentication](https://www.nomadproject.io/docs/drivers/docker.html#authentication) 50 docs for more information. 51 52 - `spark.nomad.docker.serverAddress` `(string: nil)` - Specifies the server 53 address (domain/IP without the protocol) to use when downloading the Docker 54 image specified by [spark.nomad.dockerImage](#spark.nomad.dockerImage). Docker 55 Hub is used by default. See the 56 [Docker driver authentication](https://www.nomadproject.io/docs/drivers/docker.html#authentication) 57 docs for more information. 58 59 - `spark.nomad.docker.username` `(string: nil)` - Specifies the username to use 60 when downloading the Docker image specified by 61 [spark.nomad.dockerImage](#spark-nomad-dockerImage). See the 62 [Docker driver authentication](https://www.nomadproject.io/docs/drivers/docker.html#authentication) 63 docs for more information. 64 65 - `spark.nomad.dockerImage` `(string: nil)` - Specifies the `URL` for the 66 [Docker image](https://www.nomadproject.io/docs/drivers/docker.html#image) to 67 use to run Spark with Nomad's `docker` driver. When not specified, Nomad's 68 `exec` driver will be used instead. 69 70 - `spark.nomad.driver.cpu` `(string: "1000")` - Specifies the CPU in MHz that 71 should be reserved for driver tasks. 72 73 - `spark.nomad.driver.logMaxFileSize` `(string: "1m")` - Specifies the maximum 74 size by time that Nomad should use for driver task log files. 75 76 - `spark.nomad.driver.logMaxFiles` `(string: "5")` - Specifies the number of log 77 files that Nomad should keep for driver tasks. 78 79 - `spark.nomad.driver.networkMBits` `(string: "1")` - Specifies the network 80 bandwidth that Nomad should allocate to driver tasks. 81 82 - `spark.nomad.driver.retryAttempts` `(string: "5")` - Specifies the number of 83 times that Nomad should retry driver task groups upon failure. 84 85 - `spark.nomad.driver.retryDelay` `(string: "15s")` - Specifies the length of 86 time that Nomad should wait before retrying driver task groups upon failure. 87 88 - `spark.nomad.driver.retryInterval` `(string: "1d")` - Specifies Nomad's retry 89 interval for driver task groups. 90 91 - `spark.nomad.executor.cpu` `(string: "1000")` - Specifies the CPU in MHz that 92 should be reserved for executor tasks. 93 94 - `spark.nomad.executor.logMaxFileSize` `(string: "1m")` - Specifies the maximum 95 size by time that Nomad should use for executor task log files. 96 97 - `spark.nomad.executor.logMaxFiles` `(string: "5")` - Specifies the number of 98 log files that Nomad should keep for executor tasks. 99 100 - `spark.nomad.executor.networkMBits` `(string: "1")` - Specifies the network 101 bandwidth that Nomad should allocate to executor tasks. 102 103 - `spark.nomad.executor.retryAttempts` `(string: "5")` - Specifies the number of 104 times that Nomad should retry executor task groups upon failure. 105 106 - `spark.nomad.executor.retryDelay` `(string: "15s")` - Specifies the length of 107 time that Nomad should wait before retrying executor task groups upon failure. 108 109 - `spark.nomad.executor.retryInterval` `(string: "1d")` - Specifies Nomad's retry 110 interval for executor task groups. 111 112 - `spark.nomad.job.template` `(string: nil)` - Specifies the path to a JSON file 113 containing a Nomad job to use as a template. This can also be set with 114 `spark-submit's --nomad-template` parameter. 115 116 - `spark.nomad.namespace` `(string: nil)` - Specifies the namespace to use. This 117 falls back first to the NOMAD_NAMESPACE environment variable and then to Nomad's 118 default namespace. 119 120 - `spark.nomad.priority` `(string: nil)` - Specifies the priority for the 121 Nomad job. 122 123 - `spark.nomad.region` `(string: dynamic)` - Specifies the Nomad region to use. 124 This property defaults to the region of the first Nomad server contacted. 125 126 - `spark.nomad.shuffle.cpu` `(string: "1000")` - Specifies the CPU in MHz that 127 should be reserved for shuffle service tasks. 128 129 - `spark.nomad.shuffle.logMaxFileSize` `(string: "1m")` - Specifies the maximum 130 size by time that Nomad should use for shuffle service task log files.. 131 132 - `spark.nomad.shuffle.logMaxFiles` `(string: "5")` - Specifies the number of 133 log files that Nomad should keep for shuffle service tasks. 134 135 - `spark.nomad.shuffle.memory` `(string: "256m")` - Specifies the memory that 136 Nomad should allocate for the shuffle service tasks. 137 138 - `spark.nomad.shuffle.networkMBits` `(string: "1")` - Specifies the network 139 bandwidth that Nomad should allocate to shuffle service tasks. 140 141 - `spark.nomad.sparkDistribution` `(string: nil)` - Specifies the location of 142 the Spark distribution archive file to use. 143 144 - `spark.nomad.tls.caCert` `(string: nil)` - Specifies the path to a `.pem` file 145 containing the certificate authority that should be used to validate the Nomad 146 server's TLS certificate. 147 148 - `spark.nomad.tls.cert` `(string: nil)` - Specifies the path to a `.pem` file 149 containing the TLS certificate to present to the Nomad server. 150 151 - `spark.nomad.tls.trustStorePassword` `(string: nil)` - Specifies the path to a 152 `.pem` file containing the private key corresponding to the certificate in 153 [spark.nomad.tls.cert](#spark-nomad-tls-cert). 154