SparkConf Class
Definition
Important
Some information relates to prerelease product that may be substantially modified before it’s released. Microsoft makes no warranties, express or implied, with respect to the information provided here.
Configuration for a Spark application. Used to set various Spark parameters as key-value pairs.
public sealed class SparkConftype SparkConf = classPublic NotInheritable Class SparkConf- Inheritance
-
SparkConf
Remarks
Note that once a SparkConf object is passed to Spark, it is cloned and can no longer be modified by the user. Spark does not support modifying the configuration at runtime.
Constructors
| SparkConf(Boolean) |
Constructor. |
Methods
| Get(String, String) |
Get a string parameter value, falling back to a default if not set |
| GetAll() |
Get all parameters as a list of pairs. |
| GetInt(String, Int32) |
Get a int parameter value, falling back to a default if not set |
| Set(String, String) |
Set the value of a string config |
| SetAppName(String) |
Set a name for your application. Shown in the Spark web UI. |
| SetMaster(String) |
The master URL to connect to, such as "local" to run locally with one thread, "local[4]" to run locally with 4 cores, or "spark://master:7077" to run on a Spark standalone cluster. |
| SetSparkHome(String) |
Set the location where Spark is installed on worker nodes. |