SparkSessionExtensions.GetAssemblyInfo(SparkSession, Int32) Method
Definition
Important
Some information relates to prerelease product that may be substantially modified before it’s released. Microsoft makes no warranties, express or implied, with respect to the information provided here.
Get the Microsoft.Spark.Utils.AssemblyInfoProvider.AssemblyInfo for the "Microsoft.Spark" assembly running on the Spark Driver and make a "best effort" attempt in determining the Microsoft.Spark.Utils.AssemblyInfoProvider.AssemblyInfo of "Microsoft.Spark.Worker" assembly on the Spark Executors.
There is no guarantee that a Spark Executor will be run on all the nodes in
a cluster. To increase the likelyhood, the spark conf spark.executor.instances
and the numPartitions settings should be adjusted to a
reasonable number relative to the number of nodes in the Spark cluster.
public static Microsoft.Spark.Sql.DataFrame GetAssemblyInfo (this Microsoft.Spark.Sql.SparkSession session, int numPartitions = 10);static member GetAssemblyInfo : Microsoft.Spark.Sql.SparkSession * int -> Microsoft.Spark.Sql.DataFrame<Extension()>
Public Function GetAssemblyInfo (session As SparkSession, Optional numPartitions As Integer = 10) As DataFrameParameters
- session
- SparkSession
The SparkSession
- numPartitions
- Int32
Number of partitions
Returns
A DataFrame containing the Microsoft.Spark.Utils.AssemblyInfoProvider.AssemblyInfo