Introducing Spark Machine Learning on SQL Server Big Data Clusters
Applies to: 
SQL Server 2019 (15.x)
Important
The Microsoft SQL Server 2019 Big Data Clusters add-on will be retired. Support for SQL Server 2019 Big Data Clusters will end on February 28, 2025. All existing users of SQL Server 2019 with Software Assurance will be fully supported on the platform and the software will continue to be maintained through SQL Server cumulative updates until that time. For more information, see the announcement blog post and Big data options on the Microsoft SQL Server platform.
This article explains how to effectively use Spark for Machine Learning on SQL Server Big Data Clusters.
Spark Machine Learning in SQL Server Big Data Clusters
SQL Server Big Data Clusters enables machine learning scenarios and solutions using different technology stacks: SQL Server Machine Learning Services and Apache Spark ML.
To better understand when to use each technology stack, refer to Machine Learning guide for SQL Server Big Data Clusters. This guide covers Apache Spark ML.
For big data-based machine learning scenarios, the usage of HDFS for big data hosting and Apache Spark ML capabilities is a more cost-effective, scalable, and powerful approach. Yet this is far from an exhaustive list of the possibilities of what can be achieved with Spark Machine Learning, for a complete list of features see: Spark MLlib.
The next section provides a curated list of scenarios and references for Spark in SQL Server Big Data Clusters.
Building blocks for Spark Machine Learning on SQL Server Big Data Clusters
| Learn | Contents | Link |
|---|---|---|
| SQL Server Big Data Clusters runtime for Apache Spark | This will show what's included with each release | SQL Server Big Data Clusters runtime for Apache Spark Guide |
| The Storage Pool | How to store and use HDFS + Spark together to unlock data for machine learning | Introducing the storage pool in SQL Server Big Data Clusters |
| Use notebook-based experiences and your tools of choice | Connect Spark-Livy endpoint using your tools of choice | Submit Spark jobs on SQL Server Big Data Clusters in Azure Data Studio Submit Spark jobs on SQL Server big data cluster in Visual Studio Code Use sparklyr in SQL Server big data cluster |
| How to install extra packages | In the case a package is not provided out-of-the-box, install it | Spark library management |
| How to troubleshoot | In case it breaks | Troubleshoot a pyspark notebookDebug and Diagnose Spark Applications on SQL Server Big Data Clusters in Spark History Server |
| How to submit machine learning batch jobs | Make ML training and batch scoring run using the command line | Submit Spark jobs by using command-line tools |
| How to quickly move data between SQL Server and Spark | Make SQL Server source and/or destination for your Spark ML scenarios. Usage of HDFS is not mandatory | Use the Apache Spark Connector for SQL Server and Azure SQL |
| Spark model operationalization | After training, operationalize using MLeap | Create, export, and score Spark machine learning models on SQL Server Big Data Clusters |
| Data wrangling | Along with Spark's powerful data wrangling capabilities, we ship PROSE | Data Wrangling using PROSE Code Accelerator |
Next steps
For more information, see Introducing SQL Server Big Data Clusters.