Managing PCI-DSS Compliance and Access to Serverless Features in Azure Databricks

Question

Hello Azure Community,

I am currently using Azure Databricks with PCI-DSS compliance enabled in our workspace, as maintaining stringent security standards is crucial for our organization. However, I've discovered that once PCI-DSS compliance is turned on, it cannot be disabled, and this has impacted our ability to access certain features, such as serverless compute.

My Questions:

• Are there any ways to adjust compliance settings or policies to enable serverless compute while still adhering to PCI-DSS standards?
• Would using a separate, non-compliant workspace for specific tasks be a recommended practice, and if so, how can this be managed effectively alongside our compliant workspace?

Context:

• Region: North Europe
• Current Challenge: Serverless compute is not available due to the PCI-DSS setting, which is essential for our security needs.

Any guidance on managing this balance between compliance and feature flexibility would be greatly appreciated. If you have any insights, best practices, or similar experiences, please share your advice.

Thank you for your support!

Answer 1

Are there any ways to adjust compliance settings or policies to enable serverless compute while still adhering to PCI-DSS standards?

Once PCI-DSS compliance is enabled in an Azure Databricks workspace, it cannot be directly turned off or modified due to its stringent security requirements. The limitation exists to ensure that all configurations, data handling, and access controls remain compliant with PCI-DSS standards. Currently, Azure Databricks restricts access to serverless compute in PCI-DSS-compliant environments because serverless compute does not yet meet all PCI-DSS requirements.

To address this challenge, consider the following approaches:

• Submit feedback to Microsoft Azure via the Azure Feedback Portal or your Azure support representative, requesting PCI-DSS-compliant serverless compute features. This can inform Microsoft's roadmap for enhancing features in compliant environments.
• Instead of serverless compute, evaluate the use of cluster pools or single-node clusters, which can offer cost efficiency and scalability while adhering to PCI-DSS standards. These options might require additional tuning but can be configured securely within your compliant workspace.

Would using a separate, non-compliant workspace for specific tasks be a recommended practice, and if so, how can this be managed effectively alongside our compliant workspace?

Using a separate, non-compliant Azure Databricks workspace for non-PCI-DSS-critical tasks is a common and practical approach. This strategy ensures compliance for sensitive data while providing flexibility for tasks that do not involve PCI-DSS scope. To manage this effectively:

1. Clear Data Segmentation: Establish strict boundaries between the compliant and non-compliant workspaces. Sensitive data subject to PCI-DSS requirements must never be transferred or processed in the non-compliant workspace.
2. Role-Based Access Control (RBAC): Use Azure RBAC and Databricks workspace-level permissions to define who has access to each workspace, ensuring only authorized personnel can access sensitive environments.
3. Data Workflow Management: Utilize tools like Azure Data Factory or Databricks Connect to orchestrate workflows between compliant and non-compliant environments while keeping PCI-DSS data securely in the compliant workspace.
4. Documentation and Monitoring: Maintain comprehensive documentation of data flows, workspace purposes, and access controls to ensure auditors understand the segregation of duties and data handling practices. Use Azure Monitor and Databricks audit logs to track compliance and detect anomalies.

Links to help you :

https://learn.microsoft.com/en-us/azure/databricks/security/privacy/pci

https://learn.microsoft.com/en-us/azure/databricks/security/privacy/security-profile

https://learn.microsoft.com/en-us/azure/databricks/admin/workspace-settings/serverless

https://learn.microsoft.com/en-us/azure/databricks/security/privacy/enhanced-security-compliance

https://www.databricks.com/product/azure/security-and-compliance