Azure identity package issue after updating pyspark from 3.3 to 3.4.1

Question

I've updating my PySpark Synapse project to use 3.4.1 Spark instead of 3.3.

After upgrade, it works locally but fails on Synapse with AttributeError: module 'x509' has no attribute 'VerifiedClient'.

Bottom of call stack:

  File "/home/trusted-service-user/cluster-env/env/lib/python3.10/site-packages/azure/identity/_credentials/__init__.py", line 5, in <module>
    from .authorization_code import AuthorizationCodeCredential
  File "/home/trusted-service-user/cluster-env/env/lib/python3.10/site-packages/azure/identity/_credentials/authorization_code.py", line 9, in <module>
    from .._internal.aad_client import AadClient
  File "/home/trusted-service-user/cluster-env/env/lib/python3.10/site-packages/azure/identity/_internal/__init__.py", line 5, in <module>
    from .aad_client import AadClient
  File "/home/trusted-service-user/cluster-env/env/lib/python3.10/site-packages/azure/identity/_internal/aad_client.py", line 11, in <module>
    from .aad_client_base import AadClientBase
  File "/home/trusted-service-user/cluster-env/env/lib/python3.10/site-packages/azure/identity/_internal/aad_client_base.py", line 20, in <module>
    from .aadclient_certificate import AadClientCertificate
  File "/home/trusted-service-user/cluster-env/env/lib/python3.10/site-packages/azure/identity/_internal/aadclient_certificate.py", line 7, in <module>
    from cryptography import x509
  File "/home/trusted-service-user/cluster-env/env/lib/python3.10/site-packages/cryptography/x509/__init__.py", line 7, in <module>
    from cryptography.x509 import certificate_transparency, verification
  File "/home/trusted-service-user/cluster-env/env/lib/python3.10/site-packages/cryptography/x509/verification.py", line 24, in <module>
    VerifiedClient = rust_x509.VerifiedClient
AttributeError: module 'x509' has no attribute 'VerifiedClient'

I've checked recent changes both in azure-identity and cryptography and found nothing suspicious. Do you have any ideas what is worth to check?

Answer 1

Have you checked if the versions of azure-identity and cryptography you are using are compatible with each other ?

Look for any known issues or bug reports in the GitHub repositories for azure-identity and cryptography :

• azure-identity GitHub Issues
• cryptography GitHub Issues

Or simply try reinstalling the libraries to make sure there are no corrupted files.