In Synapse Notebook using pyspark inbuilt functions how to convert country code to country name as description

SaiSekhar, MahasivaRavi (Philadelphia) 60

Hi Team,

Could you please help us that how to change country code to country name (as description) for a column in synapse notebook using pyspark (modules or libraries) could you please share the code.

we have used below but it is not working.
Eg:

import country_converter as coco
from pyspark.sql.types import StringType
from pyspark.sql.functions import udf, lit,col

def country_converter(country):
    if country:
        return coco.convert(country, to='IS02', not_found=None)
    return None

cc_udf = udf(country_converter, StringType())

Smaran Thoomu 19,140 Reputation points Microsoft Vendor

2025-01-17T09:42:08.19+00:00
Hi @SaiSekhar, MahasivaRavi (Philadelphia)
Welcome to Microsoft Q&A platform and thanks for posting your query here.

I understand you're trying to convert country codes to country names in a Synapse Notebook using PySpark. The approach you shared is on the right track, but I recommend trying the following revised implementation.

I have provided an updated code snippet for you to test. It uses the country_converter library and ensures proper handling of country codes. Please try running this code in your Synapse environment:

import country_converter as coco from pyspark.sql.types import StringType from pyspark.sql.functions import udf def country_converter(code): try: return coco.convert(names=code, to='name_short', not_found=None) except: return None cc_udf = udf(country_converter, StringType()) data = [("US",), ("IN",), ("GB",), ("FR",), (None,)] columns = ["country_code"] df = spark.createDataFrame(data, columns) result_df = df.withColumn("country_name", cc_udf(df["country_code"])) result_df.show()

Additionally, there is a related discussion on Stack Overflow that might be helpful: PySpark DataFrame: Convert country names to ISO codes with country_converter. While the example in the post focuses on converting country names to ISO codes, the principles and the library usage are similar.

Please try the solution and let us know if it resolves your issue.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.
SaiSekhar, MahasivaRavi (Philadelphia) 60 Reputation points

2025-01-17T10:48:54.67+00:00

HI @Smaran Thoomu ,

Thanks for the update, but when we are writing into delta file it is throughing an error could you please check.
Error:
e "/opt/spark/python/lib/pyspark.zip/pyspark/cloudpickle/cloudpickle.py", line 649, in subimport import(name) ModuleNotFoundError: No module named 'country_converter'
SaiSekhar, MahasivaRavi (Philadelphia) 60 Reputation points

2025-01-17T10:52:21.3233333+00:00

Hi @Smaran Thoomu , we are getting the below error while writing the file as delta could you please check using the above script. Error: "/opt/spark/python/lib/pyspark.zip/pyspark/cloudpickle/cloudpickle.py", line 649, in subimport import(name) ModuleNotFoundError: No module named 'country_converter'
Smaran Thoomu 19,140 Reputation points Microsoft Vendor

2025-01-17T13:32:59.16+00:00
Hi @SaiSekhar, MahasivaRavi (Philadelphia)

It seems like the error you are encountering, ModuleNotFoundError: No module named 'country_converter', indicates that the country_converter library is not available in the Synapse environment where you're trying to write the data as a Delta file.To resolve this issue, you'll need to ensure that the country_converter library is installed and accessible within your Spark pool or environment.

In Azure Synapse Analytics, you need to install the country_converter library on your Spark pool.

The error you're seeing suggests that the module is missing, so we need to install it in the environment where the notebook is being executed.

If country_converter is not available in the default packages, download the library's .whl file or .tar.gz from PyPI.

In Synapse Studio, go to Manage > Apache Spark Pools > Packages and upload the .whl or .tar.gz file as per below.

Attach the library to your Spark pool and restart the pool.

Use the following test code to verify if the country_converter module is available:

Once the library is installed and the Spark pool is restarted, try executing the notebook again. The error should no longer appear when you're trying to write the data as Delta.

If the issue persists, please share the specifics of your Synapse environment (such as version or custom configurations), and I can assist you further.

Share via

In Synapse Notebook using pyspark inbuilt functions how to convert country code to country name as description

Your answer