Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,914 questions
I am getting Negsignal.SIGKILL Airflow
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(32, 73%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECL%3C/text%3E%3C/svg%3E)
Chirag Lalwani
0
Reputation points
I have multiple integration running but for one integration when I extract zip file size of zip is 2-3 gbs, extract and upload each file to our storage.It shows Negsignal.SIGKILL. It was running fine a month ago but now its this error. The file is same always where it fails.
I am using Airflow for Azure Data Factory. When I run it like 1 more time it will get successfully run It seems to be issue with resources but I cant don’t know how much resouces airflow is using and how to increase it.
Is there a way to increase resources in DAG or will adding multiple workers solve the issue?or will I had to add concurrency in case of multiple works.
Dag code is simple we are using PythonOperator
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 65%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECB%3C/text%3E%3C/svg%3E)
Chandra Boorla 3,460 Reputation points Microsoft Vendor2024-11-22T08:18:00.9833333+00:00 Greetings & Welcome to Microsoft Q&A forum! Thanks for posting your query!
The Negsignal.SIGKILL error you are encountering in Airflow typically indicates that the process was terminated by the operating system, often due to resource constraints such as memory limits. Since you mentioned that the issue occurs with a large zip file (2-3 GB) and that it runs successfully upon retry, it does suggest that the resources allocated to your Airflow task may not be sufficient for the size of the data being processed.
Here are some troubleshooting steps that might help you in resolving the issue:
I am using Airflow for Azure Data Factory. When I run it like 1 more time it will get successfully run It seems to be issue with resources but I cant don’t know how much resouces airflow is using and how to increase it.
For more details, please refer: Workflow Orchestration Manager in Azure Data Factory offers a range of powerful features.
Is there a way to increase resources in DAG or will adding multiple workers solve the issue?or will I had to add concurrency in case of multiple works.
Yes, you can increase the resources allocated to your DAG by adding more workers or increasing the concurrency. Adding more workers will increase the number of parallel tasks that can be executed, while increasing the concurrency will allow more tasks to be executed simultaneously.
To add more workers, you can increase the value of the worker_count parameter in your Airflow configuration file. Similarly, to increase the concurrency, you can increase the value of the parallelism parameter in your DAG definition.
However, keep in mind that adding more workers or increasing the concurrency will also increase the resource utilization and cost. Therefore, it is important to optimize your DAG code to reduce the memory footprint and minimize the resource usage. Since your DAG code is simple and you are using PythonOperator, you can try optimizing your Python code to reduce the memory usage.
For additional information, please refer: https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html
I hope this information helps. Please do let us know if you have any further queries.
Thank you.
Sign in to comment
Sign in to answer