Host Restarts in Azure Function Triggered by EventHub, Causing Data Loss

Jusuf Ramic 0

Hello,

I'm currently working with a ~v4 Azure Function running on Linux, implemented in python 3.9 (using v2 programming model), which is triggered by EventHub. The function processes incoming payloads, performs transformations, and writes the data into an SQL Server database by executing stored procedures, using pyodbc library.

Issue: Occasionally, the function fails in such a way that it restarts the host, bypassing the retry functionality. Looking at Application Insights, the failed invocation includes the message "Invocation cancelled - exiting retry loop". Sometimes this happens even after the function executes all steps successfully and prints the final log. This causes data loss when the failure occurs before the SQL Server procedure execution, since once the host restarts, the same event is not reprocessed. Of course, if the failure happens after the procedure execution, no data loss is observed.

Details:

Runtime: Python 3.9
Azure Function: written using v2 programming model
App Service Plan: B2: 1
Trigger: EventHub
Target: SQL Server (using pyodbc to execute stored procedures)
The function has retry logic in place, but host restarts circumvent this mechanism, leading to lost events.
The failures occur during increased load, and is most likely related to database transactions, even though sometimes it occurs after the transaction is successfully completed
Looking at "Diagnose and solve problems" section seems to point to error 139 (segmentation fault) - python exited with error 139 (0x8B)
The failures are always accompanied by the following trace in app insights: Exceeded language worker retry restart count for runtime: python. Shutting down and proactively recycling the Functions Host to recover.

1 answer

Ryan Hill 28,091 Microsoft Employee

Hi @Jusuf Ramic.

The python 139 error, there's a troubleshooting guide that you can take a look at. See Troubleshoot: python exited with code 139 for steps you can take to mitigate the issue. If none of them don't help, please do let me know.

Jusuf Ramic 0

Hi Ryan,

thank you for the response. Yes, I have tried the options that are relevant. To be more specific:

A regression from third-party packages - the only libraries in my requirements.txt are pyodbc and azure-functions , which are not pinned to a specific version, so the latest version should be installed. pyodbc has not had a newer version come out in the time since February, so it cannot be the issue.
Unpickling from a malformed .pkl file - this is not relevant to my case since there is no unpickling or reading of any files

Pyodbc connection collision - this is the only one that could be relevant in my opinion, and I did try this suggestion to implement singleton pattern, but it still crashes during increased traffic of incoming events. It never happened during single, isolated events. This is how it is implemented:

  class DBService:
      def __init__(self, db_host=None, db_name=None, db_user=None, db_password=None):
          self.db_host = f"tcp:{db_host}" if db_host else None
          self.db_name = db_name
          self.db_user = db_user
          self.db_password = db_password
          self.connection = None
     
      def connect(self, driver_version=17):
          if self.connection is None:
              connection_str = (f"DRIVER={{ODBC Driver {driver_version} for SQL Server}};"
                                f"SERVER={self.db_host};"
                                f"DATABASE={self.db_name};"
                                f"UID={self.db_user};"
                                f"PWD={self.db_password}")
              self.connection = pyodbc.connect(connection_str,autocommit=True)
   
      def close(self):
          if self.connection is not None:
              self.connection.close()
              self.connection = None
   
      def execute_query(self, query_str, params=None):
          self.connect()
          try:
  			with self.connection.cursor() as cursor:
  	            if params is not None:
  	                self.cursor.execute(query_str, params)	         
  	            else:
  	                self.cursor.execute(query_str)  
  				
  				while cursor.nextset():
  					logger.info(cursor.messages)	             
          except pyodbc.Error as e:
              logger.error(f"Pyodbc error {type(e).__name__} occured: {str(e)}.")
              self.connection.rollback()
              raise
          finally:
              self.close()
    
  class SingletonDBService(DBService):
      _instance = None
   
      def __new__(cls, *args, **kwargs):
          if not cls._instance:
              cls._instance = super().__new__(cls)
          return cls._instance

In the function app code, I will receive events, process them and construct SQL query and parameters, instantiate SingletonDBService, and then run execute_query() method. Sometimes, it will fail after I get the messages from the database that the query has run successfully (from the line logger.info(cursor.messages)), in which case there would be no data loss, however the problem is the other case when it happens before.

Jusuf Ramic 0

Sorry, I don't see a way to edit my previous answer to your comment, but there is a small mistake in the code I wrote here, let me just correct it so that we do not focus on the wrong thing. The execute_query method should be:

def execute_query(self, query_str, params=None):         
	self.connect()         
	try: 
		with self.connection.cursor() as cursor:             
			if params is not None:                 
				cursor.execute(query_str, params)                     
			else:                 
				cursor.execute(query_str)   
	
			while cursor.nextset(): 
				logger.info(cursor.messages)                       
	except pyodbc.Error as e:             
		logger.error(f"Pyodbc error {type(e).__name__} occured: {str(e)}.")             	self.connection.rollback()             
		raise         
	finally:             
		self.close()

Share via

Host Restarts in Azure Function Triggered by EventHub, Causing Data Loss

1 answer

Your answer