Azure ML prompt flow QnA model works in worskspace but the deployed endpoint does not work

Oussama Chiboub 45

I have been working on a prompt flow QnA chatbot for a while now and it has been working well. Recently i redeployed my model with minor changes and the endpoint does not work anymore: below is the error that i have : "Execution failure in 'lookup': (TypeError) unsupported operand type(s) for +: 'NoneType' and 'int' azure". Keep in mind that i haven't changed anything in the "lookup" node, and i couldn't find where the + operand is being done on the flow.

User's image

The prompt flow model works well in azure ML studio but deploying somehow makes this error.

Thank you for your help ! It's kind of an urgent matter

YutongTie-MSFT 53,211 Reputation points

2024-09-27T18:52:31.9333333+00:00

Hello Oussama Chiboub,

Thanks for reaching out to us, could you please share which region you are working on?
Oussama Chiboub 45 Reputation points

2024-09-27T18:55:48.09+00:00

Hello Yutong ! Thanks for the reply ! My Azure ML studio ressource is in Canada Central.
Oussama Chiboub 45 Reputation points

2024-09-27T18:56:50.8733333+00:00

@YutongTie-MSFT Hello ! thanks for the reply ! My Azure ML studio is hosted in Canada Central

Oussama Chiboub 45

@YutongTie-MSFT FYI : this is the full error code :

 result = context.invoke_tool(node, f, kwargs=kwargs)\n  File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/_core/flow_execution_context.py", line 90, in invoke_tool\n    result = self._invoke_tool_inner(node, f, kwargs)\n  File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/_core/flow_execution_context.py", line 206, in _invoke_tool_inner\n    raise ToolExecutionError(node_name=node_name, module=module) from e\n', 'innerException': {'type': 'TypeError', 'message': "unsupported operand type(s) for +: 'NoneType' and 'int'", 'stackTrace': 'Traceback (most recent call last):\n  File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/_core/flow_execution_context.py", line 182, in _invoke_tool_inner\n    return f(**kwargs)\n  File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/tracing/_trace.py", line 561, in wrapped\n    token_collector.collect_openai_tokens_for_parent_span(span)\n  File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/tracing/_trace.py", line 143, in collect_openai_tokens_for_parent_span\n    merged_tokens = {\n  File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/tracing/_trace.py", line 144, in <dictcomp>\n    key: self._span_id_to_tokens[parent_span_id].get(key, 0) + tokens.get(key, 0)\n', 'innerException': None}}}

Martin Kňažovič 15 Reputation points

2024-09-28T19:23:01.4066667+00:00

I'm experiencing the same issue where everything worked fine while in prompt flow or playground but once deployed - same error as post author.

Prompt flow and everything was setup based on MS Learn activity (https://microsoftlearning.github.io/mslearn-ai-studio/Instructions/04-Use-own-data.html)
Michal Wypych 5 Reputation points Microsoft Employee

2024-09-29T17:51:37.11+00:00

The same is in Sweden Central. I made a everything by the book and the same issue.
Erik 15 Reputation points

2024-09-30T10:41:20.1533333+00:00
A temporary fix for me was to disable tracing:

os.environ['PF_DISABLE_TRACING'] = 'true'

4 answers

YutongTie-MSFT 53,211 Reputation points

2024-09-29T09:11:38.1133333+00:00

Hello everyone,

Thanks for reporting this issue, product team has acknowledged this issue and working on a fix. Will update here again if the issue has any progress.

Regards,

Yutong

-Please kindly accept the answer if you feel helpful to support the community, thanks a lot.
Please sign in to rate this answer.

2 people found this answer helpful.
Development - Clink Digital Studio 10 Reputation points

2024-09-30T10:44:35.11+00:00

I have the same problem and today I have a demo with an important customer. Any news regards to when this issue will be solved ?

YutongTie-MSFT 53,211 Reputation points

2024-09-30T18:24:02.21+00:00

Team is working on a fix, at this moment, please try the workaround to disable tracing and redeploy the flow by setting the env var "PF_DISABLE_TRACING" to true, this can avoid the potential error at this time.

I hope this helps.

Iyarodi Inetimi 0 Reputation points

2024-10-04T11:25:28.7566667+00:00

Interesting...
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
Erik 15 Reputation points

2024-09-30T10:28:44.38+00:00
A temporary fix for me was to turn of tracing by setting an env variable:

os.environ['PF_DISABLE_TRACING'] = 'true'
Please sign in to rate this answer.

2 people found this answer helpful.
Development - Clink Digital Studio 10 Reputation points

2024-09-30T10:45:35.19+00:00

Where can I set this variable ?

Erik 15 Reputation points

2024-09-30T10:57:08.4033333+00:00

I have a python node in my flow where I can set it. If you dont have any python nodes you need to set it in your environment configuration. I guess either in an image or your flow.dag.yaml file

Development - Clink Digital Studio 10 Reputation points

2024-09-30T11:10:42.4133333+00:00

Yes I have, so I can try on the first python node of the flow cause it's always run

Development - Clink Digital Studio 10 Reputation points

2024-09-30T16:33:26.9433333+00:00

It worked for me, thank you.

Miguel Aguilar 0 Reputation points

2024-10-02T12:16:13.97+00:00

Thank you, it worked for me!

Iyarodi Inetimi 0 Reputation points

2024-10-04T11:26:51.89+00:00

Please can you give a step by step if possible? Please

Jeffrey De Meulemeester 30 Reputation points

2024-10-04T21:00:03.59+00:00

I don't see where I can set this?
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
Oussama Chiboub 45 Reputation points

2024-09-30T13:25:01.3033333+00:00

Hi everyone ! I found a fix pour this error : Apparently latest open ai versions cause a problem with prompt flow tracing, so reverting to = < 1.44.1 openai python library in the requirements file fixed the issue for me

https://pypi.org/project/openai/#history
Please sign in to rate this answer.

1 person found this answer helpful.
Iyarodi Inetimi 0 Reputation points

2024-10-04T11:28:25.52+00:00

Thanks for the suggestion. Please can you give a step by step if possible? Please

Iyarodi Inetimi 0 Reputation points

2024-10-04T11:30:22.61+00:00

Thanks for the suggestion. Please can you give a step by step if possible?

DrGriff 0 Reputation points

2024-11-18T16:18:38.4033333+00:00

Needed tracing so had to use this approach. However, when will we know that it's safe to remove this restriction? We don't want to be stuck on this version for ever. @YutongTie-MSFT, will you update this thread when the issue is fully resolved?
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
suwit 0 Reputation points

2024-09-30T04:46:40.9233333+00:00
error:

The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/_core/flow_execution_context.py", line 90, in invoke_tool result = self._invoke_tool_inner(node, f, kwargs) File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/_core/flow_execution_context.py", line 206, in _invoke_tool_inner raise ToolExecutionError(node_name=node_name, module=module) from e promptflow._core._errors.ToolExecutionError: Execution failure in 'opportunities': (TypeError) unsupported operand type(s) for +: 'NoneType' and 'NoneType' 2024-09-30 04:39:01 +0000 22 execution.flow WARNING Failed to calculate metrics due to exception: unsupported operand type(s) for +: 'int' and 'NoneType'.
Please sign in to rate this answer.
Karol Kawalerczyk 0 Reputation points

2024-09-30T08:01:17.0533333+00:00

I have the same problem in Germany West Central, the error is:

{"error":{"code":"UserError","message":"Execution failure in 'answer_the_question_with_context': (TypeError) unsupported operand type(s) for +: 'int' and 'NoneType'"}}

Promptflow playground works fine and endpoint does not.

Kari 5 Reputation points

2024-09-30T08:08:33.56+00:00

We have the same issue. The location of search is centralsweden and ml workspace is on westeurope. Hope, the teams fixes the issue globally.

Erik 15 Reputation points

2024-09-30T09:04:45.44+00:00

I have encountered the same problem in norwayeast
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Azure ML prompt flow QnA model works in worskspace but the deployed endpoint does not work

4 answers

Your answer