Assuming Azure OpenAI assistant does not accpet image input Is that doable to trigger GPT-4 vision from Azure OpenAi assistants and streamline the flow of conversation? not an isolated prompt?

Hi there Mohamed Hussein thanks for using QandA platform i guess, it is possible to integrate GPT-4 Vision into your Azure OpenAI assistant workflow, Azure OpenAI Assistant doesn’t directly handle image inputs at the moment, integrating GPT-4 Vision into the flow is possible using Azure Cognitive Services to process images and Azure Functions to handle the orchestration. this can keep the conversation seamless while using GPT-4's vision capabilities. give it a try If this helps kindly accept the answer thanks much.

Triggering GPT-4 vision from Assistants

Accepted answer

Azar 26,185 Reputation points MVP

2024-11-22T20:04:47.34+00:00

Hi there Mohamed Hussein

thanks for using QandA platform

i guess, it is possible to integrate GPT-4 Vision into your Azure OpenAI assistant workflow,

Azure OpenAI Assistant doesn’t directly handle image inputs at the moment, integrating GPT-4 Vision into the flow is possible using Azure Cognitive Services to process images and Azure Functions to handle the orchestration.

this can keep the conversation seamless while using GPT-4's vision capabilities. give it a try

If this helps kindly accept the answer thanks much.
Please sign in to rate this answer.
YutongTie-MSFT 53,891 Reputation points

2024-11-25T23:43:07.4066667+00:00

Hello Mohemad

Just checking in to see if the below answer provided by Azar helped.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Regards,

Yutong

Mohamed Hussein 610 Reputation points

2024-11-26T00:49:10.86+00:00

Thank you all for your replies, really appreciated

Actually I was asking about some details.

My app layer would receive user's image, and upload it blob Storage? AzOpenAi data storage?

Should my system call the function away from the assistant? or leave an instruction to the assistant by adding - system role msg - eg. If user uploaded an image ( or image link ) trigger X function?

The weather example at docs was text to text, which's quite different than this case.

Will the assistant recognize the output of trubo vision as a feeded knowledge? Or my application layer should manually add another - system type - message e.g this is a chart about AAPL stock in Feb 2024

Later on, if the user asked about this image again, will the assistant automatically remember AAPL stock price in 2024 - which is after the knowledge cutoff? Or how it shall be handled?

YutongTie-MSFT 53,891 Reputation points

2024-11-26T05:55:25.65+00:00

Hello @Mohamed Hussein Thanks for your detailed information, since this question is across different product, I would recommend you raise a support ticket to solve it. Please let us know if you have no support plan, I am happy to enable you a free ticket.

To answer your question generally, I think Blob Storage will be your app end - User sends an image → App stores it in Blob Storage.

Then the App triggers a vision model (e.g., Azure AI Vision/ GTP-4 turbo vision) for processing.

And the vision model outputs text/metadata, which is passed back to the assistant as a new system or user message.

Finally, the follow-up questions are answered by referencing stored metadata in your app layer.

I hope this helps! Please let me know if you need a support ticket.

Regards,

Yutong

Mohamed Hussein 610 Reputation points

2024-11-26T06:11:21.78+00:00

Thanks alot @YutongTie-MSFT

Appreciate your usual support 🙂

YutongTie-MSFT 53,891 Reputation points

2024-11-27T22:51:09.5833333+00:00

Hello @Mohamed Hussein Does my answer help? Please let me know if you need a support ticket or you have more question.

Regards,

Yutong

Azar 26,185 Reputation points MVP

2024-11-29T13:12:17.4633333+00:00

Hi there, checking if this answer helped, if it did kindly accept the answer thanks much.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Triggering GPT-4 vision from Assistants

0 additional answers

Your answer