Prompt Flow of image with text input in Azure AI Foundry

Yoke Huan Shum 20

Hi, I'm seeking help with creating a prompt flow in Azure AI Foundry.

I'm trying to create a prompt flow that can handle both image and text inputs simultaneously. I was referring to this link do it and using the GPT-4o model. However, I couldn't make it work.

It's showing this error when I test the prompt flow in the chat session.

User's image

Appreciate if anyone can help me with this. Thanks a lot!

navba-MSFT 26,125 Reputation points Microsoft Employee

2024-12-03T04:43:30.0766667+00:00
@Yoke Huan Shum Welcome to Microsoft Q&A Forum, Thank you for posting your query here! .

I have tried following this article to build a chatbot to process images but I am unable to reproduce this error. It worked fine for me:

.

I have not done any modifications or customization to the prompt and kept it very simple.

Questions:

Could you please share the screenshot of your prompt flow ?

Please collect browser HAR trace while reproding this error and share the traces here.

Please confirm if you are using the Azure Open AI GPT4-turbo model (GPT4-Vision).

Do you have any content safety created for this model ?

Instead of using Azure AI Foundry, Could you please use ML Studio and check if that helps ?

Awaiting your reply.
Yoke Huan Shum 20 Reputation points

2024-12-03T05:28:42.27+00:00

@navba-MSFT Thank you for your reply!

I did try the prompt flow without any modification. It's working right now. Thanks!

However, I still couldn't prompt with image and text simultaneously, even though I have changed the input type to 'List' where it should be available for mixed queries.

Do you have any idea where I should modify it to allow both image and text input?

Thanks for the help and looking forward to your reply!
navba-MSFT 26,125 Reputation points Microsoft Employee

2024-12-03T09:36:08.3866667+00:00

@Yoke Huan Shum I am able to reproduce this issue at my end. Even after changing the type as list the chat is still not taking the text with image input.

However our documentation says below:

To enable your chatbot to respond with rich text and images, make the chat output list type. The list should consist of strings (for text) and prompt flow Image objects (for images) in custom order.

I am discussing this internally to know if this is a bug or design behavior. I will keep you posted. Apologies for the inconvenience caused due to this.
Farah Diana Masri 35 Reputation points

2024-12-04T01:07:02.5633333+00:00

Hi @navba-MSFT I am interested to know the answer from Yoke Huan's question also about the list type. Hope you can share it once you know the steps. Thank you.
navba-MSFT 26,125 Reputation points Microsoft Employee

2024-12-04T04:35:59.0633333+00:00

@Yoke Huan Shum @Farah Diana Masri I am discussing this internally. I have reached out to the Product owners. I will keep you posted.

Share via

Prompt Flow of image with text input in Azure AI Foundry

Your answer