Azure OpenAI Whisper hallucinates source audio language

Shamus Sim 0

Hi,

I have been experimenting with the Whisper service via Azure OpenAI services.

There are alot of cases where the audio language send to the API is (clealry) English but it comes out as a Malay(Bahasa Malaysia) text, despite clearly using the transcription API and not the translation API.

Sample prompt and API call

            `transcription = client_tts.audio.transcriptions.create(`
````                    model="whisper",`

`                    file=audio_file,`

`                    prompt="""Transcribe the following audio file, it is taken in medical clinic setting, `

`                    return the original language and the transcription. Only return English, Bahasa Malaysia, and Mandarin language"""`

`                )`

SriLakshmi C 2,845 Reputation points Microsoft Vendor

2025-02-27T12:04:28.5533333+00:00

Hello Shamus Sim,

Greetings and Welcome to Microsoft Q&A!

I understand that you're encountering an issue with Whisper's language detection incorrectly identifying the source language. Please try these steps:

Try explicitly specifying the language in your API request by setting the "language" parameter to "en" for English.

Ensure that the audio quality is clear, as background noise or accents can sometimes affect detection accuracy. If the problem persists, consider testing with different audio samples and checking if the issue is consistent across multiple requests.

Try segmenting the audio into smaller parts before sending it to the API. This can help reduce errors caused by long or complex audio inputs.

Experiment with different Whisper models, as some versions may handle language detection and transcription more accurately than others. Testing various approaches can help identify the best configuration for your specific use case.

I hope you understand. And, if you have any further query do let us know.

Thank you!

Share via

Azure OpenAI Whisper hallucinates source audio language

Your answer