Internal error while fine tuning Whisper

Question

While trying to fine tuning Whisper Large V2 I get an internal error:
"Model training failedAn internal server error has occurred. Retry after a while. If the issue persists".

I've tried several times and different region but the problem persists. I've also tried with the sample azure data but I still observe the same problem.

Answer 1

Hi Pier Luigi Susini,

Welcome to Microsoft Q&A forum. Thanks for posting your query.

Yes! The "Internal server error" while fine-tuning Whisper Large V2 in Azure OpenAI Service can be caused by several issues, including service availability, quota limits, or backend issues in Azure. Here’s a step-by-step troubleshooting guide to resolve it:

1. Check Azure OpenAI Service Status

Sometimes, the error is due to temporary service issues. Check the Azure status page for OpenAI services:

Visit Azure Status and check if there are any ongoing issues in your selected region.

If there are reported issues, wait until Microsoft resolves them.

2. Ensure Fine-Tuning is Enabled for Your Subscription

Fine-tuning Whisper Large V2 requires explicit approval from Microsoft. You must check if fine-tuning is enabled for your Azure OpenAI resource.

How to Check and Request Access:

Go to Azure OpenAI Studio → Click on your OpenAI resource.

Navigate to "Limits and Quotas" and check if fine-tuning is available for Whisper Large V2.

If fine-tuning is not enabled, raise a support request to Microsoft via Azure Support.

3. Check Quotas and Limits

Even if fine-tuning is enabled, your subscription might have exceeded the allowed limits.

How to Check Quota:

Open Azure Portal → Go to your Azure OpenAI resource.

Click on Usage + quotas.

Look for the fine-tuning token limit (e.g., number of tokens allowed per month).

If you're exceeding the limit, request an increase from Microsoft via Azure Support.

4. Use a Different Region

Since you've tried different regions and still see the issue, only a few regions fully support Whisper fine-tuning.

Recommended Regions for Whisper Fine-Tuning:

East US

West Europe

South Central US

Try fine-tuning in one of these regions.

5. Use Correct Data Format and Parameters

Fine-tuning errors may also be caused by incorrect dataset formatting.

Check Your Training Data

Ensure your dataset follows Azure OpenAI’s Whisper fine-tuning format:

The dataset must be in JSONL format.

Each line must contain a prompt and completion field

The dataset should not exceed 100M tokens.

Verify Training Command

Use the correct fine-tuning command:

Code

6. Retry After Some Time (Rate Limiting Issue)

If too many users are fine-tuning at the same time, Azure might rate-limit your request.

Solution: Wait for a few hours and retry.

Hope this helps, if you still face any issue, please share me your training arguments so that I can deep dive more into the issue.

And also, please refer below link for

Whisper v2 model details and fine-tuning steps:https://huggingface.co/openai/whisper-large-v2

If this answers your query, do click Accept Answer and Yes for was this answer helpful.

Thank you.