Why does the latest version of the Microsoft Document Intelligence API (2024-11-30) seem moderately worse than the previous version (the July preview)?

Turtle 0

I have created a program that utilizes the JSON results generated using the document intelligence model API (using the receipt model) for business related purposes. For several months I was using the July preview version of the API and was very impressed with the amount of correct data.

However after updating my npm packages (and making the necessary changes in order to use the 2024-11-30 version (1.0.0 from 1.0.0-beta.3), I notice that some information that I am getting using the same PDF files is now somewhat different - in a negative way (e.g. more letters/numbers read inaccurately, less fields populated in the finished JSON files, etc.).

Is this just because the API is currently sitting at version 1.0.0 and therefore needs some time to mature? Or are there other issues at play such as model overfitting - leading to degradation of performance - which were absent in previous versions of the API?

As of right now I have reverted to using the July preview API version along with the older NPM package which (beta-3) which still allows its use. However, this is only a temporary solution since I am certain it will be deprecated at some point.

Are there any others who have noticed a similar phenomenon? And if so, are there any ways to mitigate these issues?

Casey 0 Reputation points

2025-01-27T11:08:33.28+00:00

I am seeing the same behavior. It's not a lot, but it seems to be a 3-5% worse performance on populating fields and quality of OCR.
santoshkc 11,800 Reputation points Microsoft Vendor

2025-01-27T11:15:40.46+00:00

Hi @Turtle,

Thank you for reaching out to Microsoft Q&A forum!

The differences you're seeing in the 2024-11-30 version of the Azure Document Intelligence API could be due to adjustments in the model to improve general performance, which might have unintentionally impacted how well it handles specific cases like your documents. Newer versions often aim for broader accuracy, but this can sometimes cause slight trade-offs for certain document types or formats. It's not uncommon for a newly released stable version to feel different compared to a well-tuned preview release that you were accustomed to.

Reverting to the July preview version is a practical temporary solution, especially if it better meets your needs right now. As the API evolves, it’s likely that these differences will balance out over time. For the long term, transitioning to the latest version with a few tweaks in your program to handle any variations will ensure you're ready for future updates while maintaining the accuracy you need.

Thank you for your understanding.
Casey 0 Reputation points

2025-01-27T11:23:35.8133333+00:00

@SantoshK I guess I'm confused what use case they are supposed to cover if "receipts" model gets worse on Walmart, Target and the like. Like where is the general performance improvement to be seen if the model is worse than the preview on both online (digital) receipts from those places AND scanned paper receipts? That would make sense if we were using a more generic model, but the model in question is specifically the receipts/invoices models.

And the solution for us at least seems to be "get a human to do the thing that the old API did" if we had to make the switch. I mean we already have to have a human check a decent bit, but obviously going regression on a update isn't ideal.

Put it this way, we handle 100s of thousands of receipts and invoices for dozens of clients from hundreds of retailers and wholesalers. If the receipt/invoice models are worse on our system, they are worse on pretty much everyones. Doesn't mean the check or some other model didn't get better, but the receipt/invoice one got worse.

How long until the August preview model is removed? Prior models (including the August one) were very obviously across the board improvements on older models. This one is not, so I guess I'm afraid of what it means for my small business that built it's tech stack around document intelligence if the models regression isn't noticed/were to get worse.
santoshkc 11,800 Reputation points Microsoft Vendor

2025-01-27T15:07:23.6733333+00:00

Hi Casey,

It’s understandable to be concerned when the latest update to the receipts and invoices model doesn’t meet expectations, especially for major retailers like Walmart and Target. While updates are meant to improve general performance, they can sometimes impact specific use cases.

If the current model isn’t working as well, it might be worth exploring a custom extraction model tailored to your needs. Custom models can be trained to better handle the specific types of receipts and invoices you work with. The August preview model may still be available for a while, but eventually, newer versions will take over. In the meantime, using the older version or a custom model could provide a more reliable solution as the new model continues to evolve.Thank you.
Casey 0 Reputation points

2025-01-28T12:09:47.2533333+00:00

@santoshkc And I am still waiting to hear what specific use case would be impacted that is suddenly better? Like the new build is worse on literally every major retailers receipts and invoices, so where did it improve for that trade off? I'm not saying "it's better on some and worse on others" our use case is literally the vast majority of major and a huge % of minor receipts used in America. Did someone train the receipt model on data out of Bangladesh? Because otherwise it makes no sense that it would regress on LITERALLY it's use case. We aren't using it for anything esoteric here, it's literally basic info that is built into the model has gotten worse.

Like is there somewhere to report this that isn't going to just brush off what is effectively a bug report for the new model?
santoshkc 11,800 Reputation points Microsoft Vendor

2025-01-28T12:45:56.6833333+00:00

Hi @Turtle,

Did you got any chance to check the above response?

1 answer

Turtle 0 Reputation points

2025-01-28T15:24:33.8666667+00:00

@santoshkc Yes i have had a chance to check the answers above. I think that for the time being I will continue to use the preview version and in the mean time hope that the newer model irons out any issues that it is having so that when I have to switch to the newer API version there will be a much smoother transition on my end.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Why does the latest version of the Microsoft Document Intelligence API (2024-11-30) seem moderately worse than the previous version (the July preview)?

1 answer

Your answer