When do I use a composed model vs a custom model in azure ai document intelligence studio?

Question

Our company receives invoices from various vendors, each with unique layouts, but they all contain similar key data fields, such as invoice number, line items, and total amounts. We need an automated solution to extract these fields accurately from invoices, regardless of their varying formats

My question is do I just create 1 model and train that model on all of the different pdfs from the different vendors? Or do I create 1 model per vendor and then create a composed model with all of the models per vendor? I feel that if there is 1 model per vendor to keep the models unique per invoice, there will be too many different models.Thoughts?

Answer 1

AFAIK, for your use case, the most suitable approach would depend on the complexity and variability of your invoice layouts. Effectively you might want to consider the following options:

Single model (generalized model)

Pros:

• Easier to manage and scale.
• A well-trained model can generalize across multiple invoice formats.
• Less overhead compared to maintaining multiple models.

Cons:

• If the invoice layouts vary significantly, generalization might be harder.
• Requires a larger, well-annotated training dataset covering all variations.
• Could result in lower accuracy if the layouts are drastically different.

Multiple models (one per vendor) + composed model

Pros:

• Higher accuracy per vendor since each model is tailored to a specific format.
• Easier to troubleshoot errors for a specific vendor.
• Composed models allow automatic routing to the correct model.

Cons:

• Managing and maintaining multiple models is complex.
• Increased training and retraining effort.
• If new vendors are introduced, a new model must be created.

Hybrid approach (composed model for outliers)

A single general model should be your starting point. If accuracy drops for certain vendors, introduce specific models only for those vendors and use a composed model that routes invoices accordingly.

• Use a general model for most invoices.
• For problematic vendors/layouts, train specialized models only where necessary.
• If using Azure Form Recognizer, classify invoices first and then route them to the best-fit model.

Effectively, you might want to:

1. Start with a single model trained on diverse invoices.
2. If the model struggles with certain vendors, create vendor-specific models only where necessary.
3. Use a composed model to auto-route invoices to the right model if needed.

---

If the above response helps answer your question, remember to "Accept Answer" so that others in the community facing similar issues can easily find the solution. Your contribution is highly appreciated.

hth

Marcin