Custom model misses a lot of check boxes

Uzi Livneh 0

Hello,

We're trying to train with 'Document Intelligence' a custom model that will capture the data from a form that has table with 3 columns of check-boxes - 'Daily', 'PRN' or 'Test Requested'. We try to create 'Fixed' table with these three "Selectionmark" columns ('Daily', 'PRN' or 'Test Requested') , but we got very poor results - and every time we run the custom model a very big part of the boxes are not captured at all (so some boxes are captured, and some are not). Even if we run a form that was part of the training data set - and in the training all the boxes were entered - it still can't recognize all of them.

SriLakshmi C 505 Reputation points Microsoft Vendor

2024-09-16T10:38:05.31+00:00

Hello Uzi Livneh,

Welcome to Microsoft Q&A forum!

As I understand that you are experiencing significant challenges with the custom model used in Document Intelligence for capturing checkbox data from a form with a table of three columns: ‘Daily’, ‘PRN’, and ‘Test Requested’. You noted that while some checkboxes are captured, a sizeable portion remains unrecognized even in forms that were part of the training dataset.

Ensure that the custom model is trained on a diverse and sufficient number of forms, especially focusing on forms that have variations in checkbox placement and alignment.

When labeling checkboxes in the training data, make sure the checkboxes are properly marked, and adjust the table layout structure to correctly map each column (Daily, PRN, Test Requested) in the model.

If the table structure is too complex, simplifying the table or using more detailed labels for each checkbox in the training data can help improve detection.

please check the document Custom document models.

Do let me know if that helps or have any other queries.
Uzi Livneh 0 Reputation points

2024-09-16T13:38:40.92+00:00

Thank you for your reply

Unfortunately, retraining, adding more samples and even changing tables structures didn't improve the model significantly. It seems like the main issue is with checkboxes that have blue color - in the samples we train there're two types of checkboxes - black and blue. The black checkboxes are mostly ok. But the blue checkboxes, although there are visible and we can check them when training, are not detected when we test (even checkboxes that we marked in the training samples, are sometimes not detected when we test with exactly the same sample).
Uzi Livneh 0 Reputation points

2024-09-16T13:46:53.29+00:00

Thank you for your reply @SriLakshmi C . Unfortunately adding more samples, improving quality of the samples and simplifying tables didn't help (we tried all). Seems like the main problem issues from the colour of the checkboxes. The form we train has black and blue checkboxes. The black checkboxes are picked up mostly ok when we test, but the blue check boxes are not picked up, although they are visible in the Layout. Sometimes blue check boxes that we marked as part of the training are not picked-up when we test, even if it’s exactly the same form that was used for training.
SriLakshmi C 505 Reputation points Microsoft Vendor

2024-09-17T05:36:07.67+00:00

Hi Uzi Livneh,

Following up to see if the above response was helpful. Thank you!
Uzi Livneh 0 Reputation points

2024-09-17T07:26:28.1066667+00:00

Thank you for your reply @SriLakshmi C (Quadrant Resource LLC) . Unfortunately adding more samples, improving quality of the samples and simplifying tables didn't help (we tried all). Seems like the main problem issues from the colour of the checkboxes. The form we train has black and blue checkboxes. The black checkboxes are picked up mostly ok when we test, but the blue check boxes are not picked up, although they are visible in the Layout. Sometimes blue check boxes that we marked as part of the training are not picked-up when we test, even if it’s exactly the same form that was used for training.
SriLakshmi C 505 Reputation points Microsoft Vendor

2024-09-17T14:10:58.5666667+00:00

Hello Uzi Livneh,

Thank you for your follow up query.

It seems that the challenge with detecting blue checkboxes may stem from issues related to color contrast or the OCR engine’s interpretation of different colors. To address this, consider enhancing your image preprocessing techniques by adjusting contrast, which can help the model focus on the shapes rather than the colors. Additionally, augment your training dataset with more examples of blue checkboxes, including a range of shades and intensities, to improve the model’s ability to recognize them. Ensure that the checkboxes are correctly labeled in your training data, and explore any available settings in the OCR engine that might enhance color recognition. Fine-tuning the model’s parameters and verifying the quality of your training data can also help improve performance.

I tried to repro the issue able to detect the checkboxes.

I hope you understand, Thank you!
Uzi Livneh 0 Reputation points

2024-09-17T19:29:14.7333333+00:00

Thank you for your reply @SriLakshmi C (Quadrant Resource LLC).

We'll try to get more samples, although it's not easy in this case.

We'll try also to play with contrast - and I'll update how it went.
SriLakshmi C 505 Reputation points Microsoft Vendor

2024-09-18T11:18:56.7466667+00:00

Hi Uzi Livneh,

Thank you for your response and please update how it went.

Share via

Custom model misses a lot of check boxes

Your answer