Decent results using the Custom Classification model for invoices got worse after increasing the number of training documents

Question

We initially trained a model to detect invoices using 10 documents. We saw decent results, but found that some obvious invoice documents (to a human) resulted in very low confidence levels. We then increased the number of training documents to 99 and the model began producing worse results in the other direction, meaning that documents that were obviously not invoices were classified as invoices with very high confidence. Has anyone else seen this behavior?