AI Search Indexer started failing with field mapping error

Question

Starting on 02/24/2025 around 7pm EST, a set of search indexers in one of our AI Search services started failing. These indexers previously worked without issue, and are structurally identical with other indexers we have on other AI Search resources that are working correctly.

The indexer fails with the message:

Field mapping specifies target field 'filename' that is not present in the target index

These are datasource/indexer/skillset/index setups that were created by using the Azure Blob storage option for RAG on the AI Foundry. The indexers in question do have a fieldMappings section that looks like this:

  "fieldMappings": [
    {
      "sourceFieldName": "metadata_storage_name",
      "targetFieldName": "filename",
      "mappingFunction": null
    }
  ],

There is no filename field in the index, but that is the case for all of our other indexer setups that are still working - I am still figuring out how all of this works, but it would appear that the skillset chunks and vectorizes the input documents from the blob storage container and projects those intermediate chunks out to the actual index fields.

For one of our indexers that was failing, I cleared the fieldMappings section and re-ran the indexer. After doing that, it succeeded, although I am seeing that it indexed some of the new files in the blob storage, but also ignored some PDFs which it should have processed. I've checked on the document size limitations, but these are very small PDFs and previously when I have hit that problem, there have been errors or warnings listed for the indexer run result

I cannot find a reason why this indexer used to work, and then stopped working; we made no changes to any of the indexer resources. Identical setups in other AI Search resources are working as expected.

On further review, it seems like most of our indexers for AI Search resources that are in East US and East US 2 are failing.

Indexers on AI Search resources in Central US and West US 2 are still operating normally.

Answer 1

@Eric Richards Apologies for inconvenience.

As per discussion This is By Design and workaround is to address the error. You'll need to either declare these fields in the index or remove them from the field mapping. This is due to product design and limitations.

Additionally, Feel free to express your feedback regarding the Azure AI Search on Uservoice.

All input shared in these forums undergoes monitoring and review by the Microsoft engineering teams responsible for developing Azure. Furthermore, users with similar requests have the opportunity to up-vote your post and contribute their comments.

Your feedback is actively considered in the ongoing development of the Azure.

Hope this helps.