Azure AI Search embeddings: max content length exceeded?

Barry Briggs 60

Creating embeddings for an array of 100 JSON objects in order to create an index. Loop through the objects creating an embedding for the body field (six simple fields and one vector field) and get error "This model's maximum context length is 8192 tokens, however you requested 16893 tokens (16893 in your prompt; 0 for the completion). Please reduce your prompt; or completion length."

However:

Using the tiktoken library:

The current object has 1778 tokens

The running total of 32 objects is 38355

The text for the embedding (for the offending object) is 7487 characters

I have put delays in the code to no avail. If I simply get an embedding for the offending object (no loop) it works which suggests it has something to do with the accumulating load -- but not sure what it could be.

SnehaAgrawal-MSFT 21,926 Reputation points

2024-12-24T05:02:36.1466667+00:00

Thanks for asking question! Rather than creating an embedding for the entire object, recommend just creating a single embedding for each text field and storing each separately in a single document.
Barry Briggs 60 Reputation points

2024-12-24T15:42:30.2566667+00:00
Thanks -- I wasn't clear in my original question -- that's what I'm doing. Basically in a loop I read files each of which holds a JSON object. Then I add a single vector field (to the body field) and generate an embedding.

Pseudocode:

string[] files=Directory.GetFiles(folder, "*.json"); foreach (string file in files) { string json=File.ReadAllText(file); doc=JsonConvert.DeserializeObject(json); VectorDoc v=new VectorDoc(doc); //same as doc except "bodyVector" field added OpenAIEmbedding embedding=OpenAIEmbeddingClient.GenerateEmbedding(doc.body); v.bodyVector=embedding.ToFloats(); .. }

Always fails after the 32nd file with token count as above. If I simply try it with the 32nd file (which has 1778 tokens), it works (so it has to do with the quantity not the specific file).

Share via

Azure AI Search embeddings: max content length exceeded?

Your answer