Count the # of Prompt caching tokens for Azure OpenAI service
youyang
0
Reputation points
Hi, Azure team, I deploy the gpt-4o-mini-2024-07-18 model on azure openai service and call it using AzureOpenAI client:
client = AzureOpenAI(
api_key=<api key>,
azure_endpoint=https://xxxx.openai.azure.com/
api_version=2024-10-01-preview,
)
and send messages using:
completion = client.chat.completions.create(
model="gpt-4o-mini",
messages=messages,
stream=False,
temperature=0.4,
)
However, when I print the completion.Usage, it outputs:
usage=CompletionUsage(completion_tokens=212, prompt_tokens=12554, total_tokens=12766)
I can't find fields like "prompt_tokens_details" or "cached_tokens" as the shown in https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/prompt-caching Is there any work around if I want to count the # of cached tokens in prompt?
Thanks
Sign in to answer