Inconsistent but frequent 403 FORBIDDEN API responses from graph API when creating or renewing Teams channel message subscriptions

Question

Starting at approximately 1PM PST on 1/27/25, my team started to observe a significant increase in 403 Forbidden responses from the MS Graph API when renewing subscriptions (using this endpoint) for MS Teams channels (chatMessage resource). No change occurred in our systems at that time.

Attached is a histogram generated from our logs indicating error counts per day due to this problem. Retrying the renew/create does sometimes go through. On the 29th we implemented a retry policy for this code which cuts down on the number of overall failures, which you can see represented in the attached chart as well. (My teammates report that creating subscriptions suffers from a similar problem as renewing, but the attached chart represents failure rate only for renewing subscriptions.)

Our team is accessing the API using delegated permissions, via a service account, on behalf of our clients, so this error is observed not just for API calls made within our organization, but across many different Teams customers. We have double-checked our permission configuration and believe it to be correct, especially because these requests can be retried and eventually do go through. I have a few Graph request IDs from Wednesday the 29th for some failures within our org that I have attached; if more recent ones are needed I can gather those as well.

Errors observed:

Request IDs:

teams-request-ids.txt