Is there a solution if I have lost Azure Functions apps logs?

Dewa　Tsukasa 40

Hello Everyone.

I am using a BI tool to check Azure Functions apps logs, but am having problems with some data being missing.

I followed the instructions on the official page

https://learn.microsoft.com/en-us/troubleshoot/azure/azure-functions/monitoring/functions-monitoring-appinsightslogs#logs-are-missing-or-partial

and was told that I could adjust it in host.json, so I entered the following information:

{
  "version": "2.0",
  "logging": {
    "applicationInsights": {
      "samplingSettings": {
        "isEnabled": false
      },
      "enableLiveMetricsFilters": true
    },
    "logLevel": {
      "default": "Warning",
      "Function.Function1.User": "Information",
      "Function.Function2.User": "Information",
      "Function.Function3.User": "Information",
      "Host.Results": "Information"
    }
  },
  "concurrency": {
    "dynamicConcurrencyEnabled": true,
    "snapshotPersistenceEnabled": true
  },
  "functionTimeout": "00:00:20"
}

But data was still missing:

union requests,dependencies,pageViews,browserTimings,exceptions,traces
| where timestamp > todatetime("11/27/2024 15:00:00") and timestamp < todatetime("11/28/2024 14:59:59")
| summarize TelemetrySavedPercentage = 100/avg(itemCount), TelemetryDroppedPercentage = 100-100/avg(itemCount) by bin(timestamp, 1h), itemType
| sort by timestamp asc

query_data.csv.txt

The solution I looked into was to set the maxTelemetryItemsPerSecond value in host.json to a large value such as 100 (the default value is 20), but I'm not sure if that really solves the problem.

Please tell me how to solve this problem.

Rahul Podila 805 Reputation points Microsoft Vendor

2024-12-13T01:30:14.98+00:00

Hi @Dewa　Tsukasa
If you had a chance to see my comment to your question. If it was helpful, please click "Upvote" on my post let us know Thank you...!
Rahul Podila 805 Reputation points Microsoft Vendor

2024-12-16T07:20:03.07+00:00

Hi @Dewa　Tsukasa

We haven't heard back from you. Please reply if you have any questions in this matter and we will gladly continue the discussion.
Dewa　Tsukasa 40 Reputation points

2024-12-17T05:20:35.64+00:00

deleted.
Dewa　Tsukasa 40 Reputation points

2024-12-17T05:23:29.3333333+00:00

deleted

2 answers

Rahul Podila 805 Reputation points Microsoft Vendor

2024-12-12T01:52:24.51+00:00

Hi @Dewa　Tsukasa

Welcome to the Microsoft Q&A Platform! Thank you for asking your question here.

Setting the maxTelemetryItemsPerSecond value in host.json to a larger value can help in increasing the amount of data that is collected and sent to Application Insights. However, it may not solve the problem of missing data completely.

There could be other reasons why data is missing in the logs. One reason could be that the logs are not being generated due to a configuration issue. Another reason could be that the logs are being generated, but are not being sent to Application Insights due to a connectivity issue.

To troubleshoot this issue, you can try the following steps:

Check if the logs are being generated in the first place. You can do this by checking the logs in the Azure portal or by using the Azure Functions Core Tools.

Check if the logs are being sent to Application Insights. You can do this by checking the Application Insights logs in the Azure portal or by using the Application Insights API.

Check if there are any configuration issues with the Azure Functions app or the BI tool that you are using. You can do this by reviewing the configuration settings and making sure that they are correct.

Check if there are any connectivity issues between the Azure Functions app and Application Insights. You can check the network settings and make sure that they are correct. You can also try pinging the Application Insights endpoint to see if there are any connectivity issues.

If you are still facing issues with missing data in the logs, you can try reaching out to Microsoft support for further assistance.

If you have any further queries, do let us know.
Please sign in to rate this answer.

1 person found this answer helpful.
Dewa　Tsukasa 40 Reputation points

2024-12-17T05:31:41.6266667+00:00

deleted.

Dewa　Tsukasa 40 Reputation points

2024-12-17T07:18:07.74+00:00

Hi @Rahul Podila

Sorry for the late reply.

I will answer including the contents of your message on December 24th.

First, regarding the current host.json, I have made the following changes since my last post:

{ "version": "2.0", "logging": { "applicationInsights": { "samplingSettings": { "isEnabled": true, "maxTelemetryItemsPerSecond": 40 }, "enableLiveMetricsFilters": true }, "logLevel": { "default": "Warning", "Function.Avatars.User": "Information", "Function.DatabaseActivate.User": "Information", "Function.GameClients.User": "Information", "Function.GameClientsLink.User": "Information", "Function.GameClientsReset.User": "Information", "Function.GameClientsUnlinkUserSerials.User": "Information", "Function.Photos.User": "Information", "Function.PhotosDelete.User": "Information", "Function.PhotosRegister.User": "Information", "Function.UsersRegister.User": "Information", "Host.Results": "Information" } }, "concurrency": { "dynamicConcurrencyEnabled": true, "snapshotPersistenceEnabled": true }, "functionTimeout": "00:00:20" }

The first change was to change isEnabled to true.

In the last update, setting a large value such as maxTelemetryItemsPerSecond to 500 did not have any effect, so I thought that it would not be effective unless isEnabled was enabled, and changed it to true.

The second change was to set maxTelemetryItemsPerSecond to 40.

The reason for this is that if isEnabled was enabled and operated normally with the previous update value of 500, there is a possibility that the cost would be enormous, so I set it to 40 and am observing the situation.

Next, I will answer your message.

Regarding whether the log was generated, I was able to search using Kusto queries in the Application Insights logs on the Azure portal, so I determined that the generation was successful.

Regarding the configuration settings of the Azure Functions app being used, we determined that there was no problem with the configuration settings because the resources displayed in Application Insights in the monitoring of the Azure Functions app were correctly set, and the connection string was correctly set in the APPLICATIONINSIGHTS_CONNECTION_STRING environment variable in the settings of the Azure Functions app.

Regarding the connection between the Azure Functions app and Application Insights, we determined that there was no problem with the connection because we were able to successfully ping the Application Insights endpoint.

Regarding the missing logs, even though a certain user in the database had been updated, we tried to check the traces log for the communication content, but the log was not found, but the traces log for the communication content of another user was found in the same way, so we determined that there was a missing log.

As for future actions, we plan to increase the value of maxTelemetryItemsPerSecond in host.json until the TelemetrySavedPercentage in the log reaches 100. However, it remains unclear whether this is the correct change.

union requests,dependencies,pageViews,browserTimings,exceptions,traces | where timestamp > todatetime("12/12/2024 15:00:00") and timestamp < todatetime("12/17/2024 02:59:59") | summarize TelemetrySavedPercentage = 100/avg(itemCount), TelemetryDroppedPercentage = 100-100/avg(itemCount) by bin(timestamp, 1h), itemType | order by bin(timestamp, 1h) asc, itemType asc

query_data.csv.txt

Rahul Podila 805 Reputation points Microsoft Vendor

2024-12-19T03:19:14.72+00:00

Hi @Dewa　Tsukasa

Thanks for your update! It seems like you’ve made some good progress with the missing logs issue. Here are some simple steps you can try to fix it:

First, keep increasing the maxTelemetryItemsPerSecond value little by little. After each change, check if more logs are showing up. If you have sampling turned on, try turning it off for now. This way, you can capture all logs without skipping any.

Next, check your function code to make sure it’s logging everything correctly. Make sure the log statements are in the right places and that nothing is being missed.

Also, be aware of the limits set by Application Insights on how many logs it can take at once. Make sure your settings don’t go over these limits to avoid losing logs.

You might want to set your logging level to "Verbose" for a while. This will give you more detailed information and help you see if there are specific problems causing some logs to be missing. Use Kusto queries in Application Insights to look for patterns or specific times when logs are missing, which might help you find the issue.

If there are any throttling issues with Azure Functions or Application Insights that could be affecting log transmission. Also, make sure there are no connectivity problems between your Azure Functions app and Application Insights.

Finally, run tests with different users or actions to see if the missing logs happen all the time or just in certain situations. If you’ve tried all these steps and still have problems, consider reaching out to Microsoft support for more help.

By following these steps, you should be able to find and fix the missing logs in your Azure Functions app.

If you have any further queries, do let us know

If the answer is helpful, please click "Accept Answer" and "Upvote it".

Dewa　Tsukasa 40 Reputation points

2024-12-20T09:40:09.8666667+00:00

Hi. @Rahul Podila

thank you for your reply.

However, due to my lack of understanding, I was unable to understand all of the responses.

I have a few questions.

First, keep increasing the maxTelemetryItemsPerSecond value little by little. After each change, check if more logs are showing up. If you have sampling turned on, try turning it off for now. This way, you can capture all logs without skipping any.

Previously around December 4th, I changed the sampling setting isEnabled to false and changed maxTelemetryItemsPerSecond to 500. However, this did not have any effect, so I changed sampling to true.

Therefore, there are some suspicious items in the answer, but I don't know how to check them, so please tell me.

Next, check your function code to make sure it’s logging everything correctly. Make sure the log statements are in the right places and that nothing is being missed.

First, I checked the code and ran it multiple times in Application Insights in the staging environments, and confirmed that the logs were being output, so I decided that there was no problem.

Also, be aware of the limits set by Application Insights on how many logs it can take at once. Make sure your settings don’t go over these limits to avoid losing logs.

Is this explanation about the number of results that can be obtained by Kusto queries? Or is it about Application Insights settings?

For the former Kusto query, we can confirm that the query result display limit is not being hit because we can perform a search narrowed down by Where conditions and get 0 results.

For the latter Application Insights settings, I understood that maxTelemetryItemsPerSecond in host.json was this setting, but are there any other settings that limit the number of results obtained besides maxTelemetryItemsPerSecond?

I read the official documentation for host.json, but I couldn’t understand what controls Application Insights.

You might want to set your logging level to "Verbose" for a while. This will give you more detailed information and help you see if there are specific problems causing some logs to be missing. Use Kusto queries in Application Insights to look for patterns or specific times when logs are missing, which might help you find the issue.

I read the official documentation about this and there is no mention of Verbose, so does setting Verbose to logLevel cause any special behavior?

I apologize for asking so many questions regarding the instructions I received, but I appreciate your answers.

Rahul Podila 805 Reputation points Microsoft Vendor

2024-12-23T05:35:14.8266667+00:00

Hi

I apologize for any confusion caused by my previous response. Let me try to clarify my answers to your questions:

Sampling sometimes skips logs to reduce data load. To make sure all logs are captured, go ahead and turn off sampling in your host.json. You can do this by setting isEnabled to false:

"samplingSettings": { "isEnabled": false }

If the logs are being dropped because too many are being sent too quickly, increasing the maxTelemetryItemsPerSecond value might help. Try setting it to 60 or even 100 to allow more logs to be sent:

"maxTelemetryItemsPerSecond": 60

If your log level is set to Warning, you might miss some logs that are at a lower level (like Information or Trace). Try changing it to Trace so you capture everything:

"logLevel": { "default": "Trace" }

After making these changes, test your app and see if the missing logs start showing up in Application Insights. You can also run some Kusto queries to check if the logs are being recorded properly.

If the issue continues even after these changes, it might be worth checking your function code to ensure logs are being generated for every expected action.

If you have any further queries, do let us know

If the answer is helpful, please click "Accept Answer" and "Upvote it".
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
Pinaki Ghatak 5,310 Reputation points Microsoft Employee

2024-12-03T09:53:54.1+00:00
Hi @Dewa　Tsukasa

Before adjusting the maxTelemetryItemsPerSecond value, I recommend checking a few things:

Make sure that the logs you're looking for are within the retention period of your Application Insights instance. By default, logs are retained for 90 days, but this can be changed in the Application Insights settings.

Check that the logs you're looking for are being sent to the correct Application Insights instance. You can check this by looking at the instrumentationKey value in your local.settings.json file or in the Application Insights settings.

Verify that the logs you're looking for are being generated by your Azure Functions app. You can do this by checking the code for your app and making sure that logging statements are being used appropriately.

If you've checked these things and are still experiencing missing logs, you can try adjusting the maxTelemetryItemsPerSecond value in the host.json file. However, keep in mind that increasing this value can increase the cost of your Application Insights instance.

I hope this helps.
Please sign in to rate this answer.
Dewa　Tsukasa 40 Reputation points

2024-12-04T04:06:54.52+00:00

@Pinaki Ghatak Thank you.I think your answer will help me solve the problem.

Dewa　Tsukasa 40 Reputation points

2024-12-05T15:28:50.3033333+00:00

I was unable to solve the problem of missing logs.

Based on the answer I received from Mr. Pinaki Ghatak, I changed the contents of host.json as follows:

{ "version": "2.0", "logging": { "applicationInsights": { "samplingSettings": { "isEnabled": false, "maxTelemetryItemsPerSecond": 500 }, "enableLiveMetricsFilters": true }, "logLevel": { "default": "Warning", "Function.Function1.User": "Information", "Function.Function2.User": "Information", "Function.Function3.User": "Information", "Host.Results": "Information" } }, "concurrency": { "dynamicConcurrencyEnabled": true, "snapshotPersistenceEnabled": true }, "functionTimeout": "00:00:20" }

The changes I made to host.json were to overwrite maxTelemetryItemsPerSecond from the initial value of 20 to 25 times the default value of 500, but this did not have much effect.

Do I need to set isEnabled to true to apply maxTelemetryItemsPerSecond?

Dewa　Tsukasa 40 Reputation points

2024-12-05T15:38:09.2733333+00:00

union requests,dependencies,pageViews,browserTimings,exceptions,traces | where timestamp > todatetime("12/04/2024 15:00:00") and timestamp < todatetime("12/05/2024 14:59:59") | summarize TelemetrySavedPercentage = 100/avg(itemCount), TelemetryDroppedPercentage = 100-100/avg(itemCount) by bin(timestamp, 1h), itemType | order by timestamp asc

query_data.csv.txt
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Is there a solution if I have lost Azure Functions apps logs?

2 answers

Your answer