About more logging detail in Azure Synapse

Question

Hi,

I recently built an Azure Workbook to minitor an ADF Pipeline. Everything went well, since in Log Analytics (via Diagnostic Settings) I had the enough detail to write my Kusto query (inputs and outputs of a pipeline and activity).

I was asked to do the same with some Synapse pipelines but I realized that I didn't have the enough detail in Log Analytics. Moreover, Azure Monitor log tables for Synapse doesn't have input and output as a queriable field.

How could I log this information to be queried later?

ADF Pipeline Run log table

Synapse Pipeline Run log table

Regards

Jona

Answer 1

Hi Jona
Welcome to the Microsoft Q&A Platform! Thank you for asking your question here.

Use Custom Logging for Inputs and Outputs

Since the default Synapse logs in Log Analytics do not include detailed inputs and outputs, you can implement custom logging within your Synapse pipelines:

a. Add a Web Activity for Custom Logging

• Use a Web Activity in your Synapse pipeline to send custom logs to Log Analytics.
• Configure the Web Activity to call the Azure Log Analytics Data Collector API.
• Pass the inputs, outputs, and other relevant details as a JSON payload in the Web Activity.

b. Steps to Implement Custom Logging:

1. Create a Log Analytics Custom Log Table:
   • In your Log Analytics workspace, create a custom log table (e.g., SynapsePipelineLogs_CL) to store the custom logs.
2. Generate a Shared Key for Log Analytics:
   • Go to your Log Analytics workspace in the Azure portal.
   • Navigate to Agents management and copy the Workspace ID and Primary Key.
3. Configure the Web Activity:
   • In your Synapse pipeline, add a Web Activity after the activity whose inputs and outputs you want to log.
   • Set the Web Activity's URL to the Log Analytics Data Collector API endpoint:
   ·       https://<WorkspaceID>.ods.opinsights.azure.com/api/logs?api-version=2016-04-01
   • Set the HTTP method to POST.
   Add the following headers:

• Authorization: Generate a signature using the workspace ID and primary key.
• Log-Type: Specify the custom log table name (e.g., SynapsePipelineLogs_CL).
• x-ms-date: The current date in RFC 1123 format.
• In the body of the request, include the inputs, outputs, pipeline name, activity name, and any other relevant details as a JSON payload. Example Payload:

json
{
    "PipelineName": "@{pipeline().Pipeline}",
    "ActivityName": "@{activity().Name}",
    "Input": "@{activity('YourActivityName').Input}",
    "Output": "@{activity('YourActivityName').Output}",
    "Status": "@{activity('YourActivityName').Status}",
    "Timestamp": "@{utcnow()}"

Query Custom Logs in Log Analytics

Once the custom logs are sent to Log Analytics, you can query them using Kusto queries. For example:

SynapsePipelineLogs_CL
| where PipelineName == "YourPipelineName"
| project Timestamp, PipelineName, ActivityName, Input, Output, Status

 Alternative Method: Use Azure Functions for Custom Logging

If you prefer not to use the Web Activity directly, you can create an Azure Function to handle the logging. The Synapse pipeline can call the Azure Function via a Web Activity, and the function can send the logs to Log Analytics.

Feel free to reach out if you have any further questions or need additional information—I’m happy to assist!

Please provide your valuable comments

Please do not forget to "Accept the answer” and “upvote it” wherever the information provided helps you, this can be beneficial to other community members.it would be greatly appreciated and helpful to others.