ADF Dataflow issue with source API query

Daniel Baird 0

am trying to setup an Azure Data Factory Dataflow.

The source is a REST API from Monday.com which uses GraphQL. The API requires POST method, with 3 headers Content-Type "application/json", API-Version "2025-01 " and Authorization: "API token"

The request body requires a query to pull the data. Heres my query example

{ "query":"query { boards(ids: someid) {items_page(limit: 1) {cursor items {id name column_values {column {title} text}}}}}" }

When i run the dataflow it fails with the following

InvalidOperationException: DSL compilation failed: DF-DSL-001 - DSL stream has parsing errors\\nLine 8 Position 11: body: '{ \\nmismatched input

I have tried placing the query into a single line but it is being rejected by the API source with the following error:

at Source 'source2': Failure to read most recent page request: DF-REST_001- Error response from server: Some(<html> <head><title>400 Bad Request</title></head> <body> <center><h1>400 Bad Request</h1></center> <hr><center>cloudflare</center> </body> </html>), Status code: 400. Please check your request url and body. (url:https://api.monday.com/v2/,request body: Some({ "query":"query { boards(ids: some id) {items_page(limit: 1) { cursor items { id name column_values { column { title } text } } } }}" }), request method: POST

The dataflow JSON is automatically created by ADF which shows this and as you can see it is placing , after the first { and last }

    "source(allowSchemaDrift: true,",
                "     validateSchema: false,",
                "     inferDriftedColumnTypes: true,",
                "     format: 'rest',",
                "     timeout: 30,",
                "     headers: ['Content-Type' -> 'application/json', 'API-Version' -> '2025-01', 'Authorization' -> 'Some API Token', 'Accept' -> 'application/json'],",
                "     httpMethod: 'POST',",
                "     body: '{ ",
                "\"query\":\"query { boards(ids: some id) {items_page(limit: 1) { cursor items { id name column_values { column { title } text } } } }}\" ",
                "}',",
                "     paginationRules: ['supportRFC5988' -> 'true'],",
                "     responseFormat: ['type' -> 'json', 'documentForm' -> 'arrayOfDocuments']) ~> source2">```
jsonrestapigraphqlazure-data-factory

I have also changed the source format option from none to JSON and got a Error 415 response advising to check the URL and/or body

Daniel Baird 0

Heres the 415 Error for reference:

at Source 'source2': Failure to read most recent page request: DF-REST_001 - Error response from server: Some(), Status code: 415. Please check your request url and body. (url:https://api.monday.com/v2,request body: Some({"query: "query { boards(ids: some id) {items_page(limit: 1) {cursor items {id name column_values {column {title} text}}}}}"}), request method: POST)

J N S S Kasyap 270 Reputation points Microsoft Vendor

2025-02-24T02:49:20.7666667+00:00
Hi @Daniel Baird

Thanks for posting your query!

The 415 error you're facing is due to an issue with the Content-Type or body format of the request sent to the Monday.com GraphQL API. From your description, it looks like there is a formatting issue in the JSON body. The GraphQL query is not being correctly encoded, and this is causing the server to reject the request.

Headers Section:
The issues with the code include the use of single quotes (') instead of double quotes (") for JSON keys and values, which is incorrect according to JSON standards. Additionally, the arrow (->) is used incorrectly to separate key-value pairs, whereas JSON requires a colon (:) to properly assign values to keys.

Correct Code:

"headers": [ {"Content-Type": "application/json"}, {"API-Version": "2025-01"}, {"Authorization": "Some API Token"}, {"Accept": "application/json"} ],

The headers are now in the correct JSON format as an array of objects.

Body Section:
The issue stems from unnecessary extra quotes and commas in the body, where the query is improperly split, causing incorrect formatting. While escaping quotes inside strings is necessary, the body has excessive escape characters around the query itself, making it hard to read and incorrect. Additionally, the body is not properly wrapped as a valid JSON object, and the structure is fragmented with unnecessary quotes, leading to parsing errors.
Correct code:

"body": { "query": "query { boards(ids: someid) { items_page(limit: 1) { cursor items { id name column_values { column { title } text } } } } }" },

Once the request headers and body are properly formatted, the 415 error should no longer occur.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Thank you.

Daniel Baird 0

Thanks for your response.

Below is the full JSON from ADF dataflow. The JSON is automatically created and structured by ADF.

{
    "name": "dataflow1",
    "properties": {
        "type": "MappingDataFlow",
        "typeProperties": {
            "sources": [
                {
                    "dataset": {
                        "referenceName": "monday_api",
                        "type": "DatasetReference"
                    },
                    "name": "source1"
                }
            ],
            "sinks": [],
            "transformations": [],
            "scriptLines": [
                "source(allowSchemaDrift: true,",
                "     validateSchema: false,",
                "     httpMethod: 'POST',",
                "     body: '{",
                "\"query\": \"query { boards(ids: someid) { items_page(limit: 1) { cursor items { id name column_values { column { title } text } } } } }\"",
                "}',",
                "     timeout: 30,",
                "     headers: ['Content-Type' -> 'application/json', 'API-Version' -> '2025-01', 'Authorization' -> 'Some API Token'],",
                "     paginationRules: ['supportRFC5988' -> 'true'],",
                "     responseFormat: ['type' -> 'json', 'documentForm' -> 'documentPerLine']) ~> source1"
            ]
        }
    }
}

J N S S Kasyap 270 Reputation points Microsoft Vendor

2025-02-25T09:01:11.96+00:00

@Daniel Baird

ADF’s low-code environment generates code based on UI settings, which may not meet strict API requirements like GraphQL’s JSON format. It misinterprets GraphQL queries as raw REST payloads due to the lack of built-in support.
JSON bodies are treated as string literals instead of proper JSON objects, causing extra escaping issues.
Pagination rules are applied based on REST standards, which do not align with GraphQL’s cursor-based pagination.

ADF automatically applies pagination rules, such as paginationRules: ['supportRFC5988' -> 'true'], even when they are unnecessary. However, GraphQL manages pagination within the query using cursors, making ADF's pagination settings incompatible. This mismatch can confuse the API, leading to errors like 400 Bad Request or 415 Unsupported Media Type.

Alternative solutions:

Manually override headers and body by using a parameterized dataset instead of relying on ADF’s auto-generated JSON to ensure correct formatting.
Use a Web Activity instead of a Mapping Data Flow to maintain full control over the request format and avoid ADF’s auto-formatting issues when sending raw JSON.

I hope the above information helps, If the issue continues, let me know, and we can look into other possible causes!
J N S S Kasyap 270 Reputation points Microsoft Vendor

2025-02-26T07:50:15.4066667+00:00

@Daniel Baird
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Daniel Baird 0 Reputation points

2025-02-26T11:48:00.48+00:00

@J N S S Kasyap Thanks for your response.

Just to give some additional background on this.

I built the Graph query in Monday.com API playground. The query works in the body of both web and copy data activities. Unfortunately, the schema column values are repeated from the source, and I need to map the column value {text} to their respective columns in Azure SQL table.

I have the process working in a pipeline fine by using an initial web activity to then store the cursor into a variable which then an Until activity runs a copy activity with another web activity/set variable to continue to update the cursor until the if condition is met which is cursor results to null.

The problem I have with this method is it takes approximately 15 seconds to run each process and to write the data using the limit to 1 result per cursor (this enables the rows to be wrote separately for the repeated column values). I have a large amount of data to query and write and would take days.

My idea was to use the dataflow so I could call the source REST API and then flatten the JSON to be able to then re-map the schema before its wrote to the Azure SQL table.

I have tried manually overriding the headers and body by using a parameterized dataset but it is failing with parse issues.

Can you provide an example of how to override the headers and body in the dataset?

Thanks
J N S S Kasyap 270 Reputation points Microsoft Vendor

2025-02-27T08:55:13.6566667+00:00
@Daniel Baird
Thanks for the providing additional information

Parameterize the body (GraphQL query) and headers so that they can be passed dynamically into the dataset when invoked in the data flow.

{ "name": "monday_api", "properties": { "type": "RestDataset", "typeProperties": { "linkedServiceName": { "referenceName": "monday_com_api_linked_service", "type": "LinkedServiceReference" }, "relativeUrl": "/v2/", "requestMethod": "POST", "requestBody": { "value": "@{activity('Get GraphQL Query').output.queryBody}" }, "headers": { "Content-Type": "application/json", "API-Version": "2025-01", "Authorization": "@{linkedService().apiToken}", "Accept": "application/json" }, "requestTimeout": "00:01:00", "paginationRules": {} } } }

Since GraphQL handles pagination via cursors within the query, you might not need the standard REST pagination. You can leave it empty.

In the pipeline, first use a Set Variable activity to store the GraphQL query, then use a Web Activity to trigger the API call and store the cursor for pagination. Finally, leverage a Mapping Data Flow to flatten the API response and transform it into the desired format before loading it into SQL.

By using a Web Activity with a Set Variable for the query and using a parameterized REST Dataset, you'll have more control over how headers and bodies are formatted and passed. This method should allow you to get rid of the parsing issues and improve the speed of your data load, as you're directly interacting with the API

I hope the above information helps, If the issue continues, let me know, and we can look into other possible causes!
J N S S Kasyap 270 Reputation points Microsoft Vendor

2025-02-28T00:30:42.15+00:00

@Daniel Baird
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Share via

ADF Dataflow issue with source API query

Your answer