Ingestion Jobs - Create

Reference

Service:: Azure AI Services

API Version:: 2024-08-01-preview

Creates an ingestion job with the specified job id.

PUT {endpoint}/openai/ingestion/jobs/{job-id}?api-version=2024-08-01-preview

URI Parameters

Name	In	Required	Type	Description
endpoint	path	True	string url	Supported Cognitive Services endpoints (protocol and hostname, for example: https://aoairesource.openai.azure.com. Replace "aoairesource" with your Azure OpenAI account name).
job-id	path	True	string	The id of the job that will be created.
api-version	query	True	string	The requested API version.

Request Header

Name	Required	Type	Description
mgmt-user-token		string	The token used to access the workspace (needed only for user compute jobs).
aml-user-token		string	The token used to access the resources within the job in the workspace (needed only for user compute jobs).

Request Body

The request body can be one of the following:

Name	Description
IngestionJobSystemCompute
IngestionJobUserCompute

IngestionJobSystemCompute

Name	Required	Type	Description
kind	True	string: SystemCompute	IngestionJobType The job type.
completionAction		IngestionJobCompletionAction	The completion action.
dataRefreshIntervalInHours		integer
datasource		SystemComputeDatasource: SystemComputeStorage SystemComputeUrl	SystemComputeDatasource
jobId		string
searchServiceConnection		BaseConnection: ConnectionStringConnection DeploymentConnection EndpointKeyConnection EndpointMIConnection WorkspaceConnection	BaseConnection A connection to a resource.

IngestionJobUserCompute

Name	Required	Type	Description
kind	True	string: UserCompute	IngestionJobType The job type.
workspaceId	True	string
compute		JobCompute: CustomCompute ServerlessCompute	JobCompute The compute settings of the job.
dataRefreshIntervalInHours		integer
datasource		UserComputeDatasource: UserComputeDataset UserComputeUrl	UserComputeDatasource
jobId		string
target		TargetIndex: AzureAISearchIndex CosmosDBIndex PineconeIndex	TargetIndex Information about the index to be created.

Responses

Name	Type	Description
200 OK	IngestionJob: IngestionJobSystemCompute IngestionJobUserCompute	Success
Other Status Codes	ErrorResponse	An error occurred.

Security

api-key

API key authentication

Type: apiKey
In: header

OAuth2Auth

OAuth2 authentication

Type: oauth2
Flow: implicit
Authorization URL: https://login.microsoftonline.com/common/oauth2/v2.0/authorize

Scopes

Name	Description
https://cognitiveservices.azure.com/.default

Examples

Create a system-compute ingestion job

Create a user-compute ingestion job

Create a system-compute ingestion job

Sample request

HTTP

PUT {endpoint}/openai/ingestion/jobs/ingestion-job?api-version=2024-08-01-preview

{
  "kind": "SystemCompute",
  "searchServiceConnection": {
    "kind": "EndpointWithManagedIdentity",
    "endpoint": "https://aykame-dev-search.search.windows.net"
  },
  "datasource": {
    "kind": "Storage",
    "connection": {
      "kind": "EndpointWithManagedIdentity",
      "endpoint": "https://mystorage.blob.core.windows.net/",
      "resourceId": "/subscriptions/1234567-abcd-1234-5678-1234abcd/resourceGroups/my-resource/providers/Microsoft.Storage/storageAccounts/mystorage"
    },
    "containerName": "container",
    "chunking": {
      "maxChunkSizeInTokens": 2048
    },
    "embeddings": [
      {
        "connection": {
          "kind": "RelativeConnection"
        },
        "deploymentName": "Ada"
      }
    ]
  },
  "dataRefreshIntervalInHours": 24,
  "completionAction": "keepAllAssets"
}

Sample response

Status code:: 200

operation-location: https://aoairesource.openai.azure.com/openai/ingestion/jobs/ingestion-job/runs/72a2792ef7d24ba7b82c7fe4a37e379f?api-version=2024-08-01-preview

{
  "kind": "SystemCompute",
  "jobId": "ingestion-job",
  "searchServiceConnection": {
    "kind": "EndpointWithManagedIdentity",
    "endpoint": "https://aykame-dev-search.search.windows.net"
  },
  "datasource": {
    "kind": "Storage",
    "connection": {
      "kind": "EndpointWithManagedIdentity",
      "endpoint": "https://mystorage.blob.core.windows.net/",
      "resourceId": "/subscriptions/1234567-abcd-1234-5678-1234abcd/resourceGroups/my-resource/providers/Microsoft.Storage/storageAccounts/mystorage"
    },
    "containerName": "container",
    "chunking": {
      "maxChunkSizeInTokens": 2048
    },
    "embeddings": [
      {
        "connection": {
          "kind": "RelativeConnection"
        },
        "deploymentName": "Ada"
      }
    ]
  },
  "dataRefreshIntervalInHours": 24,
  "completionAction": "keepAllAssets"
}

Create a user-compute ingestion job

Sample request

HTTP

PUT {endpoint}/openai/ingestion/jobs/ingestion-job?api-version=2024-08-01-preview

{
  "kind": "UserCompute",
  "workspaceId": "/subscriptions/f375b912-331c-4fc5-8e9f-2d7205e3e036/resourceGroups/adrama-copilot-demo/providers/Microsoft.MachineLearningServices/workspaces/adrama-rag-dev",
  "compute": {
    "kind": "ServerlessCompute"
  },
  "target": {
    "kind": "AzureAISearch",
    "connectionId": "/subscriptions/f375b912-331c-4fc5-8e9f-2d7205e3e036/resourceGroups/adrama-copilot-demo/providers/Microsoft.MachineLearningServices/workspaces/adrama-rag-dev/connections/search-connection"
  },
  "datasource": {
    "kind": "Dataset",
    "datasetId": "azureml://locations/centraluseuap/workspaces/83317fe6-efa6-4e4a-b020-d0edd11ec382/data/PlainText/versions/1",
    "datasetType": "uri_folder"
  }
}

Sample response

Status code:: 200

operation-location: https://aoairesource.openai.azure.com/openai/ingestion/jobs/ingestion-job/runs/72a2792ef7d24ba7b82c7fe4a37e379f?api-version=2024-08-01-preview

{
  "kind": "UserCompute",
  "jobId": "ingestion-job",
  "workspaceId": "/subscriptions/f375b912-331c-4fc5-8e9f-2d7205e3e036/resourceGroups/adrama-copilot-demo/providers/Microsoft.MachineLearningServices/workspaces/adrama-rag-dev",
  "compute": {
    "kind": "ServerlessCompute"
  },
  "target": {
    "kind": "AzureAISearch",
    "connectionId": "/subscriptions/f375b912-331c-4fc5-8e9f-2d7205e3e036/resourceGroups/adrama-copilot-demo/providers/Microsoft.MachineLearningServices/workspaces/adrama-rag-dev/connections/search-connection"
  },
  "datasource": {
    "kind": "Dataset",
    "datasetId": "azureml://locations/centraluseuap/workspaces/83317fe6-efa6-4e4a-b020-d0edd11ec382/data/PlainText/versions/1",
    "datasetType": "uri_folder"
  }
}

Definitions

Name	Description
AzureAISearchIndex	Azure AI Search Index.
ChunkingSettings	ChunkingSettings
ComputeType	The compute type.
ConnectionStringConnection	Connection string connection.
ConnectionType	The connection type.
CosmosDBIndex	CosmosDB Index.
CrawlingSettings	CrawlingSettings
CustomCompute	Custom compute.
DeploymentConnection	Relative deployment connection.
EndpointKeyConnection	Endpoint key connection.
EndpointMIConnection	Endpoint Managed Identity connection.
Error	Error
ErrorCode	ErrorCode
ErrorResponse	ErrorResponse
GenericEmbeddingSettings	ConnectionEmbeddingSettings
IngestionJobCompletionAction	The completion action.
IngestionJobSystemCompute
IngestionJobType	IngestionJobType
IngestionJobUserCompute
InnerError	InnerError
InnerErrorCode	InnerErrorCode
PineconeIndex	Pinecone Index.
ServerlessCompute	Serverless compute.
SystemComputeDatasourceType	The datasource type.
SystemComputeStorage	SystemComputeStorage
SystemComputeUrl	SystemComputeUrl
TargetType	The target type.
UserComputeDataset	UserComputeStorage
UserComputeDatasourceType	The datasource type.
UserComputeUrl	UserComputeUrl
WorkspaceConnection	AML Workspace connection.
WorkspaceConnectionEmbeddingSettings	WorkspaceConnectionEmbeddingSettings

AzureAISearchIndex

Azure AI Search Index.

Name	Type	Description
connectionId	string	The id of the connection pointing to the Azure AI Search Index.
kind	string: AzureAISearch	The target type.

ChunkingSettings

Name	Type	Description
maxChunkSizeInTokens	integer

ComputeType

The compute type.

Name	Type	Description
CustomCompute	string	Custom user compute.
ServerlessCompute	string	Serverless user compute.

ConnectionStringConnection

Connection string connection.

Name	Type	Description
connectionString	string	Connection string
kind	string: ConnectionString	The connection type.

ConnectionType

The connection type.

Name	Type	Description
ConnectionString	string	Connection string.
EndpointWithKey	string	Endpoint and key connection.
EndpointWithManagedIdentity	string	Endpoint and managed identity.
RelativeConnection	string	Relative deployment
WorkspaceConnection	string	AML Workspace connection.

CosmosDBIndex

CosmosDB Index.

Name	Type	Description
collectionName	string	The name of the cosmos DB collection.
connectionId	string	The id of the connection pointing to the cosmos DB.
databaseName	string	The name of the cosmos DB database.
kind	string: CosmosDB	The target type.

CrawlingSettings

Name	Type	Description
maxCrawlDepth	integer
maxCrawlTimeInMins	integer
maxDownloadTimeInMins	integer
maxFileSize	integer
maxFiles	integer
maxRedirects	integer

CustomCompute

Custom compute.

Name	Type	Description
computeId	string	Id of the custom compute
kind	string: CustomCompute	The compute type.

DeploymentConnection

Relative deployment connection.

Name	Type	Description
kind	string: RelativeConnection	The connection type.

EndpointKeyConnection

Endpoint key connection.

Name	Type	Description
endpoint	string	Endpoint
key	string	Key
kind	string: EndpointWithKey	The connection type.

EndpointMIConnection

Endpoint Managed Identity connection.

Name	Type	Description
endpoint	string	Endpoint
kind	string: EndpointWithManagedIdentity	The connection type.
resourceId	string	Resource Id

Error

Name	Type	Description
code	ErrorCode	ErrorCode Error codes as defined in the Microsoft REST guidelines (https://github.com/microsoft/api-guidelines/blob/vNext/Guidelines.md#7102-error-condition-responses).
details	Error[]	The error details if available.
innererror	InnerError	InnerError Inner error as defined in the Microsoft REST guidelines (https://github.com/microsoft/api-guidelines/blob/vNext/Guidelines.md#7102-error-condition-responses).
message	string	The message of this error.
target	string	The location where the error happened if available.

ErrorCode

Name	Type	Description
conflict	string	The requested operation conflicts with the current resource state.
contentFilter	string	Image generation failed as a result of our safety system.
fileImportFailed	string	Import of file failed.
forbidden	string	The operation is forbidden for the current user/api key.
internalFailure	string	Internal error. Please retry.
invalidPayload	string	The request data is invalid for this operation.
itemDoesAlreadyExist	string	The item does already exist.
jsonlValidationFailed	string	Validation of jsonl data failed.
notFound	string	The resource is not found.
quotaExceeded	string	Quota exceeded.
serviceUnavailable	string	The service is currently not available.
tooManyRequests	string	Too many requests. Please retry later.
unauthorized	string	The current user/api key is not authorized for the operation.
unexpectedEntityState	string	The operation cannot be executed in the current resource's state.

ErrorResponse

Name	Type	Description
error	Error	Error Error content as defined in the Microsoft REST guidelines (https://github.com/microsoft/api-guidelines/blob/vNext/Guidelines.md#7102-error-condition-responses).

GenericEmbeddingSettings

ConnectionEmbeddingSettings

Name	Type	Description
connection	BaseConnection: ConnectionStringConnection DeploymentConnection EndpointKeyConnection EndpointMIConnection WorkspaceConnection	BaseConnection A connection to a resource.
deploymentName	string
modelName	string

IngestionJobCompletionAction

The completion action.

Name	Type	Description
cleanUpTempAssets	string	Will clean up intermediate assets created during the ingestion process.
keepAllAssets	string	Will not clean up any of the intermediate assets created during the ingestion process.

IngestionJobSystemCompute

Name	Type	Description
completionAction	IngestionJobCompletionAction	The completion action.
dataRefreshIntervalInHours	integer
datasource	SystemComputeDatasource: SystemComputeStorage SystemComputeUrl	SystemComputeDatasource
jobId	string
kind	string: SystemCompute	IngestionJobType The job type.
searchServiceConnection	BaseConnection: ConnectionStringConnection DeploymentConnection EndpointKeyConnection EndpointMIConnection WorkspaceConnection	BaseConnection A connection to a resource.

IngestionJobType

Name	Type	Description
SystemCompute	string	Jobs that run on service owned resources.
UserCompute	string	Jobs that run on user owned workspace.

IngestionJobUserCompute

Name	Type	Description
compute	JobCompute: CustomCompute ServerlessCompute	JobCompute The compute settings of the job.
dataRefreshIntervalInHours	integer
datasource	UserComputeDatasource: UserComputeDataset UserComputeUrl	UserComputeDatasource
jobId	string
kind	string: UserCompute	IngestionJobType The job type.
target	TargetIndex: AzureAISearchIndex CosmosDBIndex PineconeIndex	TargetIndex Information about the index to be created.
workspaceId	string

InnerError

Name	Type	Description
code	InnerErrorCode	InnerErrorCode Inner error codes as defined in the Microsoft REST guidelines (https://github.com/microsoft/api-guidelines/blob/vNext/Guidelines.md#7102-error-condition-responses).
innererror	InnerError	InnerError Inner error as defined in the Microsoft REST guidelines (https://github.com/microsoft/api-guidelines/blob/vNext/Guidelines.md#7102-error-condition-responses).

InnerErrorCode

Name	Type	Description
invalidPayload	string	The request data is invalid for this operation.

PineconeIndex

Pinecone Index.

Name	Type	Description
connectionId	string	The id of the connection pointing to the pinecone.
kind	string: Pinecone	The target type.

ServerlessCompute

Serverless compute.

Name	Type	Description
instanceCount	integer	The count of instances to run the job on.
kind	string: ServerlessCompute	The compute type.
sku	string	SKU Level

SystemComputeDatasourceType

The datasource type.

Name	Type	Description
Storage	string	Azure Storage Account.
Urls	string	URLs.

SystemComputeStorage

Name	Type	Description
chunking	ChunkingSettings	ChunkingSettings Chunking settings
connection	BaseConnection: ConnectionStringConnection DeploymentConnection EndpointKeyConnection EndpointMIConnection WorkspaceConnection	BaseConnection A connection to a resource.
containerName	string	container name
embeddings	GenericEmbeddingSettings[]	ConnectionEmbeddingSettings Connection Embedding Settings
kind	string: Storage	The datasource type.

SystemComputeUrl

Name	Type	Description
chunking	ChunkingSettings	ChunkingSettings Chunking settings
connection	BaseConnection: ConnectionStringConnection DeploymentConnection EndpointKeyConnection EndpointMIConnection WorkspaceConnection	BaseConnection A connection to a resource.
containerName	string	container name
crawling	CrawlingSettings	CrawlingSettings Crawling settings
embeddings	GenericEmbeddingSettings[]	ConnectionEmbeddingSettings Connection Embedding Settings
kind	string: Urls	The datasource type.
urls	string[]

TargetType

The target type.

Name	Type	Description
AzureAISearch	string	Azure AI Search Index.
CosmosDB	string	CosmosDB Index.
Pinecone	string	Pinecone Index.

UserComputeDataset

UserComputeStorage

Name	Type	Description
chunking	ChunkingSettings	ChunkingSettings Chunking settings
datasetId	string
datasetType	string
embeddings	WorkspaceConnectionEmbeddingSettings[]	WorkspaceConnectionEmbeddingSettings Connection id to the embedding model
kind	string: Dataset	The datasource type.

UserComputeDatasourceType

The datasource type.

Name	Type	Description
Dataset	string	Workspace Dataset.
Urls	string	URLs.

UserComputeUrl

Name	Type	Description
chunking	ChunkingSettings	ChunkingSettings Chunking settings
crawling	CrawlingSettings	CrawlingSettings Crawling settings
embeddings	WorkspaceConnectionEmbeddingSettings[]	WorkspaceConnectionEmbeddingSettings Connection id to the embedding model
kind	string: Urls	The datasource type.
urls	string[]

WorkspaceConnection

AML Workspace connection.

Name	Type	Description
connectionId	string	ConnectionId
kind	string: WorkspaceConnection	The connection type.

WorkspaceConnectionEmbeddingSettings

Name	Type	Description
connectionId	string
deploymentName	string
modelName	string

Partager via

Ingestion Jobs - Create

URI Parameters

Request Header

Request Body

IngestionJobSystemCompute

IngestionJobUserCompute

Responses

Security

api-key

OAuth2Auth

Scopes

Examples

Create a system-compute ingestion job

Sample request

Sample response

Create a user-compute ingestion job

Sample request

Sample response

Definitions

AzureAISearchIndex

ChunkingSettings

ComputeType

ConnectionStringConnection

ConnectionType

CosmosDBIndex

CrawlingSettings

CustomCompute

DeploymentConnection

EndpointKeyConnection

EndpointMIConnection

Error

ErrorCode

ErrorResponse

GenericEmbeddingSettings

IngestionJobCompletionAction

IngestionJobSystemCompute

IngestionJobType

IngestionJobUserCompute

InnerError

InnerErrorCode

PineconeIndex

ServerlessCompute

SystemComputeDatasourceType

SystemComputeStorage

SystemComputeUrl

TargetType

UserComputeDataset

UserComputeDatasourceType

UserComputeUrl

WorkspaceConnection

WorkspaceConnectionEmbeddingSettings

Ressources supplémentaires