Keep Microsoft 365 Application Compliant Continuously with App Compliance Automation Tool for Microsoft 365

Article
12/12/2023

The Microsoft 365 App Compliance Automation Tool (ACAT) streamlines the essential controls for the Microsoft 365 Certification. Keep continuous compliance for your Microsoft 365 application using ACAT. Stay updated on compliance failures through notifications and seamlessly integrate ACAT into your Continuous Integration/Continuous Deployment pipeline.

Note

ACAT is currently in public preview and only supports applications built on Azure. Future updates will include functionality for apps built on non-Microsoft hosted cloud services. For any feedback on ACAT public preview, please complete this form. An ACAT product team specialist will follow up with you as soon as possible.

Get the latest control assessment of your compliance report via notifications

After you create a compliance report for your Microsoft 365 application or environment, ACAT automatically collects compliance data and conducts daily control assessments. Additionally, you can receive notifications for any changes in report settings or assessments.

Get notifications with webhook

Create a webhook to receive notifications from ACAT.

Go to Reports on the left.
Open the report that you would like to receive notifications for.
Select Notifications from the toolbar.
Select Create notification and then choose Webhook
- Webhook name: This is the unique identifier of the webhook within this report.
- Payload URL: The payload URL is the URL of the server that receives webhook POST requests from ACAT.
- Content type: ACAT currently only supports the application/json content type, which delivers the JSON payload directly as the body of the POST request.
- Secret: Setting a webhook secret allows you to ensure that POST requests sent to the payload URL are from ACAT. When the secret is set, ACAT uses it to create a hash signature with each payload. This hash signature is included with the headers of each request as x-acat-signature-256.
- SSL verification: SSL verification would only display when your payload URL is a secure site (HTTPS), and it helps ensure that payloads are delivered to your payload URL securely. We recommend keeping Enable SSL verification selected.
- Trigger events: Subscribe to ACAT events to get notifications.
Configuration to receive notifications via webhook.

Tip

Learn more about webhook payload.

Get quick compliance evaluation in Continuous Integration/Continuous Deployment pipeline

In addition to regular portal usage and receiving up-to-date notifications, ACAT can perform a quick compliance evaluation within your Continuous Integration/Continuous Deployment (CI/CD) pipeline. Ensuring the continuous compliance of your Microsoft 365 application.

In ACAT, control assessments are structured compliance assessments categorized according to Microsoft 365 Certification security domains, control families, and individual controls. To initiate control assessments within ACAT raw compliance data must be gathered. However, some of this raw compliance information may not be readily accessible from a technical perspective. To address this challenge, ACAT introduces a concept known as 'quick compliance evaluation.' This feature is designed to provide a rapid assessment of compliance status, allowing for the immediate collection of essential raw compliance data.

The implementation of quick compliance evaluation serves two purposes:

Timely Insights: facilitates real-time understanding of your compliance posture, giving visibility into an application's adherence to security standards and best practices.
Seamless Integration: Quick compliance evaluation can seamlessly integrate into your CI/CD pipeline, ensuring your applications remain robust and aligned with compliance requirements.

Integration with CI/CD pipeline by GitHub Actions

Note

ACAT GitHub Actions support is preview and only supports apps built on Microsoft Azure. Future updates will include functionality for apps built on other clouds.

Quick compliance evaluations are available in the CI/CD pipeline using the Microsoft 365 certification quick evaluation GitHub Action. There are two possible uses for this GitHub Action:

Use the latest deployment in the CI/CD pipeline as the definitive state for the application: ACAT conducts quick compliance evaluations based on the most recent deployment within the pipeline. Additionally, you have the option to request ACAT to refresh a corresponding compliance report based on this deployment, granting you comprehensive control assessments with daily updates.
Rely on an existing compliance report as reference for the application. ACAT utilizes the resources defined in an existing compliance report to perform the quick compliance evaluation.

Integration with CI/CD pipeline through ACAT REST API

Get quick compliance evaluations in CI/CD pipeline through ACAT REST API.

Connect to Azure resources with Service Principal Authentication. Follow this guide to learn how to create a service principal that can access resources. Once the service principal is created, follow the guidance from your pipeline tool to store the credentials. For example, connect to Microsoft Azure with an ARM service connection.

Get the list of resources that you would like to use for the quick compliance evaluation.

Use the latest deployment in the CI/CD pipeline. Review Azure PowerShell - Get-AzResourceGroupDeploymentOperation to get resources from deployment.

$resourceGroupName
$deploymentName

$resourceIds = @()
Get-AzResourceGroupDeploymentOperation `
    -ResourceGroupName $resourceGroupName `
    -Name $deploymentName `
| ForEach-Object {
    if (![String]::IsNullOrEmpty($_.TargetResource)) {
        $resourceIds += $_.TargetResource
    }
}

To acquire resources based on cloud resource management policy (for example, resources with specific tags, resources in particular resource groups, etc.), Review the Azure PowerShell - Get-AzResource.

# Get resources with same tag
$key = "<key-of-your-tag>"
$value = "<value-of-your-tag>"

$resourceIds = (Get-AzResource -TagName $key -TagValue $value).ResourceId

To use an existing ACAT compliance report, select ACAT REST API Report - Get to get the resources from report definition.

try {
    $token = "<your-Azure-credentials>"
    $reportName = "<report-name>"
    $apiVersion = "2023-02-15-preview"

    $header = @{}
    $header["Authorization"] = $token
    $header["x-ms-aad-user-token"] = $token
    $header["accept"] = "application/json"

    $uri = "https://management.azure.com/providers/Microsoft.AppComplianceAutomation/reports/" + $reportName + "?api-version=" + $apiVersion
    $response = Invoke-WebRequest $uri `
        -Method GET `
        -Headers $header `
        -ContentType "application/json" `
        -Verbose `
        -UseBasicParsing

    $resourceIds = @()
    if ($response.StatusCode -ne 200) {
        Write-Host "Failed to get resources from compliance report: $response"
        return 
    }

    $resources = $response.Content | ConvertFrom-Json
    $resourceIds = $resources.properties.resources.resourceId
}
catch {
    Write-Output "Failed to get resources from compliance report with exception: $_"
}

Use the triggerEvaluation API to get a quick compliance evaluation for the given resources.

try {
    $token = "<your-Azure-credentials>"
    $resourceIds = "<resource-ids-from-previous-step>"
    $apiVersion = "2023-02-15-preview"

    $header = @{}
    $header["Authorization"] = $token
    $header["x-ms-aad-user-token"] = $token
    $header["accept"] = "application/json"

    $body = @{resourceIds = $resourceIds }

    $uri = "https://management.azure.com/providers/Microsoft.AppComplianceAutomation/triggerEvaluation?api-version=$apiVersion"
    $response = Invoke-WebRequest $uri `
        -Method POST `
        -Headers $header `
        -ContentType "application/json" `
        -Body (ConvertTo-Json $body -Depth 8) `
        -Verbose `
        -UseBasicParsing
    
    # The triggerEvaluation API is asynchronous. Therefore, you need to pull the status to check whether it is completed. 
    # StatusCode 200: OK. The response indicates the quick compliance evaluation for given resource ids is completed.
    # StatusCode 202: Accepted. The response indicates the quick compliance evaluation for given resource ids is triggered and performing in backend. 
    if ($response.StatusCode -eq 200) {
        $result = $response.Content | ConvertFrom-Json
        Write-Host "Successfully get evaluation result:$result"
        return $result
    }
    elseif ($response.StatusCode -eq 202) {
        $retry_url = $response.Headers["Location"]

        do {
            Start-Sleep 10
            Write-Host "retry_url: $retry_url"
            $opResponse = Invoke-WebRequest `
                -Uri $retry_url `
                -ContentType "application/json" `
                -Verbose `
                -Method GET `
                -Headers $header `
                -UseBasicParsing

            if ($opResponse.StatusCode -eq 200) {
                $opResult = $opResponse.Content | ConvertFrom-Json
                Write-Host "Successfully get evaluation result: $opResult"
                return $opResult
            }
            elseif ($opResponse.StatusCode -eq 202) {
                continue
            }
            else {
                Write-Host "Failed to get evaluation result"
                break
            }
        } while ($true)
    }
    else {
        Write-Host "Failed to get compliance evaluation with triggerEvaluation API: $response"
    }
}
catch {
    Write-Output "Failed to get quick compliance assessment with exception: $_"
}

Get the quick compliance evaluation from the result of triggerEvaluation API.

Reference

Webhook payloads

For the ACAT webhook payloads for each event, they contain some common properties.

Key

Required?

Type

Description

EventDesc

Yes

String

Specific event that triggers the notification.

Message

Yes

String

The content of the notification. It usually contains the report name as the unique identifier and the timestamp when this event happens.

Details

String

This is JSON format content that contains the details of this notification. Taking the control assessments failure as an example, the details include all failed customer responsibilities for each control and affected resources.

ACAT REST API triggerEvaluation

API Name: triggerEvaluation

Service: App Compliance Automation
API Version: 2023-02-15-preview

Get quick compliance evaluation for given resources.

POST https://management.azure.com/providers/Microsoft.AppComplianceAutomation/triggerEvaluation?api-version=2023-02-15-preview

URI Parameters

Name

Required

Type

Description

api-version

query

True

string

The API version to use for this operation. You can use "2023-02-15-preview"

Request Header

Name

Required

Type

Description

Authorization

head

True

string

Bearer + " " + Access Token

x-ms-aad-user-token

head

True

string

Bearer + " " + Access Token

Content-Type

head

True

string

value: "application/json"

Request Body

Name

Required

Type

Description

resourceIds

True

string array

List of resource IDs to be evaluated

Responses Header

Name

Type

Description

200 OK

TriggerEvaluationResponse

The response indicates the quick compliance evaluation for given resource IDs is completed.

202 Accepted

The response indicates the quick compliance evaluation for given resource ids is triggered and performing in backend.

Other Status Codes

ErrorResponse

Error response.

TriggerEvaluationResponse

Name

Type

Description

triggerTime

string

The time when the evaluation is triggered.

evaluationEndTime

string

The time when the evaluation is end.

resourceIds

string[]

List of resource IDs to be evaluated

quickAssessments

QuickAssessment[]

List of quick assessments

QuickAssessment

Name

Type

Description

resourceId

string

Resource ID

responsibilityId

string

Responsibility ID

timestamp

string

The timestamp of resource creation (UTC).

resourceStatus

ResourceStatus

Quick assessment status.

displayName

string

Quick assessment display name.

description

string

Description of quick assessments.

remediationLink

string

Link to remediation steps for this quick assessment.

Share via

Keep Microsoft 365 Application Compliant Continuously with App Compliance Automation Tool for Microsoft 365

Get the latest control assessment of your compliance report via notifications

Get notifications with webhook

Get quick compliance evaluation in Continuous Integration/Continuous Deployment pipeline

Integration with CI/CD pipeline by GitHub Actions

Integration with CI/CD pipeline through ACAT REST API

Reference

Webhook payloads

ACAT REST API triggerEvaluation

API Name: triggerEvaluation

URI Parameters

Request Header

Request Body

Responses Header

TriggerEvaluationResponse

QuickAssessment

Additional resources