Audit Microsoft 365 Copilot interactions with Microsoft Purview

6 minutes

Microsoft 365 Copilot is an AI tool that enhances productivity in Word, Excel, PowerPoint, Outlook, and Teams. It uses large language models, including GPT-4, for tasks like writing and summarizing. By connecting to Microsoft Graph and accessing authorized emails, chats, and documents, Copilot offers context-based assistance. Its functions include drafting documents, creating PowerPoint presentations, summarizing emails, and providing real-time help in Teams meetings. This broad application raises data governance and security concerns.

Compliance and security in Microsoft 365 Copilot

Managing Copilot's AI functions within Microsoft 365 is critical for data governance compliance. As Copilot interacts with various Microsoft 365 data, it's important to monitor these interactions for compliance and security. Microsoft Purview is the essential auditing tool for this purpose, ensuring transparency, data security, and adherence to ethical and legal standards. Effective use of Microsoft Purview Audit is key to maintaining the integrity of data within Microsoft 365 environments.

Microsoft Purview Audit overview

Microsoft Purview Audit searches and analyzes activities in Microsoft 365. It records user and administrator actions across services, storing them in a unified audit log. This log can be searched via the compliance portal or the Search-UnifiedAuditLog cmdlet for specific activities, users, or time frames. Available in Standard and Premium versions, it offers an overview of organizational operations, essential for compliance and security. For those managing AI tools like Microsoft 365 Copilot, it's a key resource for ensuring compliance with organizational policies and regulatory requirements.

Search the audit log for Copilot interactions

Microsoft Purview Audit's ability to searches Copilot interactions is critical for compliance management. It covers Copilot's integration into applications like Word, Excel, PowerPoint, Teams, Loop, Whiteboard, OneNote, and Microsoft 365 Chat. The audit records identify Copilot interactions by the app in which they occur, providing detailed insights into Copilot usage across different contexts.

Prerequisites for using Microsoft Purview Audit to search Microsoft 365 Copilot interactions

Before you search and analyze Copilot interactions using Microsoft Purview Audit, there are a few steps to ensure your environment is ready. Follow these prerequisites to set up your Microsoft 365 and Purview Audit configurations:

Step	Description	Learn more
Verify prerequisites for Copilot	Ensure your IT infrastructure is ready for Copilot and Audit, including necessary network configurations and software updates.	- Microsoft 365 Copilot requirements
Understand searching with Audit	Understand the search functionalities in Microsoft Purview Audit to effectively analyze activities within Microsoft 365.	- Audit New Search
Check licensing requirements	Confirm that you have the appropriate Microsoft 365 E3/E5 licenses for Copilot and Microsoft Purview Audit.	- Microsoft 365 Copilot service description - Microsoft Purview Audit service description

Note: Microsoft Purview Audit logging is turned on by default, but when setting up a new Microsoft 365 organization, you should verify the auditing status for your organization. If auditing isn't turned on for your organization, you can turn it on in the compliance portal or by using Exchange Online PowerShell. For more information on verifying that auditing is enabled and enabling the audit sign in Microsoft Purview, see Turn auditing on or off.

Search the audit log for Copilot interactions in Microsoft Purview

When auditing Copilot interactions, Microsoft Purview Audit captures detailed events including user interactions with Copilot, the Microsoft 365 service where the activity occurred, and references to any accessed files stored in Microsoft 365. If these files have sensitivity labels, this information is also recorded. To search for these interactions:

Sign into the Microsoft Purview compliance portal.
Select the Audit tab on the left panel of the homepage to navigate to the Audit tool.
Select New Search tab at the top of the Audit page.
Configure your search on the New Search tab:
1. Set the Start date and End date for your search, with the last seven days selected by default.
2. Enter relevant keywords or phrases in the Keyword Search, using asterisks (*) to replace special characters.
3. Select administrative units from the Admin Units dropdown if needed.
4. Under Activities - friendly names select specific activities relevant to Copilot by navigating to Copilot activities and selecting Interacted with Copilot. You can also use the search bar to find activities related to Copilot by entering Copilot.
5. For precise searches, use Activities - operations names and enter CopilotInteraction as the operation name for Copilot activities.
6. In the Record types dropdown, select record types linked to Copilot activities. Enter Copilot in the search box above the list for easier selection.
7. Name your search in the Search name field for easy identification.
8. Enter specific users in the Users field or leave it blank to return entries for all users (and service accounts) in your organization.
9. Enter File, folder, or site names for targeted searches, or leave this box blank to return entries for all files and folders in your organization.
Select Search to start your search job. A maximum of 10 search jobs can be run in parallel for one user account. If a user requires more than 10 search jobs, they must wait for an In progress job to finish or delete a search job.

Limitations and considerations for auditing Copilot interactions

When implementing compliance management solutions for Copilot in Microsoft 365, it's important to be aware of the limitations and considerations:

Scope of auditing: Auditing captures the occurrence of Copilot activities, such as search events, but doesn't record the actual user prompts or responses. For detailed interaction data, eDiscovery tools should be used.
Admin-related changes exclusion: Changes related to Copilot administration, such as configuration adjustments, aren't currently captured in the auditing logs.
Device Identity Information: Device identity information, which can be important for comprehensive auditing, isn't included in the audit details for Copilot activities.

Application-specific limitations:

Copilot in Teams: When transcripts are turned off, auditing capabilities for Copilot interactions aren't supported. Actions involving referencing transcripts aren't captured in the audit logs.
Identification of Source App: The source of Copilot interactions is identified by the app name in the audit logs, such as Copilot in Word or Copilot in Teams.