Share via

Analyze your copilot's customer satisfaction metrics (preview)

  • Article

The Customer Satisfaction tab of the Analytics page provides a detailed view of customer satisfaction (CSAT) survey data, including the average CSAT score, primary user query themes, and actionable insights on drivers of satisfaction or dissatisfaction with your copilot's responses.

By default, the page shows key performance indicators for the last seven days. To change the time period, use the date pickers at the top of the page. You can retrieve data for any period within the last 45 days.

The Customer satisfaction page.

Customer satisfaction score

The Customer satisfaction score chart provides a graphical view of the average of CSAT scores for sessions in which customers respond to an end of conversation request to take the survey. The CSAT survey asks customers to rate their experience on a scale of 1 to 5. If an end user responds to more than one survey in the same session, only the most recent is used.

This chart also provides a period-over-period change indicator. For example, when you select a three-day period, the indicator shows the percentage change relative to the three days before the selected period. The period-over-period indicator only appears if your copilot has CSAT survey data available for the prior period. If there are no CSAT survey data available for the same trailing period of time relative to what is selected in the filter, then the period-over-period indicator doesn't appear.

CSAT survey response rate

The CSAT survey response rate chart shows the number of end of conversation CSAT surveys that were presented and the percentage of surveys that were completed.

Customer satisfaction breakdown

The Customer satisfaction breakdown chart shows that percentage of sessions that were satisfied, dissatisfied, or neutral in the selected time period. The Customer satisfaction status pane provides more detail on the various signals used to determine session satisfaction status.

Customer satisfaction status

The Customer satisfaction status chart provides key insights on the themes users search for, and the satisfaction level of the users about the copilot's responses. Sessions with similar themes are grouped together. The chart shows the number of sessions for each theme during the selected period, and the percentage of these sessions that were satisfied or dissatisfied. Sessions that weren't satisfied and not dissatisfied are considered neutral sessions and don't appear on this chart.

The theme of a given session is derived using ML models. Before being sent to the analytics dashboard, themes are processed to remove any personal data or sensitive information, such as phone numbers. Additionally, if themes contain profane or harmful language, these themes are masked.

To see the specific satisfaction or dissatisfaction drivers, hover over each segment of the chart. If any of the criteria are true for a given session, then the session is classified as satisfied or dissatisfied:

  • A session is considered dissatisfied if:

    • The user gave two stars or less at the end of conversation survey.
    • The user was asked to rephrase their query twice or more than twice in the (system fallback topic).
    • The user abandoned the session.
    • The user escalated the session to a live agent.
    • The overall sentiment of the user about their conversation with the copilot is classified as negative. Sentiment is determined using a publicly available ML model fine-tuned for sentiment analysis.

  • A session is considered satisfied if:

    • The user gave four stars or more at the end of conversation survey.
    • The user wasn't asked to rephrase their query more than once in the (system fallback topic).
    • The session was resolved.
    • The overall sentiment of the user about their conversation with the copilot is classified as positive.

Sessions that don't meet any of the above criteria are considered neutral sessions and don't appear on this chart.

On the Summary tab of the Analytics page, you can use the Information icons to learn more about engagement, escalation, abandon, and resolution rates.

Theme and session sentiment extraction

Copilot Studio uses natural language processing (NLP) techniques to extract themes and assign sentiment to a given copilot session.

For every session, Copilot Studio extracts themes from the first user utterance. Individual sessions with similar themes are aggregated, and appear as a single item on the Customer satisfaction status chart.

To evaluate session sentiment, the underlying NLP model is trained on public English language datasets. This process analyzes the text of the session to determine if the overall sentiment is positive, negative, or neutral. The process also preprocesses user queries to remove false positives. For example, this preprocessing ensures that a query such as "what is the best option?" isn't classified as positive solely because the word "best" appears in the query.