Share via

Measure Databricks Assistant impact

  • Article

This article provides information on measuring the impact of Databricks Assistant in terms of adoption, egagement, and reported productivity gains.

Requirements

In order to measure the impact of Databricks Assistant, you need:

  • Account admin privileges required to enable system tables. See Enable system tables.
  • An internal survey to get subjective feedback on Assistant from your team.

Data vs. metrics

You get raw data from the system tables and a survey. To understand Assistant impact, you must analyze and report data as metrics. Metrics are calculated values you use to measure specific aspects or activities related to Assistant impact. This article also refers to metrics as measures.

Tips for measuring Assistant impact

To understand how your organization is using Databricks Assistant, start by measuring its adoption and engagement with Databricks Assistant. This data can be calculated from system tables.

Review your data routinely and make it easily available in a shared dashboard. For a dashboard example and template, see Databricks Assistant system table reference and example.

Assistant impact metrics

The following are recommended measures of Assistant impact, both from system tables and from user feedback. For examples of metrics calculations, download the Assistant events dashboard file from GitHub and read the calculations in JSON. To learn how to import a dashboard file, see Import a dashboard file

Surveying your organization also helps you understand the effectiveness of its engagement with Assistant. See Recommended survey questions.

Measure Definition Stage Data source
Top users overall Users in a given period who interact most frequently with Assistant Adoption Calculate from Databricks Assistant system table data.
Submissions data: per workspace and total Number of requests submitted to Assistant per workspace and per account Adoption Calculate from Databricks Assistant system table data.
Active users by day and month Unique users who have received and accepted 1+ suggestions or participated in 1+ chats on a given day. Engagement Calculate from Databricks Assistant system table data.
Active users per workspace Unique users in a given workspace who interact with Assistant Engagement Calculate from Databricks Assistant system table data.
Top job roles using Databricks Assistant Numbers of people from each role identified in your org who use Assistant Engagement Survey of your organization
Top tasks for Assistant Most common tasks Assistant helps with Engagement Survey of your organization
How often do you use Assistant Self-reported frequency of Assistant use per user Engagement Survey of your organization
Top areas of Assistant use Self-reported usage areas: SQL editor, notebooks, or both Engagement Survey of your organization
User satisfaction with Assistant help Self-reported satisfaction with Assistant’s answers on a scale of 1-5 User satisfaction Survey of your organization
Gains in user productivity Self-reported increase in productivity gained using Assistant on a scale of 1-5 User satisfaction Survey of your organization
Amount of time users save using Assistant Self-reported percentage of time saved by using Assistant to complete tasks User satisfaction Survey of your organization

System tables won’t capture all information about how your organization uses Databricks Assistant. Use survey questions, such as the ones below, to understand how your organization uses Assistant.

  • What is your role? List roles in your organization
  • How often do you use Assistant?
    • Hourly
    • Daily
    • Weekly
    • Monthly
  • Where do you use the AI Assistant?
    • SQL editor
    • Notebooks
    • Both
  • What are your main use cases for Assistant?
    • Fixing errors / troubleshooting
    • Help writing code
    • Code refactoring
    • Writing SQL queries
    • Writing unit tests
    • Looking up documentation
  • How satisfied are you with Assistant’s answers? Scale of 1-5, Not satisfied to Extremely satisfied
  • Since using the AI Assistant, would you say you’re more productive? Scale of 1-5, Much less productive to Much more productive
  • How much time do you believe you save on an average task using the Assistant?
    • 0-10%
    • 10-30%
    • 30-50%
    • 50-70%
    • More than 70%