Maturity Model for Microsoft 365 - AI & Cognitive Business Competency

Note

This is an open-source article with the community providing support for it. For official Microsoft content, see Microsoft 365 documentation.

Overview of the Concepts [tl;dr]

Artificial intelligence (AI) and machine learning (ML) are rapidly evolving fields, with new breakthroughs, services, and capabilities seemingly every week.

In many ways, this revolution is following a similar track to other world changing technologies, creating opportunity, threat, FUD, excesses of imagination and paranoia and more. While the technology and its monetization are fearsomely rapid, forward-thinking organizations should be anticipating the roles of various forms of cognitive business and incorporate these into their culture, ways of working, processes and thinking. While pessimists may be concerned about these technologies displacing ‘white-collar’ / knowledge worker roles, pragmatic organizations will be considering the impact of the changes and planning how they can be integrated into their cognitive functions – knowledge creation and access, analysis and assessment, task planning and decision making.

Definition of this competency

Cognitive businesses are those that actively employ thinking in its many forms across operational activities and decision making. They are characterized by having significant numbers of 'knowledge worker' staff in operational roles, not just management roles.

This Cognitive Business Competency considers how and where organizations might deploy advanced, computer based cognitive services across their teams and operations.

The technologies used will encompass (narrow and generalized narrow) AI, ML and other advanced technologies that simulate or mimic human intellectual, analytical and creative activity. For convenience, these will be referred to simply as AI in the rest of this article; from a Microsoft 365 perspective, we also will refer to Copilot, in its many forms. There are several hundred different Copilots in development across Microsoft 365.

The technology scope includes pre-built AI, including those integrated into off the shelf products and suites, ‘invisible’ technologies that don’t provide an apparent user interface, direct human-machine interfaces (such as voice assistants & chat bots), AI platforms and services and custom-built engines for the using organization’s own data. The competency also acknowledges the continued and essential role of human cognition and the relationship between how people and AI work together.

Cognitive Business maturity describes the extent to which organizations have understood, adopted and embedded AI-related capabilities in the right combination to improve and, ultimately, optimize the business.

Terminology

There are several new terms used within the AI space and these will be described briefly to help the digestion of this competency.

Retrieval Augmented Generation (RAG) is used to improve the output of a large language model. It is the process of taking a user's input and using it with an external knowledge source to provide relevant information. It can be applied to natural language processing tasks such as question/answering, summarization and dialogue generation. The main reason is to use the external knowledge source to obtain the most relevant information and then use the large language model to process the response and give a more informative and digestible response to the user. For more information, read Retrieval Augmented Generation in Azure AI Search.

Where are we today?

Cognitive business and AI developments are primarily in the Enthusiasts and Visionaries stages. As noted, AI-related technology is changing quickly and use of this competency should reflect that.

ChatGPT has this to say about Cognitive Business:

"Cognitive business is a term used to describe the integration of cognitive computing technologies, such as artificial intelligence (AI), machine learning, natural language processing, and robotics, into various business processes and decision-making frameworks. It involves leveraging the power of these technologies to enhance business operations, automate tasks, and gain insights from data to drive innovation, improve customer experiences, and increase efficiency.

The goal of cognitive business is to create a more intelligent and adaptive organization that can quickly respond to changing market conditions and customer needs. By using cognitive technologies to analyze vast amounts of data and make better decisions, businesses can gain a competitive edge and create new opportunities for growth.

Cognitive business can be applied across a range of industries, including healthcare, finance, retail, manufacturing, and more. Examples of cognitive business applications include chatbots and virtual assistants for customer service, predictive analytics for supply chain management, and autonomous vehicles for transportation and logistics."

Evolution of this competency

There are many aspects to cognitive business, but an important consideration is how cognitive business enables people and 'intelligent machines' to work together. The maturity model considers this alongside the deepening capabilities, integration, and governance of AI in organizations.

As Satya Nadella has said, AI is moving from Autopilot to Copilot:

See the Maturity Model for Microsoft 365 - Introduction for definitions of the Maturity Model levels. Some characteristics should, perhaps, be addressed a little more urgently than others; we have marked these with the 'Sparkles' emoji: ✨

Level 100 - Initial

Organizations at this level are unaware of, uninterested in or dismissive of AI supported cognitive business. They rely on their staff to maintain their competitiveness and do not perceive an opportunity or threat to the way they currently operate. Knowledge worker tasks are delivered by staff.

Initial level characteristics include:

100 General

• AI is not knowingly used. It may be unknowingly used in standard products and services.
• Internal technology teams are not aware of or have not evaluated any the add-on services which provide AI capabilities, like Copilot.
• Organizational content is not centrally stored and available for AI ingestion.
• There is no strategy for a cognitive business beyond the capabilities of employees, nor is there any intent to explore it. The cognitive nature of the business is not considered.
• AI may be looked on with disdain, as a passing fad or as unable to outperform members of staff.
• People are doing almost all the cognitive activities across the business, perhaps supported by 'ordinary' technology. These may be efficient and effective, and, as such, they see no reason to change.

100 Technology

• AI embedded in applications and suites is not noticed or used by most staff; no attempt is made to adopt, adapt, or explore it.

100 Governance, Risk, Compliance and Security

• No consideration is given to governance or risks associated with cognitive business activities. Staff are unaware of intellectual property (IP) implications where AI tools are used.
• Some part of the organization may be using unsanctioned AI capabilities in multiple platforms, leading to a lack of data sovereignty.

100 Impacts

AI has no direct, measurable impact, which may mean that disruption, when it comes, is unanticipated. Competitive advantage arises through investments in the skills or staff; human limitations may create intractable challenges.

100 Next Steps

• Inventory - as best as possible - the ways people in the organization are using AI functionality.
• Begin to identify some simple use cases to use for testing and piloting cognitive business capabilities.
• Identify processes with significant repetition where human action could be replaced by AI. At the same time, identify instances where content generation might get a jump start from using AI.
• Begin evaluating the AI capabilities which are available in in Microsoft 365.

Level 200 - Managed

There is some appreciation of the emergence of potent cognitive tools and there may be some experimentation with narrow forms of these to enhance processes or address needs, mostly through the configuration of platform-based services and some use of proprietary data for training. This is Proof of Concept work, with limited process design and little to no consideration of governance and risks initially. Most staff do not have an accurate understanding of the capabilities and limitations of the tools and there are unreasonable expectations of perceptions of risk.

Managed level characteristics include:

200 General

• AI to support a cognitive business is speculated about, often with unreasonable or incorrect assumptions about what it can and cannot do.
• ✨ Staff and management are largely unaware of the AI tools built into their everyday off-the-shelf tools. Where they do, they use it to get answers to questions and copy the answers verbatim.
• There is no strategy for introducing AI, though some staff, in isolation, may identify possible areas where existing processes might be improved using AI approaches. A few staff experiment with the tools that are readily available, perhaps using in-house data; management support for this is limited or may be occurring without their knowledge.
• There is no plan for who should get what licenses, subscriptions or tools.
• ✨ There is a tendency to think that technology can solve the problem, without rigorously defining the needs. Where cognitive tools are used, they are taken at face value, without critical assessment or integration into business processes and staff response.
• Staff with knowledge about how to build cognitive models, create appropriate training sets or choose better AI processing models and methods, are largely absent.
• Content is vital to successful Generative AI solutions and at this level, there will be parts of the organization that are not all-in with their content not in the cloud. They therefore cannot get the value from Embedded AI. These parts of the organization will start ingesting content into the cloud platform so that they can take advantage of Embedded AI tooling.
• Source and training data is not well structured, clean and of sufficient quality and there are few metrics for the quality of this.

200 Technology

• Content and document management uses some automated classification, though largely with the default configuration and limited training on topics etc. Categorization may be used to improve existing processes, often in a narrow way, usually with out-of-the-box capabilities.
• AI may be in active use by the platforms used in the organization, such as protecting against security risks or supporting web searches. Only a few people recognize that this is the case.
• ✨ No technology-stack choices have been made and different platforms and services are being played with different parts of the organization at different depths.
• Cognitive capabilities are not designed into applications, products and services.

200 Governance, Risk, Compliance and Security

• ✨ There may be instances of content that is overshared and therefore the AI has access to information that it should not have. The organization will start to address this when these are found.
• ✨ Training data sets are used as-is, without considering ethics, bias or errors that may result from the data.
• Limitations are not considered, and no safeguards are in place to correct poor AI decisions.
• Intellectual Property implications are not considered.
• Understanding the costs of implementing Cognitive Business tools.

Level 200 Impacts

At this level, Cognitive business remains the domain of staff, with a few ‘mavericks’ extolling the virtues of AI and playing with the capabilities, usually with no oversight by technology, governance or management functions. AI is not contributing to productivity other than through ‘invisible’ AI that is built into off-the-shelf applications and services.

200 Next Steps

To be able to move towards Level 300, there must be sufficient people bought into the competency. To do this, think about the low-hanging fruit where Generative AI can help your organization to transform quickly. Look at departments which provide services to the rest of the organization such as Human Resources. Use Embedded AI to address self-service needs and 'quick-wins'. Share the 'quick-wins'; and success stories across the organizations to show the value of Cognitive Business solutions. Additionally, to be able to move to Level 300, your data needs to be accessible by the Embedded AI models and secured to ensure that the right people have access to the right information when using AI.

Level 300 - Defined

At level 300, the organization is actively considering how to enhance its knowledge worker activities using AI. While the scope of this may be narrow, with point solutions rather than transformative projects and adoption, there is an outline strategy and intent to enhance the organization through cognitive services. Staff use AI to support their knowledge worker activities, taking advantage of built-in tools and some standalone services to assist their day-to-day activity, while the organization begins to research or invest in bespoke business solutions.

Defined level characteristics include:

300 General

• Staff are trained in embedded AI tools in general applications (office apps, browsers, desktop etc.) and encouraged to use them productively. Training includes a requirement to check for veracity and 'real-person' language.
• There is some attempt to manage who gets what tools and licenses; this is not especially rigorous or monitored; some staff who would benefit may struggle to justify license and training investments, while others (often in positions of influence) may be allocated these resources without robust justification.
• There is some understanding that AI benefits can be around productivity gains, new capabilities, quality improvement, levelling up of less capable staff, enhanced accessibility and cost reduction. There are attempts to assess the nature and impact of the benefits of each of these in AI projects or allocation of tools and licenses.
• AI use tends to be on a per-application basis; staff don't routinely think about multi-application interactions (for example, using email content to create a presentation).
• Chat-based AI for answering questions about the organization are embedded in intranets, collaborative workspaces and elsewhere the staff spend their time to ensure they can obtain answers rapidly.
• ✨ Attempts are made to ensure source and training data for custom AI is well structured, clean and of sufficient quality. The quality of the training input to cognitive models is managed, with attempts to minimize bias and errors. A set of metrics is established to confirm this.
• ✨ Staff have some understanding of how to ask questions of the AI tools to get useful outputs.
• The limitations of the insights, knowledge and behaviors of people as the benchmark for AI ‘accuracy’ are considered.
• ✨ The organization has laid out a broad strategy for AI setting out their aspirations. Elements of this may be naïve, lacking actionable detail and measurable objectives, and may lack resources and senior sponsorship, but it acts as an important starting point and touchstone.
• The technology is not treated as internally or externally disruptive.
• Built-in AI tools in applications and services are actively used by staff.

300 Technology

• A range of AI services are used to improve existing processes, with multiple areas of improvement. Mostly these improve human-driven knowledge activities and support existing staff, however some areas no longer require human intervention.
• Content categorization is used to improve existing processes, often in a narrow way, usually with out-of-the-box capabilities; often this makes things easier for staff later in the process.
• ✨ Content and document management actively uses automated classification, configured and trained with the organization's information and document set.
• Most developments are focused on the application of pre-trained models. There are some tools developed using existing narrow organizational content to address narrow needs, creating Narrow Language Model solutions.
• True custom AI models, using data sourced or created for key organizational purposes, are beginning to be used and the effort and effectiveness are being evaluated for wider application.

300 Governance, Risk, Compliance and Security

• ✨ It is understood that AI needs some level of governance and oversight. This activity is probably vested in the IT team and is not aligned with broader business strategy and policies.
• The organization develops and models rules of engagement and governance to take advantage of the benefits of the cognitive systems.
• Governance of all aspects of the cognitive tools is proportionate to the impacts the system has on business and individuals.
• ✨ There is a content governance policy implemented to secure sensitive content and eliminate out-of-date content in order to ensure that AI using content in general business repositories provide appropriate and current responses.

Level 300 Impacts

At this level you can expect the following:

Staff have use public tools, such as AI-driven internet search, saving time allowing them to focus on more creative and valuable activities. Some processes are improved through AI and an appetite has developed for doing more, with some limited funding. AI supports staff in their cognitive tasks. Risks and concerns are emerging that the organization is unsure how to address.

300 Next Steps

To progress to level 400, the organization should work on driving adoption, creating an understanding of effective prompts and sharing success stories to drive productivity. In parallel developing critical thinking and expertise around AI prompts and interactions in key staff, across cross-functions and across multiple apps Effective practice, skills and learning need to be identified and shared together with their benefits. This might be combined with enhanced training.

The organization would publish and enforce a Responsible AI policy and framework. There should be processes for capturing new AI development ideas. This might include where AI might be allowed agency to act without human initiation, but with oversight. Staff will be expected to use AI tools and productivity gains will be measured.

Level 400 - Predictable

There are processes to build, deploy, integrate, and manage AI alongside staff in many areas of the organization. A clear process exists to accelerate AI integration into existing processes, along with budget commitments. There are mechanisms to manage and monitor AIs that are similar in importance and effect to those used with staff. Feedback processes and performance metrics drive improvements. The organization’s culture accepts and understands the role of AI and the relationship between staff and their AI tools. The organization uses its skills as a flexible cognitive business to create efficiency and business advantage.

Predictable level characteristics include:

400 General

• Staff are trained and competent in interacting with tools to optimize their useful outputs.
• There is a well defined process for allocating AI licenses, with training, to appropriate staff. Use and benefits are routinely assessed, with intervention and reallocation as required.
• ✨ The different potential benefits and costs or AI are well understood and underpin investments and licensing decisions. These align with strategy and culture, not just cost and productivity. Value extraction from AI use is well understood.
• Policies exist and are widely understood regarding AI transparency, ethics, performance, and scope. These policies are regularly reviewed and updated in recognition of the pace of change of the technology and regulatory environment.
• ✨ AI services frequently work alongside human staff, with each complementing the other. Areas that do not require human intervention have human and AI oversight, with both reviewing feedback.
• ✨ AI that replaces staff roles have line management processes, performance reviews, code of conduct guidelines, etc., that perform the equivalent role to the staff they replace.
• There is board level oversight of the cognitive business approaches, ensuring they support the organizations values, ethics, and strategy.
• Cognitive AI outputs are routinely audited and subjected to quality control in the same way as other quality processes. Methods have been developed to validate that training, queries and other inputs produce 'correct' outputs.
• Cognitive business approaches enable the activities that could not have been done without the tools. Processes are transformed rather than simply enhanced.
• The level of trust in the tools is understood, continuously re-evaluated and deficiencies addressed.
• ✨ The limits of AI are well understood, and ‘unusual’ cases are handed off to experts. There is a coherent approach to people and AIs working together, with well-defined hand-offs.
• Source and training data are actively managed to ensure quality, with metrics and active feedback.
• Expert knowledge workers compile content covering their knowledge so that their AI partners can access this and use it to personalize the content they create for individuals

400 Technology

• ✨ The cognitive business landscape is scanned, and changes and improvements are previewed and incorporated into the business roadmap; the tools are actively 'upskilled' as technology advances.
• ✨ Sources and training data are robust, updated, assessed and managed against quality and ethics standards.
• Technology limitations are well understood; safeguards and feedback loops are in place.
• Voice interfaces, natural language processing and other human-centric UIs are present across staff workspaces. There is some use of 'always-on; monitoring within the workplace and process areas.
• New applications developed within or for the organization actively incorporate cognitive elements, and these incorporate AI ethics and governance by design. Budget commitments are in place to support development using AI tools.
• ✨ Development using custom AI models have a well-established process which includes data quality security, responsible use, and audit.
• A generalized narrow AI, capable of performing many different types of tasks and with a holistic view of the organization, starts to replace many discreet cognitive services.
• AI is capable of identifying issues and carrying out auto remediation, handing off to a person where necessary.

400 Governance, Risk, Compliance and Security

• ✨ There are policies that define how cognitive business should be introduced, assessed, performance managed and monitored for effectiveness and fairness. Responsible AI initiatives and standards form the basis for this.
• The implications of compliance around AI use are broadly understood and actions are taken to minimize risks relating to regulations, Human Resource obligations etc. A board level role has accountability for responsible, ethical and fair application of AI, ensuring compliance with regulations and values.
• ✨ Training in Cognitive business for staff, management and the leadership team are maintained. This ensures understanding of ethics, compliance, best practice and drives trust. Assessment is used to improve the training and identify staff competency.
• Training data is reviewed regularly for historical bias and gaps that might compromise the ethics of the AI. There is special care taken with externally sourced and public data that may include such bias.
• Attention is given to national and organizational culture and how this might create bias in the cognitive business. Systems are reviewed against clearly stated values and principles in place in the organization.
• There are processes to hand off ethically complex issues or outliers from the systems to human arbiters.
• Inputs to cognitive systems are recorded so that they can be used to validate outputs.
• The risks of cognitive business tools deskilling staff are understood and addressed appropriately, such as through actively retaining skills or accepting that these are lost to the organization and external expertise is used when required.

Level 400 Impacts

At this level you can expect widespread adoption and acceptance of AI-based tools in many areas of an organization and that these create good competitive advantage. Staff are engaged with the tools and use them to great effect, whilst also watching for unexpected behaviors or under-performance.

Staff and processes are productive, though there may be concerns about resilience of the technology and exposure to regulatory change.

Level 400 Next Steps

• Many AI-supported processes may be at a departmental level, so consider places where AI can monitor other AI-driven processes
• Think of ways to use AI to monitor metrics and make process suggestions in real time
• Capture knowledge gained in cognitive business work and contribute to the organization's knowledge bases

Level 500 - Optimizing

Cognitive business has reached the stage of being a natural, continuous flow of interaction between staff and machines. Ethical and GRC issues are effectively monitored against a well-established framework and feedback, retraining and horizon scanning maintain this. Highly capable AIs interface with most parts of the organization and are capable of a large range of tasks working in collaboration with human staff.

Optimizing level characteristics include:

500 General

• Resources and capabilities in the organization are sufficient to drive rapid and effective cognitive business value. This is supplemented by a broad range of partners and associates with expertise in specific applications of cognitive business. The organization uses these to both rapidly react to changing needs and proactively advance the business in response to strategy and vision.
• Role and activity analysis identifies optimal allocation of AI tools and licenses to staff, tracks impacts and outcomes and initiates training, upgrades and changes in allocation across the organization.
• The cost-benefit impact of the spectrum of AI use is understood and tracked across the organization, allowing nuanced changes in line with strategy and values. Broad AI use is seen as a strategic asset and differentiator.
• ✨ Content and document management actively uses automated classification, configured and trained with the organization's information and document set, with continuous retraining and active redesign to incorporate future-looking strategic and tactical changes in the business
• AI assistants, versed in the knowledge and processes of the organization, are available to all staff to assist with their activities. They also have ‘a seat at the board table’ where they can retrieve relevant information, capture decisions and actions and provide summaries and feedback on the state of both the organization and previous meetings.
• AIs are used in the board room to advise directors/VPs, capture and track decision and actions and summarize and analyze information
• Advanced human-machine interfaces are in careful use, which may include active monitoring of conversations and activities, prospective advice and insights.
• ✨ Cognitive business is built into the organization by design. It pervades staffing criteria, product and process development, sales and marketing strategy and pervades operations.
• ✨ There are active processes for personal and organizational knowledge transfer to AI assistants and partners. This ensure the AIs can use expertise embodied in people and the organization to create more informed content, take better decision and act on behalf of people in the system. This may involve something akin to AI Apprenticeships.

500 Technology

• State of the art technologies are proactively reviewed and incorporated into the Cognitive Business strategy and roadmap. AI services may be used to help identify these.
• ✨ Active experimentation takes place, and the learning is used internally and shared with partners and aligned vendors to drive future improvements.
• The generalized narrow AI has reached a level of capability that exceeds the sum of the parts and becomes a core strategic platform specific to the organization. Most discreet cognitive services are now incorporated into this service, increasing its reach and reducing the overhead of training and integrating individual tools.
• Cognitive systems exhibit proactive interfaces; not just reactive. They will prompt humans as and when appropriate. The use of such systems follows policy and values, with appropriate safeguards.
• ✨ AIs are routinely capable of autonomous action. Where they make mistakes or choose to involve a person in the decision making, the output of this is fed back into the AI to improve it.

500 Governance, Risk, Compliance and Security

• Innovations and inventions that both drive Cognitive Business and result from it are effectively protected and secured; this includes IP, the data they access, the outputs they generate and the code and models they rely on.
• ✨ People and AIs work together harmoniously as a team. Feedback loops and oversight ensure this remains effective, upskilling both as required.
• AI approaches are used to review the performance of other AIs, of data quality and identify anomalies in data and behaviors. These might use SPC or other statistical tools
• Cognitive systems document their decision-making process, to a standard comparable with people being held to account for their accounts and decisions, enabling trust in the judgments being made.
• ✨ Cognitive processes don’t just rely on pre-training, but also embed iterative feedback to continually enhance the quality. This includes ‘Chain of Thought’ prompting. (The user prefixes their question with text that includes a couple of examples of questions and solutions, including the reasoning — illustrating a typical chain of thought — that led to the answers).
• ✨ Proprietary cognitive systems and 3rd party systems all comply with Responsible AI initiatives and standards.
• Designers of cognitive systems are held to account and assessed on their decisions to minimize and control bias.
• The outcomes from hand-offs of ethically complex issues or outliers to human arbiters are then used to retrain and improve the system.

Level 500 Impacts

At this level the organization fluidly adopts AI alongside staff in order to excel at many activities. Productivity is high and staff are given significant amounts of time to be creative in exploring how to further improve the organization, with a range of cognition tools to aid them. Staff are happy with the nature of the interactions with their AI counterparts and with the improvements to their work and quality of life.

Common Microsoft 365 Toolsets

• Azure OpenAI
• Azure Cognitive Services
• Azure Machine Learning
• Azure Bot Framework
• Azure AI Infrastructure
• Azure Monitor: Network, Applications, and Infrastructure Monitoring
• Azure Sentinel
• Bing / ChatGPT
• Microsoft 365 Copilot
• GitHub Copilot
• Microsoft 365 (embedded AI elements)
• Microsoft Power Platform
• Microsoft Teams
• Microsoft Syntex
• Microsoft Viva
• Microsoft Sentinel: Intelligent Security Analytics
• Microsoft Defender Threat Intelligence
• Information Protection
• Microsoft Purview Information Protection
• Microsoft Purview Data Lifecycle Management
• Microsoft Purview Data Loss Prevention

Resources

• MS Responsible AI Principles
• Responsible AI Maturity Model
• MS - Governing AI: A Legal and Regulatory Blueprint for the Future
• Responsible AI

Tip

Join the Maturity Model Practitioners: Each month we host sessions exploring the value and use of the Microsoft 365 Maturity Model and how you can successfully develop your organization using Microsoft 365. Each of these sessions focuses on building a community of practitioners in a safe space to hone your pitch, test your thoughts, or decide how to promote your use of the Maturity Model. Sessions include a presentation on a topic about the Maturity Model, including recent updates. Calendar link

---

Principal authors:

• Simon Hudson, MVP

---

The MM4M365 core team has evolved over time and these are the people who have been a part of it.

Core team

• Marc D Anderson, MVP
• Simon Hudson, MVP
• Simon Doy, MVP
• Sharon Weaver
• Galen Keene
• Pia Langenkrans
• Mats Warnolf

Emeritus

• Sadalit (Sadie) Van Buren
• Emily Mancini, MVP, UXMC

---