Progressive experimentation with feature flags

As DevOps teams shift to an Agile methodology that focuses on continuous delivery of features, the need to control how they become available to users becomes increasingly important. Feature flags are a great solution for limiting user access to new features, either for marketing purposes or for testing in production.

Decoupling deployment and exposure

With feature flags, a team can choose whether a given set of features is visible in the user experience and/or invoked within the functionality. New features can be built and deployed as part of the ordinary development process without having those features available for broad access. The deployment of features is conveniently decoupled from their exposure.

Flags provide runtime control down to individual user

Flags also provide granular control all the way down to the individual user. When it's time to enable a feature, whether for one user, a small group, or everyone, the team can simply change the feature flag in order to light it up without having to redeploy.

The scope of a feature flag will vary based on the nature of the feature and the audience. In some cases, a feature flag will automatically enable the functionality for everyone. In other cases, a feature will be enabled on a user-by-user basis. Teams can also use feature flags to allow users to opt in to enable a feature, if they so desire. There's really no limit to the way the feature flags are implemented.

Support early feedback and experimentation

Feature flags are a great way to support early experimentation. Some features can have rough edges early on, which may be interesting only to the earliest of adopters. Trying to push those not-quite-ready features onto a broader audience could produce dissatisfaction. But the benefit of gathering feedback from users willing to deal with an in-progress feature is invaluable.

Quick off switch

Sometimes it's helpful to be able to turn something off. For example, suppose a new feature isn't working the way it was intended, and there are side effects that cause problems elsewhere. You can use feature flags to quickly turn off the new functionality in order to roll back to trusted behavior without having to redeploy. While feature flags are often thought of in terms of user interface features, they can also easily be used for changes in architecture or infrastructure.

Standard stages

Microsoft uses a standard rollout process to turn on feature flags. There are two separate concepts: rings are for deployments, and stages are for feature flags. Learn more about rings and stages.

Stages are all about disclosure or exposure. For example, the first stage could be for a team's account and the personal accounts of members. Most users wouldn't see anything new because the only place flags are turned on is for this first stage. This allows a team to fully use and experiment with it. Once the team signs off, select customers would be able to opt into it via the second stage of feature flags.

Opt in

It's a good practice to allow users to opt in to feature flags when feasible. For example, the team may expose a preview panel associated with the user's preferences or settings.

Use flags with telemetry

Feature flags provide a way to incrementally expose updates. However, teams must continuously monitor the right metrics to gauge readiness for broader exposure. These metrics should include usage behavior, as well as the impact of the updates on the health of the system. It's important to avoid the trap of assuming everything is okay just because nothing bad seems to be happening.

A feature flag example

Consider the example below. The team added a couple buttons here for Cherry-pick and Revert in the pull request UI. These were deployed using feature flags.

Define feature flags

The first feature exposed was the Revert button. The solution uses an XML file to define all of the feature flags. There's one file per service in this case, which creates an incentive to remove old flags to prevent the section from getting really long. The team will delete old flags because there's a natural motivation to control the size of that file.

<?xml version="1.0" encoding="utf-8"?>
<!--
  In this group we should register Azure DevOps specific features and sets their states.
-->
<ServicingStepGroup name="AzureDevOpsFeatureAvailability" … >
  <Steps>
    <!-- Feature Availability -->
    <ServicingStep name="Register features" stepPerformer="FeatureAvailability" … >
      <StepData>
        <!--specifying owner to allow implicit removal of features -->
        <Features owner="AzureDevOps">
           <!-- Begin TFVC/Git -->
           <Feature name="SourceControl.Revert" description="Source control revert features" />

A common server framework encourages reuse and economies of scale across the whole team. Ideally, the project will have infrastructure in place so that a developer can simply define a flag in a central store and have the rest of the infrastructure handled for them.

Check feature flags at runtime

The feature flag used here is named SourceControl.Revert. Here's the actual TypeScript from that page that illustrates the call for a feature availability check.

private addRevertButton(): void {
 if (FeatureAvailability.isFeatureEnabled(Flags.SourceControlRevert)) {
     this._calloutButtons.unshift(
         <button onClick={ () => Dialogs.revertPullRequest(
             this.props.repositoryContext,
             this.props.pullRequest.pullRequestContract(),
             this.props.pullRequest.branchStatusContract().sourceBranchStatus,
             this.props.pullRequest.branchStatusContract().targetBranchStatus)
         }
         >
             {VCResources.PullRequest_Revert_Button}
         </button>
        );
     }
}

The example above illustrates usage in TypeScript, but it could just as easily be accessed using C#. The code checks to see if the feature is enabled and, if so, renders a button to provide the functionality. If the flag isn't enabled, then the button is skipped.

Control a feature flag

A good feature flag platform will provide multiple ways to manage whether a given flag is set. Typically, there are usage scenarios for the flag to be controlled via PowerShell and web interface. For PowerShell, all that really needs to be exposed are ways to get and set a feature flag's status, along with optional parameters for things like specific user account identifiers, if applicable.

Control feature flags through web UI

The following example uses the web UI exposed for this product by the team. Note the feature flag for SourceControl.Revert. There are two personal accounts listed here: hallux and buckh-westeur. The state is set for hallux, which happens to be in North Central, and cleared for the other account in West Europe.

The nature of the feature flag will drive the way in which the features are exposed. In some cases, the exposure will follow a ring and stage model. In others, users may opt in through configuration UI, or even by emailing the team for access.

Considerations for feature flags

Most feature flags can be retired once a feature has been rolled out to everyone. At that point, the team can delete all references to the flag in code and configuration. It's a good practice to include a feature flag review, such as at the beginning of each sprint.

At the same time, there may be a set of feature flags that persist for various reasons. For example, the team may want to keep a feature flag that branches something infrastructural for a period of time after the production service has fully switched over. However, keep in mind that this potential codepath could be reactivated in the future during an explicit clearing of the feature flag, so it needs to be tested and maintained until the option is removed.

Feature flags and branching strategy

Feature flags enable development teams to include incomplete features in main without affecting anybody else. As long as the codepath is isolated behind a feature flag, it's generally safe to build and publish that code without side effects impacting normal usage. But if there are cases where a feature requires dependencies, such as when exposing a REST endpoint, teams must consider how those dependencies can create security or maintenance work even without the feature being exposed.

Feature flags to mitigate risk

Sometimes new features have the potential to introduce destructive or disruptive changes. For example, the product may be undergoing a transformation from a wide database schema to a long one. In that scenario, the developer should create a feature branch for a small amount of time. They then make the destabilizing changes on the branch and keep the feature behind a flag. A popular practice is for teams to then merge changes up to main as soon as they're not causing any harm. This wouldn't be feasible without the ability to keep the unfinished feature hidden behind a feature flag.

Feature flags help work in main

If you follow the common-sense practices discussed in the Develop phase, working in main is a good way to tighten a DevOps cycle. When combined with feature flags, developers can quickly merge features upstream and push them through the test gauntlet. Quality code can quickly get published for testing in production. After a few sprints, developers will recognize the benefits of feature flags and use them proactively.

How to decide whether to use a feature flag

The feature teams own the decision as to whether they need a feature flag or not for a given change. Not every change requires one, so it's a judgment call for a developer when they choose make a given change. In the case of the Revert feature discussed earlier, it was important to use a feature flag to control exposure. Allowing teams to own key decisions about their feature area is part of enabling autonomy in an effective DevOps organization.

Build vs. buy

While it's possible to build your own feature flag infrastructure, adopting a platform like LaunchDarkly or Split is generally recommended. It's preferable to invest in building features instead of rebuilding feature flag functionality.

Next steps

Learn more about using feature flags in an ASP.NET Core app.