Introduction to Feature Toggles

Continuous Integration (CI) and Continuous Delivery (CD) refer to the automated process of building, testing, and deploying software in order to deliver defect fixes and new features as quickly as possible to the end user.

One of the primary goals of adopting CI/CD development practices is to allow engineers to incrementally integrate and test new features, even when those features are not as a whole ready or complete. To reduce the probability of major integration conflicts, many teams around use feature toggles as a mechanism to control which features or sections of the code get to run in production, and which ones remain as latent code, perhaps as part of a yet incomplete feature or one where the behavior has been deliberately disabled in that environment.

Using feature toggles is one of the techniques used by many teams around the world to deliver features in such a controlled way, allowing for gradual rollouts, experimentation, and the remote management of the application’s operations.

What are feature toggles?

Feature Toggles (also sometimes referred to as Feature Flags, Feature Bits, or Feature Flippers) are toggle points used within your codebase to trigger on or off certain behaviors or components of your application, and can be divided into multiple categories depending on their purpose and usage. For example, if we need to disable editing a particular graph if the current user does not have editing rights, we can have a toggle point for this behavior like the following:

if (user.canEditGraph()) {
  showEditBar()
} else {
  hideEditBar()
}

The toggle above is classified as a Permission Toggle, one of the main toggle categories used in the industry, as will be explained below. The majority of toggles used within applications follow this pattern, where, depending on the outcome of a condition, one or another branch of the application’s code will be executed at any given time. Feature toggles are used extensively by some of the largest companies in the industry, including Facebook, Google, and Netflix1.

What are the different types of feature toggles?

In his excellent article on feature toggles2, Pete Hudgson outlines the four main categories of toggles used within the industry: release toggles, experiment toggles, ops toggles, and permission toggles.

Release toggles

These are toggles allowing incomplete or in-progress features to be merged into the main branch (e.g. master), decoupling deployment from release. Release toggles decouple the roll-out or shipping of a new feature from the actual release of the feature to users.

Example: You’re trying to add in a new survey form to an existing page in your front-end application, but the API backend for it is not ready yet. You can merge the UI work for the survey form into your main base branch and hide it behind a feature toggle, and then enable it once the backend API is ready.

Experiment toggles

These are toggles used to perform A/B testing, run experiments on multiple combination of features, and analyzing their effects on different user segments. In UI development, one of the most common examples of this kind of toggles involves the changing of UI color schemes and textual information seen by the user, and analyzing how these changes influence the user experience on the website.

Example: In combination with an analytics framework, you can use PlanOut from Facebook to perform A/B testing and run randomized experiments on your application by toggling on/off either new features, or variations on existing features.

Ops toggles

These are toggles used to enable or disable specific operations or functionality of the application, and can be used to degrade components under high load conditions gracefully.

Example: You’re building a dashboard page that displays real-time 3D-graphics animation data that is expensive to generate. If your application is enduring an unusually high load at any given moment and it’s not possible to scale up, you might to decide to use an Ops toggle to disable this feature until the load is back at a manageable level.

Permission Toggles

These are toggles used to enable or disable specific features according to the permissions of the users accessing those features.

Example: You’re building a forum website that has admin and non-admin users, and admin users have higher privileges that allow them to add site-wide announcements and moderate discussions. You can use permission toggles to enable or disable certain features for users possessing specific privileges.

Should I use feature toggles?

There are many ways to implement and use feature toggles, and which approach is best (and whether to use them at all) will depend on the type of application developed and its requirements.

At the one extreme, if you’re developing a special-purpose or domain-specific application where you don’t expect any further significant changes in requirements, don’t need to run any experiments, and don’t need to implement any permission-controlled features, then you probably don’t need to worry about the impact of feature toggles as much as a team developing an app for mass market in a rapidly changing environment, where changes in requirements are expected and experimentation necessary.

When in comes to release toggles, Martin Fowler explains3:

Your first choice should be to break the feature down so you can safely introduce parts of the feature into the product. The advantages of doing this are the same ones as any strategy based on small, frequent releases. You reduce the risk of things going wrong and you get valuable feedback on how users actually use the feature that will improve the enhancements you make later.

If the feature you’re developing can be divided into parts that can be safely added to the application without introducing toggles, then it’s better to do so. This approach allows you to integrate and make use of feature toggles without affecting agile best-practices, where early and frequent user feedback is emphasized.

Feature Toggles - Architecture & Best Practices

Feature toggles are a deceptively simple concept, and many apps around the world use them to toggle all sorts of features and variations. They become a problem only when the number of toggles in your application exceeds the ability of engineers to effectively reason about the behavior of the application.

In 2012, developers in Knight Capital Group, an American global financial services firm, had accidentally caused a trading glitch4 by repurposing a feature toggle, activating functionality which was unused for years. It took 45 minutes to fix the problem, at which point the company had lost nearly 440 million dollars, causing the group to go bankrupt. Using feature toggles at scale without adopting proven best practices or implementing robust testing strategies can be detrimental to an organization.

From an architectural point of view, an application making extensive use of feature toggles (or one expected to) should be designed with a view to the proper toggling infrastructure that will be used throughout the entire app. This will result in a mix of convention and special helper utilities or components that allow you to fetch, query, and use toggles in your application in a more declarative and less error-prone way.

As a simple example, in a React application you could create a Feature component that takes a feature key, and decides to render the children based on whether that feature is enabled or not, like so:

<Feature key="showSurvey">
  <SurveyComponent/>
</Feature>

Without some kind of architectural solution in place, the result would be a series of sporadic if/else toggle points spread across the application, lacking any proper indication of their meaning or value.

Common best practices when working with feature toggles include:

  • Using continuous integration tests that have broad coverage to assess the impact of any toggle changes.
  • Using automated canary testing tools (if possible) to roll out toggle changes in a staged fashion, allowing the team to catch any non-obvious side-effects on the application before it moves to all end-users
  • Monitoring system health, and including automatic rollback conditions in case the system is negatively impacted by the change.
  • Not overusing toggles: if it’s safe to introduce parts of the feature without toggles, then do so.
  • Regular toggle cleanup activities: if you have a release toggle in your codebase that’s no longer needed, since the feature has now been enabled permanently, then it should removed from the codebase. Dead toggle code represents one of the worst kinds of technical debt, because it increases the complexity of the code and confuses developers as to the real behavior of the application. Release and experiment toggles are usually short-lived and dynamic, and should be regularly updated or removed to reflect either the completion of features or the conclusion of an experiment. Ops toggles and permission toggles on the other hand are typically long-lived, and eventually you should have e constant number of them in your application. Several companies schedule regular activities after a release where they actively hunt for unused toggles in the application and remove them.

Feature-Flag-As-A-Service

In the past, organizations used to develop their own in-house solution for various kinds of toggles. In the simplest case, you could implement a stand-alone service that returns a config JSON file containing key/value pairs of features and whether they should toggled on/off, like below:

{
  “skipOptimizations”: false
  “enableMonitoring”: true
  “useTransforms”: true
}

You could use or build on tools like etcd to implement this solution. This would most certainly be enough for applications that don’t need the kind of power afforded by a full-fledged feature toggling solution that includes tools for analytics and A/B testing. As your application needs for managing and tracking toggles increase, however, you’re better off using a third-party feature flag management system.

Some of the benefits of using such a system include:

  • creating and managing feature flags through an easy-to-use UI dashboard
  • easy rollbacks
  • debugging
  • A/B testing tools
  • automated toggling of features based on conditions
  • user/audience segmentation
  • User insights, analytics
  • SLAs ensuring a minimum QoS
  • and much more

Facebook uses an internal tool called Gatekeeper as a feature flag management system5. But unless you’re Facebook, Google, or any other organization where you can afford to build your own custom feature flag solution, it will be more cost-effective to consider some of the most popular solutions available for this purpose. These include:

In the following section, we’re going to be looking at Optimizely and see how we can create our first feature toggling solution.

Optimizely

Optimizely has a free developer tier for creating feature toggles on their platform. After creating an account and signing in, you can see the following dashboard:

first-dashboard-shadowed

We’re going to be working through an example where we’ve created a new survey form for our application, but we want to use a feature toggle to toggle on/off showing this survey form for specific environments and specific users.

You can see on the left side menu that they have sections for creating features, audiences, and even experiments. Let’s start by creating our first feature toggle, displaysurveyform:

second-create-feature-dialog.png

After successfully creating it, we can see the main settings page for this toggle. Optimizely allows you to select the environments (development or production) where this flag will be enabled. Unlike traditional flags, where the decision on whether it is enabled or not is binary (either on or off), optimizely allows a more granular approach where you can specify the type and percentage of users who will see the toggle enabled, and likewise for those who will see it disabled. For example, if you want to disable the flag for all users in the production environment, you can simply slide it the meter all the way to 0%, as you see below:

production-0-percent.png

In our case, we want to enable this flag for the “development” environment for all users, so we slide all the way to 100%:

development-100-percent.png

Once we save and exit, we can see in our features list page that our “Display new survey form” feature is turned on fully for development, and disabled for production.

We’ll also need to create user segments or “audience” for our feature. You can divide users across a wide range attributes, and the simplest approach to divide users is to separate them based geography. So for the purposes of our example, let’s say that we want to show the new survey to users in North America, but not for users in Europe. So let’s create a new North American audience:

create-new-audience.png

As you can see, for any audience, we can specify the attributes that make up that audience. For our North American audience, we can specify that if they are American or Canadian, then form a part of that audience. Once you create an audience, you can target a feature flag so it’s only available to users who qualify.

Finally, let’s try to create an experiment based on our newly created feature toggle. This experiment will reveal what percentage of our North American audience will be able to go through and complete the entire survey if we toggle on the new survey form, and similarly what percentage of those users will be able to complete the survey if we don’t show the new survey form.

experiment.png

In order to measure this, we have to specify a metric representing the completion of going through the survey form. We’ll create a new event metric called “completesurveyform” that will get triggered whenever the survey form is completed, and through which the experiment will be able know the increase in total unique conversions for the two variations we created (in case the toggle is on or off).

metric-complete-survey.png

We’ll also need to make sure to specify that this experiment will be run for our North American audience only:

experiment-audience.png

That’s about it! Let’s test it out by querying the feature toggle from an app. I’ll be using a vanilla JavaScript app to get the value of the feature toggle and display it as HTML on the page. After setting up the SDK, we can see that the toggle is indeed enabled for our dev environment:

dev-sandbox-hidden.png

And disabled for our production environment:

prod-sandbox-hidden.png

Conclusion

Feature toggles are a powerful mechanism that allows teams to ship code faster, experiment, and manage the application’s behavior remotely by simply toggling on or off specific flags. While the kind of flexibility afforded by feature toggles is valuable, too much flexibility is often considered an anti-pattern, and can in general only be achieved at a substantial cost. Following the best practices outlined above (and especially, the regular activity of dead toggle removal) will ensure that the number of toggles in your applications at any given moment is at a manageable level, reaping the benefits of CD while allowing your team to adapt quickly to any future requirement changes.


  1. Rahman, Md et al. (2016). Feature Toggles: Practitioner Practices and a Case Study. 2016 IEEE/ACM 13th Working Conference on Mining Software Repositories.

  2. https://martinfowler.com/articles/feature-toggles.html

  3. https://martinfowler.com/bliki/FeatureToggle.html

  4. https://dealbook.nytimes.com/2012/08/02/knight-capital-says-trading-mishap-cost-it-440-million/

  5. Tang, C. et al. Holistic Configuration Management at Facebook. Facebook Inc. Retrieved from: https://research.fb.com/wp-content/uploads/2016/11/holistic-configuration-management-at-facebook.pdf