Optimizely: A/B testing and experimentation for large organisations

Optimizely is an experimentation platform for testing, personalizing, and improving website and product experiences — from simple A/B tests on a landing page to server-side experiments deep inside an application. It's the best-known name in the experimentation space, and has grown into a broader digital experience platform with CMS, content, and personalization features on top. This article focuses on Optimizely as an experimentation tool: what it does, how you set it up, who it works for (and who it doesn't), and how it compares to VWO and Varify.

We mainly work with Varify ourselves, and recommend Optimizely when it fits your situation: serious traffic, an in-house development team, and a need for server-side testing. A broader comparison of the three tools is in A/B testing tools compared.

What Optimizely is

Optimizely is software for testing variants of a page, feature, or flow against an original (or against each other), and using data to decide which variant performs better. The company started over a decade ago as a pure A/B testing tool and, through acquisitions (including Episerver), grew into a platform with several products:

• Web Experimentation — client-side A/B and multivariate testing on your website through a visual editor or code changes.
• Feature Experimentation — server-side testing and feature flags, built for product teams and developers experimenting inside the application logic itself, not just in the display layer.
• Personalization — showing content and offers based on behavior, segment, or past interactions.
• Content Management System (formerly Episerver/Optimizely CMS) — a standalone CMS product, often used by organizations that want to combine content and experimentation in one environment.
• Data Platform and Opal (AI features) — newer additions for data unification and AI-assisted content production; these features change quickly, so always check the current state at optimizely.com.

This article focuses on the core most businesses consider Optimizely for: experimentation, both client-side (web) and server-side (feature flags, product experiments).

Who Optimizely works for — and who it doesn't

Optimizely is clearly an enterprise-oriented platform. That's the single biggest factor in deciding whether it fits.

Optimizely is a good fit if:

• you have an in-house development or product team that can build and maintain experiments technically;
• you need server-side experiments and feature flags, for example to roll out a new feature gradually while measuring its impact;
• you're testing across multiple teams, brands, or markets and need roles, permissions, and governance for that;
• you have enough traffic and conversions to reach statistical significance within a reasonable time — check this with the statistical significance calculator;
• budget isn't the limiting factor: Optimizely works with custom pricing that typically starts in the enterprise range.

Optimizely is not a good fit if:

• you're a small or mid-sized business or online store with a limited testing budget — Varify offers the same core client-side testing features at a fraction of the price;
• you don't have a developer or technical team to build and maintain server-side experiments;
• your site gets a few thousand visitors a month — every test will take too long, and you'll pay for features that sit unused;
• you're mainly looking for qualitative research (heatmaps, session recordings, surveys) alongside testing — look at Hotjar or VWO instead, which include those features by default.

The building blocks of an Optimizely experiment

Regardless of the exact product, an experiment in Optimizely is built from the same core pieces:

• Flag or experiment — the container that defines what you're testing, and for whom.
• Variations — the different versions you pit against each other, including a control (the original).
• Audiences — the visitor segments who see your experiment, for example by country, device, source, or logged-in status.
• Traffic allocation — the percentage of visitors included in the experiment and how it's split across variations.
• Metrics (goals) — the events that determine which variation wins: a purchase, a form, a click, a step further in the funnel.
• Events — the individual measurement points you send to Optimizely, via the snippet (web) or SDK (feature experimentation).
• Stats Engine — the statistical layer that decides when a result is reliable, including corrections for multiple metrics and early stopping.

Step by step: setting up an experiment

1. Decide what you want to learn, not just what you want to change. "Does a shorter checkout increase completed orders?" is testable; "let's make the button green" usually isn't.
2. Pick the right product. A visual change on a marketing page belongs in Web Experimentation; new pricing logic or a feature in your application belongs in Feature Experimentation.
3. Place the snippet (web) or integrate the SDK (feature experimentation) in your codebase. For server-side experiments this works through SDKs for the main languages and frameworks, so the flag logic runs inside your own application.
4. Build the variations. For web experiments you can use the visual editor; for more complex changes or feature flags, a developer writes the logic.
5. Set your audience and traffic allocation. Decide who sees the experiment and what share of your traffic is included.
6. Connect your metrics. Make sure the events you want to measure are already coming in correctly, ideally also in your own analytics tool so you can cross-check results.
7. Calculate in advance how much traffic and time you need for a reliable result, and lock that in before you start. That prevents a test being stopped halfway because the interim numbers happen to look favorable.
8. Launch the experiment and let it run for the agreed period. Don't check daily for "early winners" — stopping early skews the outcome.
9. Analyze and document, including experiments that didn't move the needle. That's the knowledge that sharpens your next tests.
10. Roll out the winning variation to 100% of traffic, or, for feature flags, gradually based on confidence in the outcome.

Experiment types you'll run into in practice

Targeting and segmentation

Audiences in Optimizely are built from attributes you provide yourself or that are available by default: country, device type, browser, traffic source, new versus returning visitor, logged-in status, or a custom property from your CRM or product data. The more finely you segment, the smaller each experiment's audience, and the longer a test takes to reach significance — so only segment when you genuinely expect the outcome to differ by segment.

Budget, team, and implementation in general terms

Optimizely doesn't publish price lists; costs are quote-based and depend on traffic, the number of products (web, feature experimentation, personalization, CMS), and the number of teams or brands using the platform. In general terms:

• License costs scale with the number of unique visitors or "decisions" (the number of times a flag makes a decision) your platform processes.
• Implementation requires developer time, especially for Feature Experimentation: integrating SDKs, setting up flags in your codebase, and forwarding events correctly.
• Ongoing management needs an owner: someone who prioritizes experiments, reviews results, and keeps the testing calendar on track. Without that role, an expensive license becomes an underused subscription.

Always check optimizely.com or talk to their sales team for current pricing and packages, since product bundling changes regularly.

Measurement: what you need before you test

An experiment is never better than the measurement underneath it. Before launching a test:

• Your goals are set up and accurate. A purchase, lead, or funnel step needs to be tracked reliably, independent of Optimizely itself.
• You have a reference point in your own analytics tool. Where possible, connect results to GA4 or another analytics platform via Google Tag Manager, so you can cross-check outcomes outside the testing platform.
• You know your baseline conversion rate and traffic. Without those numbers you can't estimate how long a test needs to run.
• You avoid contamination from internal visitors, bots, and repeat visits that skew the outcome.

Avoid vanity metrics: an experiment that produces "more clicks" but not more revenue or leads has little value. Optimize for the goal at the bottom of the funnel, not the easiest intermediate number.

Real-world examples

• Online store. A B2C store with hundreds of thousands of monthly visitors tests continuously on product pages and in checkout: price display, shipping cost messaging, number of steps. With that volume, A/B tests reach significance within days to weeks, and rolling out a winning variation delivers measurable revenue immediately.
• B2B software company. A SaaS company uses Feature Experimentation to roll out new features gradually: first to 5% of users, then 25%, then 100%, while measuring usage and impact on churn. The feature flag works as both a release mechanism and an experiment.
• Local or smaller business. With a few thousand monthly visitors, Optimizely is typically overkill in practice: tests take too long to become significant, and license costs don't match the traffic volume. Here, Varify or first investing in more traffic and a sharper offer is the more logical step.

Common mistakes

• Testing without enough traffic. An experiment that never reaches significance is wasted time and money. Calculate this in advance.
• Watching too many metrics at once. The more goals you track, the higher the chance one of them looks "significant" by chance. Pick one primary metric per experiment.
• Stopping early on a favorable interim result. The first days of a test are volatile; an early "winner" is often just noise.
• Leaving feature flags in place after rollout. Old flags nobody cleans up make your codebase unnecessarily complex and error-prone.

Optimizely versus VWO versus Varify

The three tools you'll most often run into for experimentation mainly differ in scale, price, and whether you need client-side, server-side, or both.

The difference between VWO and Optimizely is mainly one of emphasis: VWO bundles testing with qualitative research (heatmaps, session recordings, surveys) in one subscription by default, while Optimizely goes deeper into server-side experimentation and feature flag management for product teams. For most SMBs and online stores, Varify covers the same core need — client-side A/B testing and personalization — at a significantly lower entry point, though it lacks the depth in server-side experiments that Optimizely offers.

A full comparison including pros and cons per tool is in A/B testing tools compared.

How Optimizely works with other tools

• Analytics. Connect experiment results to GA4 or another analytics platform, for example via Google Tag Manager, so you can verify outcomes outside Optimizely and report on them in Looker Studio.
• CMS and personalization. If you already use the Optimizely CMS, experimentation and personalization connect seamlessly to your content management. If you use a different CMS like WordPress or WooCommerce, you simply use the experimentation products standalone.
• Development workflow. Feature Experimentation integrates via SDKs into your existing codebase and CI/CD process, turning flags into part of regular releases instead of a separate experimentation process.
• The bigger CRO picture. A testing platform is one part of a broader conversion optimization approach; qualitative research (interviews, session recordings) and clear positioning of your offer often matter more than the tool itself.

Checklist: are you ready for Optimizely?

• You have enough traffic and conversions to reach significance within weeks, not months.
• You have a developer or technical team available for implementation and maintenance.
• You want to experiment server-side or use feature flags, not just make visual changes.
• Multiple teams or brands need their own roles and permissions.
• Your measurement (goals, events) is already reliable, independent of the testing platform.
• Someone owns the testing calendar: coming up with, prioritizing, and reviewing experiments.
• Your budget fits an enterprise license; if not, consider Varify or VWO first.

Checked "no" on more than two? Chances are you'll learn the same things faster and cheaper with Varify or VWO, and can always scale up to Optimizely later.

Frequently asked questions about Optimizely

What does Optimizely cost?

Optimizely uses custom pricing, depending on traffic, the number of products you use (web, feature experimentation, personalization, CMS), and the number of teams. In practice, costs typically start in the enterprise range. Check the current situation at optimizely.com or talk to their sales team.

What's the difference between Optimizely and VWO?

Optimizely puts more emphasis on server-side experimentation and feature flags for product teams, while VWO bundles qualitative research (heatmaps, session recordings, surveys) with testing in one subscription by default. Both are mature platforms aimed at organizations with serious traffic and budget.

Is Optimizely suitable for a small business or online store?

Usually not. With a limited testing budget and little technical capacity, Varify gets you further, offering the same core client-side testing features at a fraction of the price. Optimizely only becomes the logical choice once you have serious volume and an in-house development team.

What's the difference between Web Experimentation and Feature Experimentation?

Web Experimentation tests client-side — changes visible in a page's HTML, usually through a visual editor. Feature Experimentation works server-side, inside the application logic itself, and is used for feature flags and experiments that aren't visible in the page code.

Can I use Optimizely for feature flags without running experiments?

Yes, feature flags can also be used independently of an experiment, for example to roll out a feature gradually or switch it off quickly if problems occur. Many teams combine this with experimentation to measure what a new feature does at the same time.

How much traffic do I need to test with Optimizely?

That depends on your current conversion rate and the difference you expect to measure. Check this in advance with the statistical significance calculator — without enough traffic, every test will take too long to be useful.

Does Optimizely replace Google Optimize?

Google Optimize was discontinued at the end of September 2023. Optimizely is one of the platforms former Google Optimize users moved to, though in terms of price and complexity it mainly fits organizations that were already larger than the average Google Optimize user. For smaller businesses, Varify is often a more logical replacement.

Does Optimizely work with my existing CMS?

Yes, the experimentation products work independently of your CMS; you place a snippet or integrate the SDK regardless of whether your site runs on WordPress, WooCommerce, Shopify, or a custom build. If you use the Optimizely CMS itself, the integration with experimentation and personalization is tighter.

Further reading

Want to know which testing tool fits your situation best? Read A/B testing tools compared and calculate your traffic needs with the statistical significance calculator. Curious about the more accessible alternatives? Check out Varify and VWO. Want to figure out together which approach fits your business? Book a free strategy call.