How to Run an A/B Test
Compare two versions of a message, offer, page concept, or customer experience with random assignment.
The Short Version
Compare two versions of a message, offer, page concept, or customer experience with random assignment. A practical way to begin is to write one specific decision the test should inform, then choose a single primary outcome before collecting data. Finish by making sure you report the result, sample size, uncertainty, and any exclusions alongside the business recommendation. The full walkthrough below explains each part in order.
Is This Guide for You?
Marketing, product, nonprofit, and business researchers who need a clear test of whether version B performs differently from version A.
Follow These Steps
Work through these 7 steps at your own pace. The wording is intentionally practical, and you can return to the checklist at the end when you are ready to review your work.
- Write one specific decision the test should inform.
- Choose a single primary outcome before collecting data.
- Create a survey experiment with version A as the first reference condition and version B as the second condition.
- Keep every element except the intended treatment difference constant.
- Randomly assign respondents, collect the planned sample, and avoid stopping because an early result looks favorable.
- In Results, choose the primary outcome and read the mean difference, 95% confidence interval, and treatment-versus-reference graph.
- Report the result, sample size, uncertainty, and any exclusions alongside the business recommendation.
What This Helps You Accomplish
A/B testing replaces informal preference debates with a randomized comparison. The first condition serves as the reference, and the estimated mean difference shows how the second version changed the selected outcome in the tested sample.
What a Good Result Looks Like
An A/B test should answer one decision with one primary metric, comparable versions, random assignment, and a stopping rule chosen before the result is known. Survey-based A/B tests measure stated responses; live product tests measure behavior.
Example: test a signup message before buying traffic
Randomly show half the sample a headline emphasizing low cost and half a headline emphasizing research rigor. Ask the same action-intent and credibility questions after exposure. Use the survey result to reject a clearly weak message, then validate the stronger candidate with actual landing-page conversions rather than calling stated intent a conversion forecast.
Decisions to Make Before You Begin
- Specify exactly what differs between A and B.
- Choose the primary metric and minimum meaningful lift.
- Set sample size and stopping rules in advance.
- Decide how device, source, and prior exposure will be monitored.
Common Mistakes to Avoid
- Watching results continuously and stopping when p drops below a threshold.
- Testing multiple major changes without knowing which caused the difference.
- Equating survey intent with observed purchasing behavior.
Your Practical Next Step
Start with the editable A/B test template, replace both stimuli, and keep the outcome wording identical across conditions.
A Quick Confidence Check
- Change one meaningful thing at a time.
- Select the primary outcome in advance.
- Use random assignment.
- Plan sample size before launch.
- Read the confidence interval, not only the p-value.
- Document null as well as positive results.
Common Questions
Can I use this guide if I am new to this?
Yes. Marketing, product, nonprofit, and business researchers who need a clear test of whether version B performs differently from version A. Follow the steps in order, start with a small test, and use the final checklist before you field or report anything important.
What is the simplest way to get started?
Begin by write one specific decision the test should inform. Next, choose a single primary outcome before collecting data. You do not need to perfect every setting before running a small preview or pilot.
How do I know when I am ready?
Use the confidence check above. In particular: Change one meaningful thing at a time. Select the primary outcome in advance. Use random assignment. Plan sample size before launch. When the decision is consequential, keep your study documentation and ask a qualified colleague or methods reviewer to examine the design as well.
Related Guides
Research artifact: sample data, codebook, and scriptsResearch Pathways
Product names and fielding platforms mentioned in this guide belong to their respective owners. Conjoint Survey is an independent academic research tool and is not affiliated with, sponsored by, or endorsed by those services.