How to Segment Experiment Results
Examine whether an experiment differs across customer or audience groups without turning every subgroup into a claim.
The Short Version
Examine whether an experiment differs across customer or audience groups without turning every subgroup into a claim. A practical way to begin is to choose segmentation variables measured before treatment, then specify the most important segments before seeing the results. Finish by making sure you treat unplanned subgroup patterns as exploratory and validate them in new data. The full walkthrough below explains each part in order.
Is This Guide for You?
Analysts exploring results by customer type, market, prior behavior, demographics, or other pre-treatment characteristics.
Follow These Steps
Work through these 6 steps at your own pace. The wording is intentionally practical, and you can return to the checklist at the end when you are ready to review your work.
- Choose segmentation variables measured before treatment.
- Specify the most important segments before seeing the results.
- Check sample sizes within every treatment-by-segment cell.
- Estimate an interaction between treatment and segment rather than comparing separate p-values.
- Report the overall treatment effect alongside segment estimates.
- Treat unplanned subgroup patterns as exploratory and validate them in new data.
What This Helps You Accomplish
A treatment can appear significant in one segment and not another even when the segment effects are not statistically different. Interaction analysis directly tests whether treatment effects vary across groups.
A Quick Confidence Check
- Use pre-treatment segments.
- Check cell sizes.
- Test interactions.
- Limit the number of segments.
- Label exploratory findings.
- Avoid targeting decisions based on unstable small groups.
Common Questions
Can I use this guide if I am new to this?
Yes. Analysts exploring results by customer type, market, prior behavior, demographics, or other pre-treatment characteristics. Follow the steps in order, start with a small test, and use the final checklist before you field or report anything important.
What is the simplest way to get started?
Begin by choose segmentation variables measured before treatment. Next, specify the most important segments before seeing the results. You do not need to perfect every setting before running a small preview or pilot.
How do I know when I am ready?
Use the confidence check above. In particular: Use pre-treatment segments. Check cell sizes. Test interactions. Limit the number of segments. When the decision is consequential, keep your study documentation and ask a qualified colleague or methods reviewer to examine the design as well.
Related Guides
Research artifact: sample data, codebook, and scriptsResearch Pathways
Product names and fielding platforms mentioned in this guide belong to their respective owners. Conjoint Survey is an independent academic research tool and is not affiliated with, sponsored by, or endorsed by those services.