How to Run Experimental Quantitative Text Analysis
Use open-ended survey responses as quantitative outcomes after randomized stimuli.
The Short Version
Use open-ended survey responses as quantitative outcomes after randomized stimuli. A practical way to begin is to create a survey experiment from the consent and questions screen, then add a stimulus condition for each randomized arm. Finish by making sure you export Text Summary, Text Scores, and Text Effects CSVs. The full walkthrough below explains each part in order.
Is This Guide for You?
Researchers who want paragraph responses, sentiment outcomes, word counts, and treatment-arm text comparisons.
Follow These Steps
Work through these 6 steps at your own pace. The wording is intentionally practical, and you can return to the checklist at the end when you are ready to review your work.
- Create a survey experiment from the consent and questions screen.
- Add a stimulus condition for each randomized arm.
- Add an open-ended text question after the stimulus.
- Collect responses.
- Open Results.
- Export Text Summary, Text Scores, and Text Effects CSVs.
What This Helps You Accomplish
Open-ended outcomes let respondents explain reactions in their own words while still giving researchers analyzable sentiment, effort, and language-use measures.
A Quick Confidence Check
- Preregister the text outcome.
- Use VADER sentiment as a transparent diagnostic, not a universal measure of agreement.
- Keep raw text with the replication bundle.
- Validate high-stakes measures with human coding.
Common Questions
Can I use this guide if I am new to this?
Yes. Researchers who want paragraph responses, sentiment outcomes, word counts, and treatment-arm text comparisons. Follow the steps in order, start with a small test, and use the final checklist before you field or report anything important.
What is the simplest way to get started?
Begin by create a survey experiment from the consent and questions screen. Next, add a stimulus condition for each randomized arm. You do not need to perfect every setting before running a small preview or pilot.
How do I know when I am ready?
Use the confidence check above. In particular: Preregister the text outcome. Use VADER sentiment as a transparent diagnostic, not a universal measure of agreement. Keep raw text with the replication bundle. Validate high-stakes measures with human coding. When the decision is consequential, keep your study documentation and ask a qualified colleague or methods reviewer to examine the design as well.
Related Guides
Research artifact: sample data, codebook, and scriptsResearch Pathways
Product names and fielding platforms mentioned in this guide belong to their respective owners. Conjoint Survey is an independent academic research tool and is not affiliated with, sponsored by, or endorsed by those services.