Experimental Quantitative Text Analysis
Experimental quantitative text analysis treats an open-ended response as an outcome in a randomized study. Instead of asking only a closed-ended attitude item after a treatment, a researcher can ask respondents to write a paragraph, then analyze measurable features of that text: sentiment, word count, distinctive terms, and differences across randomized arms.
The workflow is simple: randomize a stimulus, collect written reactions, score the text, and estimate whether the treatment changed how respondents wrote. It keeps the richness of open-ended responses while producing transparent quantitative outcomes that can be exported, checked, and documented.
What Conjoint Survey Does
When a survey includes an open-ended text box, Conjoint Survey automatically creates respondent-level text outcomes in the analysis export. For every text question, the app adds a word-count variable and a VADER compound sentiment score. If the survey also includes a randomized experiment, the Results page compares text outcomes by treatment arm and exports treatment-vs-reference text-effect tables.
| Output | Meaning | Use |
|---|---|---|
| Word count | Number of substantive tokens in the response | Effort, elaboration, engagement, or response completeness |
| VADER compound sentiment | Rule-based sentiment score from about -1 to +1 | Affective valence of a written reaction |
| Distinctive terms | Words appearing at different rates by randomized arm | Quick checks of what language moved across conditions |
| Text treatment effects | Arm differences in sentiment and word count | Experimental tests using open-ended DVs |
Example
A researcher randomizes respondents to see either a Black police officer or a white police officer, then asks: “Please write a few sentences about how this image makes you feel about the police.” The outcome is no longer only a closed-ended approval item. The paragraph itself becomes analyzable.
The researcher can export the raw text, VADER sentiment score, word count, treatment-arm assignment, and respondent covariates. A positive sentiment treatment effect means the treatment generated more positive language. A word-count effect means the treatment changed how much people wrote. Distinctive terms show which words became more common in one condition than another.
Validation
We validate this workflow with a known-pattern synthetic experiment. Synthetic respondents are split evenly between very liberal and very conservative personas, then randomly assigned to liberal or conservative gun-control stimuli. They write open-ended reactions, and the app scores those reactions with the same VADER compound sentiment measure used in exports.
The expected pattern is crossover alignment: liberals should write more positive paragraphs under the liberal stimulus and more negative paragraphs under the conservative stimulus; conservatives should show the reverse. That is exactly what the validation finds.
Result: Passed · 1,200 synthetic respondents · aligned-minus-misaligned VADER effect 1.794
| Persona | Stimulus | Aligned | N | Mean VADER | Mean words |
|---|---|---|---|---|---|
| Very liberal | Liberal gun-control view | Yes | 297 | 0.889 | 26.630 |
| Very liberal | Conservative gun-control view | No | 303 | -0.952 | 25.429 |
| Very conservative | Liberal gun-control view | No | 311 | -0.857 | 25.659 |
| Very conservative | Conservative gun-control view | Yes | 289 | 0.892 | 25.775 |
Aligned responses averaged 0.890. Misaligned responses averaged -0.904. The 95% confidence interval for the alignment effect was 1.786 to 1.802.
npm run validate:text
Did We Invent This?
No. The underlying idea is established in social science and computational text analysis. Scholars have analyzed open-ended survey responses for decades, and Roberts and colleagues showed that structural topic models can incorporate treatment assignment and estimate treatment effects from open-ended survey responses. More recent work also argues for treating text as behavior, where written expression is itself an effortful action worth measuring.
What is distinctive here is the product workflow: a researcher can build a randomized survey experiment, collect open-ended text, export VADER and word-count outcomes, inspect treatment-arm text effects, and cite a validation artifact from the same app. That is the claim we are comfortable making.
Use Carefully
VADER sentiment is useful, transparent, and citeable, but sentiment is not the same thing as stance, agreement, persuasion, or policy support. For political text, researchers should treat sentiment scores as one measured feature of language, not as a universal substitute for human coding or construct validation.
Best practice is to preregister the text outcome, keep the raw text, report the scoring method, validate the measure against a sample of human-coded responses when stakes are high, and use the exported text scores as a starting point for replication rather than a black box.
References
- Hutto, C., & Gilbert, E. (2014). VADER: A parsimonious rule-based model for sentiment analysis of social media text. Proceedings of the International AAAI Conference on Web and Social Media, 8(1), 216-225. https://doi.org/10.1609/icwsm.v8i1.14550
- Roberts, M. E., Stewart, B. M., Tingley, D., Lucas, C., Leder-Luis, J., Gadarian, S. K., Albertson, B., & Rand, D. G. (2014). Structural topic models for open-ended survey responses. American Journal of Political Science, 58(4), 1064-1082.
- Haselmayer, M., & Jenny, M. (2017). Sentiment analysis of political communication: Combining a dictionary approach with crowdcoding. Quality & Quantity, 51, 2623-2646.
- Wasow, O. (2026). Text as behavior. Political Analysis.