CSConjoint Survey

American Flag VADER Evaluation

VADER is useful only if it can recover a sentiment difference when the answer is known in advance. This evaluation follows that test from the experimental setup through the response-level data, statistical checks, and replication files.

Experimental Setup

We divided 1,000 generated respondents evenly between two arms. Everyone answered the same open-ended question: “What does being American mean to you?”

Control
No image

No-image arm

Positive-pattern responses were generated 15% of the time; the fixed-seed sample realized 14.6%.

TreatmentFlag of the United States used as the synthetic treatment stimulus

American-flag arm

Positive-pattern responses were generated 75% of the time; the fixed-seed sample realized 70.8%.

Result: Passed

VADER recovered the difference built into the response patterns. The direction was correct, the confidence interval excluded zero, and the permutation test found the observed gap highly unlikely under random reassignment.

Control mean0.181
Flag mean0.543
Mean difference+0.362
Permutation p< .001

The recovered treatment-minus-control difference was 0.362, with a 95% confidence interval from 0.325 to 0.400. The standardized mean difference was 1.198 across overlapping score distributions.

Prespecified Checks

Before looking at the result, we set the conditions the scoring workflow had to satisfy. Every check passed in the fixed-seed run.

  • Passexactly 500 synthetic responses were assigned to each arm
  • Passtreatment-minus-control sentiment exceeds the prespecified minimum
  • Passthe 95% confidence interval excludes zero
  • Passthe randomization-inference p-value is below 0.05
  • Passall VADER compound scores remain in the documented range

Download the Evaluation Artifact

The bundle includes the response-level data, study design, codebook, machine-readable results, and the script used to reproduce the analysis.

Complete ZIP bundle

american-flag-vader-evaluation-v1.zip

Readable report

evaluation-report.md

Machine-readable results

evaluation-report.json

1,000 synthetic responses

synthetic-responses.csv

Study design and estimand

study-design.json

Dataset codebook

codebook.md

Replication script

replication.mjs

Citation metadata

CITATION.cff

Artifact README

README.md

Reproduce the Result

npm run validate:flag-vader

Seed: conjoint-survey-american-flag-vader-v1

Citation

Conjoint Survey Research Team. (2026). American Flag VADER Known-Effect Evaluation (Version 1.0.0). System Analytics LLC.

Scope and Interpretation

This is a known-effect test of the scoring workflow. The response patterns were assigned by design, so the study shows whether VADER recovers the intended signal; it does not estimate how an American flag changes real people's attitudes.

VADER captures sentiment in the language used, not patriotism, agreement, or national identity. For interpretation details, see the VADER guide and the original Hutto and Gilbert paper. The treatment uses the standard U.S. flag documented by USAGov.