# American Flag VADER Known-Effect Evaluation Report

**Result: Passed**

## Evaluation question

Can the Conjoint Survey VADER workflow detect a known positive treatment-control difference deliberately encoded in synthetic open-ended responses?

## Design

1,000 synthetic respondents were assigned with a fixed seed to either no image or an American flag image. Both arms received the prompt: **What does being American mean to you?** Treatment responses were intentionally more positive; control responses were intentionally neutral or mixed.

| Arm | N | Response-generation rule | Mean VADER | SD |
|---|---:|---|---:|---:|
| No image | 500 | 15% positive-pattern and 85% neutral-pattern synthetic language | 0.181 | 0.299 |
| American flag | 500 | 75% positive-pattern and 25% neutral-pattern synthetic language | 0.543 | 0.306 |

## Recovered effect

- Treatment minus control: **0.362**
- Standard error: 0.019
- 95% confidence interval: 0.325 to 0.400
- Standardized mean difference: 1.198
- Two-sided randomization-inference p-value (10,000 permutations): < 0.001

## Prespecified checks

- PASS: exactly 500 synthetic responses were assigned to each arm
- PASS: treatment-minus-control sentiment exceeds the prespecified minimum
- PASS: the 95% confidence interval excludes zero
- PASS: the randomization-inference p-value is below 0.05
- PASS: all VADER compound scores remain in the documented range

## Interpretation and limitation

The evaluation tests whether the VADER scoring and treatment-comparison pipeline detects a sentiment difference deliberately encoded in synthetic responses. It does not estimate the effect of an American flag on real people. The result validates signal recovery under this synthetic design; it does not validate VADER as a measure of patriotism, support, national identity, or any other substantive construct.

## Reproducibility

Seed: `conjoint-survey-american-flag-vader-v1`. VADER package: `vader-sentiment@1.1.3`. See the response-level CSV, study design, codebook, and replication script in this artifact.