Known-Effects Validation

Why bother

A conjoint table can look perfectly respectable and still be wrong. The standard way to find out is to hand the software a problem you already know the answer to. Because we set the coefficients before generating a single response, there is a correct answer to check against rather than a plausible-looking one.

The three cases are the ones that come up constantly in applied work: a level that raises selection probability, a level that lowers it, and a level that does neither.

Simulation design

Three dichotomous randomized attributes, with true effects on the binary choice outcome of +0.10, -0.10, and 0.00. Five thousand synthetic respondents, eight tasks each, two profiles per task - 80,000 profile rows.

Every profile carries one level from each attribute. The outcome is whether that profile won its forced-choice task, drawn probabilistically from the known effects. Any gap between the true and recovered values is therefore either sampling error or a bug. There is no third option, which is the whole point of doing it this way.

Rows cluster within respondents, sixteen apiece. Real conjoint studies have the same structure, and the standard errors need to reflect it.

Estimator

The same one the app exposes for publication use: a profile-level linear probability model with indicators for every non-reference randomized attribute level entered jointly. The outcome is a 0/1 for whether the profile was chosen. Standard errors are clustered by respondent.

Each attribute here has two levels, Absent and Present, with Absent as the reference. The Present coefficient is the estimated change in selection probability associated with that level, holding the other randomized attributes in the model.

Recovery

Pass criteria

Every coefficient has to land within 0.015 of its true value, and every 95% confidence interval has to have a half-width of 0.020 or less. Both are tighter than anyone would demand of a live field study. They're calibrated to catch implementation errors, not to settle substantive uncertainty in real samples.

What the numbers say

0.099 against a true 0.100. -0.102 against a true -0.100. -0.001 against a true zero. The worst absolute error in the set is 0.002, and every interval clears the half-width threshold with room to spare.

Read this as a check on the software and the estimator, nothing more. When assignment is clean and the truth is known, Conjoint Survey finds it.

Reproducing this

Fixed seed, and the script lives in the source repository. If you're reading this on the website there's nothing to run - the table above is the current output.

With a local checkout and Node.js dependencies installed:

npm run validate:simulation

That regenerates the synthetic data, fits the model, checks the pass criteria, and writes the report artifacts.

Seed: conjoint-survey-known-effects-v1

Broader Validation Suite

We also run a broader validation suite for three claims: statistical recovery across varied known-effect designs, integrity of the randomization engine, and independent replication of the exported profile-level estimator.

Result: Passed · 5 simulation scenarios · 4 randomization checks · Python replication executed

The randomization checks test level exposure, profile position balance, randomized attribute order, survey-experiment arm assignment, and restricted designs interpreted conditional on the allowed profile space.

For cross-software replication, Node exports profile-level data and fits the model, while Python independently reads the CSV and refits the same clustered linear probability model. The maximum estimate difference was 0.000, and the maximum clustered-SE difference was 0.000.

R and Stata replication scripts are also generated for external reruns. They are not marked as locally executed here because Rscript and Stata were not installed on the validation machine.

npm run validate:rigorous

Evidence Researchers Can Inspect

Adversarial Testing

We also run adversarial checks that try to break common research workflows: malformed designs, duplicate or reserved export names, impossible restrictions, bad survey-experiment arms, tampered respondent tasks, no-conjoint exports, oversized metadata, and spreadsheet-formula strings in CSV output.

Result: Passed · 37/37 adversarial cases passed

npm run validate:adversarial

Open-Ended Text Validation

We also validate the qualitative-text scoring workflow with a known-pattern survey experiment. Synthetic respondents are split evenly between very liberal and very conservative personas, then randomly assigned to liberal or conservative gun-control stimuli. The open-ended response is scored with the same VADER compound sentiment measure exported by the app.

The expected pattern is crossover alignment: liberals should write more positive paragraphs under the liberal stimulus and more negative paragraphs under the conservative stimulus; conservatives should show the reverse pattern.

Result: Passed · 1,200 synthetic respondents · aligned-minus-misaligned VADER effect 1.794

Aligned mean: 0.890. Misaligned mean: -0.904. 95% CI for the alignment effect: 1.786 to 1.802.

npm run validate:text