Known-Effects Validation

Why This Test Matters

A conjoint table can look perfectly respectable and still be wrong. The standard way to find out is to hand the software a problem you already know the answer to. Because we set the coefficients before generating a single response, there is a correct answer to check against rather than a plausible-looking one.

The three cases are the ones that come up constantly in applied work: a level that raises selection probability, a level that lowers it, and a level that does neither.

How the Benchmark Works

The benchmark uses three dichotomous randomized attributes with true effects of +0.10, -0.10, and 0.00 on binary choice. Five thousand generated respondents complete eight tasks with two profiles apiece, producing 80,000 profile rows.

Every profile carries one level from each attribute. The outcome is whether that profile won its forced-choice task, drawn probabilistically from the known effects. Any gap between the true and recovered values is therefore either sampling error or a bug. There is no third option, which is the whole point of doing it this way.

Each respondent contributes sixteen profile rows. Because observations from the same person are related, the analysis clusters standard errors by respondent just as it would for a fielded conjoint study.

How the Model Is Estimated

The app uses a profile-level linear probability model with indicators for every non-reference randomized attribute level entered jointly. The outcome records whether each profile was chosen, and standard errors are clustered by respondent.

Each attribute here has two levels, Absent and Present, with Absent as the reference. The Present coefficient is the estimated change in selection probability associated with that level, holding the other randomized attributes in the model.

Known and Recovered Effects

Rules for Passing

Every coefficient must land within 0.015 of its true value, and every 95% confidence interval must have a half-width of 0.020 or less. These deliberately strict thresholds are meant to expose implementation errors while the benchmark still has a known answer.

What the Estimates Recovered

The positive effect came back at 0.099 against a true value of 0.100. The negative effect came back at -0.102 against -0.100, and the null effect landed at -0.001 against zero. The largest absolute error was 0.002, and every interval met the precision threshold.

Run It Yourself

The validation uses a fixed seed and a script in the source repository. With a local checkout and the Node.js dependencies installed, run:

npm run validate:simulation

That regenerates the synthetic data, fits the model, checks the pass criteria, and writes the report artifacts.

Seed: conjoint-survey-known-effects-v1

Testing More Demanding Designs

One benchmark is useful, but it should not be the only one. The broader suite varies sample size, task count, attribute count, and respondent heterogeneity, then checks statistical recovery, randomization, and independent replication of the exported estimator.

Result: Passed · 5 simulation scenarios · 4 randomization checks · Python replication executed

The randomization checks test level exposure, profile position balance, randomized attribute order, survey-experiment arm assignment, and restricted designs interpreted conditional on the allowed profile space.

For cross-software replication, Node exports profile-level data and fits the model, while Python independently reads the CSV and refits the same clustered linear probability model. The maximum estimate difference was 0.000, and the maximum clustered-SE difference was 0.000.

The artifact also includes R and Stata scripts so researchers can rerun the same model in their preferred environment.

npm run validate:rigorous

Download the Evidence

Testing Failure Cases

Recovery tests ask whether the estimator gets the right answer. These checks ask whether the rest of the workflow fails safely when it encounters malformed designs, impossible restrictions, tampered tasks, problematic export names, oversized metadata, or spreadsheet formulas embedded in CSV fields.

Result: Passed · 37/37 adversarial cases passed

npm run validate:adversarial

Testing the Text-Analysis Workflow

The text-analysis benchmark uses a crossover pattern that should be easy to recognize if the scoring pipeline is working. Generated liberal and conservative personas are randomly assigned to aligned or misaligned gun-control messages, and their open-ended responses are scored with the same VADER compound measure exported by the app.

Aligned messages should produce more positive language and misaligned messages more negative language, with the direction reversing across the two personas. That is exactly the pattern the analysis recovered.

Result: Passed · 1,200 synthetic respondents · aligned-minus-misaligned VADER effect 1.794

Aligned mean: 0.890. Misaligned mean: -0.904. 95% CI for the alignment effect: 1.786 to 1.802.

npm run validate:text

Scope and Interpretation

This validation shows that the app's profile-level estimator recovers known positive, negative, and null effects when the truth is built into synthetic data. It is a software and estimator check, not a claim that every user-created study design is automatically valid.