Conjoint Survey Academic User Guide

Version 0.1. Last updated August 2026.

Suggested citation:

Conjoint Survey Research Team. (2026). *Conjoint Survey academic user guide: Design, fielding, and analysis for research-grade conjoint studies*. Conjoint Survey. https://conjointsurvey.com/guide

Validation report citation:

Conjoint Survey Research Team. (2026). *Synthetic known-effects validation report: Recovery of positive, negative, and null conjoint effects in Conjoint Survey* [Working paper]. Conjoint Survey. https://conjointsurvey.com/validation

1. Purpose

Conjoint Survey is a web application for creating, fielding, analyzing, and documenting randomized conjoint surveys. It is designed for academic researchers who need a simple respondent URL, transparent randomization, defensible profile-level data, and exports that can support preregistration, IRB review, replication files, and journal submission.

The guiding principle is: Rigorous Science, Easy to Use.

This guide explains the complete workflow, the meaning of each feature, and the basic math behind the app's diagnostic ATE summaries and full-model AMCE/LPM estimates.

2. What A Conjoint Design Does

A conjoint survey presents respondents with profiles made from randomly varied attributes. A researcher defines attributes such as party, policy position, occupation, cost, implementation timeline, or product quality. Each attribute has levels, such as Democrat, Republican, and Independent.

The respondent sees one or more profiles in a task and chooses a preferred profile. Because the attribute levels are randomized, the researcher can estimate how changing an attribute level changes the probability that a profile is selected.

3. Core Terms

Attribute: The variable randomized in the study.

Level: A possible value of an attribute.

Reference level: The level used as the comparison category in estimates.

Profile: One complete bundle of attribute levels shown to a respondent.

Task: One respondent-facing choice screen containing multiple profiles.

Neither option: An optional response allowing the respondent to reject all shown profiles.

ATE: A diagnostic difference in observed choice probability between a level and a reference level.

AMCE: The average marginal component effect. In the app's publication path, this is estimated through a full profile-level linear probability model that includes all non-reference randomized attribute levels jointly.

4. Creating A Study

Start from the dashboard and choose either a template or a blank study. The main setup fields are study title, respondent introduction, choice prompt, tasks per respondent, profiles per task, target completed responses, minimum completion time, auto-close setting, completion message, redirect URL, and optional attention check.

5. Choosing Attributes And Levels

Good attributes are theoretically meaningful, understandable to respondents, balanced enough to estimate, and not so numerous that respondents become overloaded. Good levels are plausible, mutually exclusive within an attribute, and written in similar style and length when possible.

The app treats levels as categorical by default. Even if the researcher labels an attribute ordinal or numeric, the publication analysis path does not assume a linear scale unless the researcher modifies the exported code.

6. Reference Levels

Each attribute needs a reference level. Estimates for other levels are interpreted relative to that reference. Reference levels should be chosen before fielding when possible and recorded in preregistration materials.

7. Restrictions

Restrictions prevent impossible or undesirable profiles. They are useful, but they change the estimand. With restrictions, estimates are conditional on the allowed profile space after impossible profiles are removed.

Restrictions can induce association between attributes. For this reason, the app flags exposure imbalance and induced attribute association, and the publication path uses the full AMCE/LPM model.

8. Randomization

For fielded surveys, Conjoint Survey uses server-owned randomization. At response start, the server generates the respondent's tasks, stores the randomized profiles, and returns those tasks to the browser.

At completion, the server verifies that the submitted task profiles match the server-randomized task record.

The study design stores randomization seed metadata. Each respondent's tasks are generated from the study seed plus respondent/session information.

9. Randomization Audit

The Randomization Audit simulates respondents under the current design and reports exposure counts by attribute level. It reports attribute, level, observed simulated count, expected count, difference, and ratio of observed to expected exposure.

10. Sample Planning And MDE

The Sample Planning panel gives approximate exposure and minimum detectable effect guidance.

Profile rows = respondents x tasks per respondent x profiles per task.

Approximate rows per widest attribute level = profile rows divided by the largest number of levels in any attribute.

Approximate 80 percent power MDE:

MDE = (1.96 + 0.84) x sqrt(2 x p x (1 - p) / n_level)

The app uses p = 0.5 as a conservative baseline. The result is approximate and should be treated as planning guidance, not as a full design-specific power analysis.

11. Fielding The Survey

A saved and published study produces a public respondent URL. Researchers can distribute this URL through email lists, Prolific, MTurk, course systems, Qualtrics panels, or other recruitment systems.

URL parameters are captured and stored. Completion messages and redirect URLs can include incoming URL parameters.

12. Duplicate Detection

The app stores respondent/session identifiers and common URL parameters. It computes a respondent key using available panel identifiers and flags duplicate completions.

13. Attention And Comprehension Checks

Researchers can add an optional attention or comprehension check. The app records the response and whether it matches the expected answer.

14. Timing And Speeders

The app stores start and completion times. Researchers can define a minimum completion threshold. This can be enforced by the server for fielded responses and used later as an exclusion rule.

15. The Respondent Experience

Respondents see the study title, respondent introduction, choice prompt, profile table, radio-button choices, optional neither option, optional attention check, and completion message or redirect.

16. Data Structure

The app stores raw response data, task-level data, profile-level data, estimate tables, codebooks, reliability metadata, and workflow logs.

17. Outcome Coding

The primary profile-level outcome is chosen. chosen = 1 if the profile was selected. chosen = 0 if the profile was not selected.

If a neither option is used and the respondent selects neither, profile rows from that task are marked with neither = 1 and excluded from the standard ATE/AMCE calculation.

18. Diagnostic ATE

The dashboard ATE is a diagnostic difference in observed choice probability.

For attribute A and level l compared with reference r:

ATE_l = mean(chosen | A = l) - mean(chosen | A = r)

The standard error is:

SE = sqrt[p_l(1 - p_l)/n_l + p_r(1 - p_r)/n_r]

The 95 percent confidence interval is:

ATE_l +/- 1.96 x SE

The p-value is based on a two-sided normal approximation using z = ATE_l / SE.

19. Full-Model AMCE / LPM

The publication-oriented model is a profile-level linear probability model:

chosen_i = alpha + beta_1 X_1i + beta_2 X_2i + ... + beta_k X_ki + epsilon_i

Each X term is an indicator for a non-reference attribute level. All randomized attributes are included jointly. Standard errors are clustered by respondent because each respondent contributes multiple profile rows.

20. ATE Versus AMCE

The diagnostic ATE asks how often profiles with a level were chosen compared with profiles with the reference level. The full-model AMCE/LPM estimates the marginal effect of the level while modeling all non-reference randomized attribute levels jointly.

21. Exclusion Rule Manager

The Results page includes toggles to remove duplicate responses, attention/comprehension failures, and speeders. These rules update displayed estimates and are written into methods and reviewer exports.

22. Results Dashboard

The Results page includes field data refresh, simulation, exports, diagnostic ATE, full-model AMCE/LPM, ATE plot, R script, Stata script, publication table, LaTeX table, and academic packet.

23. Journal-Ready Tables

The publication table reports term, estimate, clustered standard error, 95 percent confidence interval, p-value, N, and respondent clusters.

24. Academic Packet

The Academic Packet export contains readme, preregistration summary, IRB template, respondent instructions, methods text, reviewer appendix, publication tables, design summary, codebook, R script, Stata script, and metadata describing exclusions, reliability, design warnings, and result warnings.

25. Preregistration

The preregistration export records study design, attributes and levels, randomization restrictions, primary estimand, analysis plan, exclusions, guardrails, and randomization audit.

26. IRB Template

The IRB template describes what participants will do, how tasks are randomized, what data are stored, planned sample size, and reminders for recruitment, consent, risk, compensation, retention, and contact information.

27. Reviewer Appendix

The reviewer appendix is designed to answer predictable reviewer questions about randomization, levels, restrictions, sample, exclusions, estimation, diagnostics, and estimate tables.

28. Replication Scripts

The R script reads the profile-level CSV, drops neither-choice rows, sets reference levels, estimates the full LPM, and reports respondent-clustered standard errors using sandwich and lmtest.

The Stata script reads the profile-level CSV, drops neither-choice rows, creates level indicators, and estimates the full model using vce(cluster response_id).

29. Best Practices

Before publishing, use theoretically meaningful attributes, keep profiles readable, avoid excessive attributes and levels, set reference levels intentionally, use restrictions only when needed, preview the respondent URL, set a target sample size, decide exclusion rules, export preregistration text, and run the randomization audit.

After fielding, refresh field data, review reliability metrics, apply preregistered exclusion rules, export raw/task/profile data, use R or Stata scripts for final analysis, and archive the academic packet.

30. Synthetic Known-Effects Validation

Conjoint Survey includes a synthetic validation study that generates fake respondent choices from a known data-generating process and then runs the same full-model linear probability approach used by the in-app AMCE table.

The validation design uses three dichotomous randomized attributes. The true effects on the binary profile-choice outcome are +0.10, -0.10, and 0.00. With 5,000 synthetic respondents, 8 tasks per respondent, and 2 profiles per task, the validation run produced 80,000 profile-level observations.

In the August 2026 validation run, the app recovered the known effects as follows:

| Attribute Level | True Effect | Estimated Effect | 95% CI | |---|---:|---:|---:| | Positive Signal: Present | 0.100 | 0.099 | 0.093 to 0.106 | | Negative Signal: Present | -0.100 | -0.102 | -0.109 to -0.096 | | Null Signal: Present | 0.000 | -0.001 | -0.008 to 0.006 |

The standalone citable report is available at https://conjointsurvey.com/validation, with a PDF version at https://conjointsurvey.com/synthetic-known-effects-validation-report.pdf.

This validation is not evidence that every real-world design is well specified. It is a software and estimator check showing that, under clean random assignment and known true effects, the app recovers positive, negative, and null effects within a narrow margin of sampling error. Researchers do not need to run the command to use the app or cite the report. Developers with a local source checkout and Node.js dependencies installed can rerun the validation with `npm run validate:simulation`.

31. Limits And Cautions

The app does not replace substantive theory, ethical review, sampling expertise, or final statistical judgment. The in-app ATE is diagnostic. The full-model AMCE/LPM and exported replication scripts are the intended publication path. Power calculations are approximate. Restrictions change the estimand.

32. How To Cite