diagnosing-experiment-results
Diagnoses bias, anomalies, and strange results on a PostHog experiment. Covers 0-exposure experiments, sample ratio mismatch, identity fragmentation, multi-variant exposure, uneven-split exclusion bias, significance traps (peeking, A/A, Bayesian vs Frequentist), PostHog-vs-SQL discrepancies, surprises after mid-run edits, and qualitative follow-up via a variant-split survey. TRIGGER when: user asks 'is my experiment biased?' or 'why 0 exposures?', references the bias banner, says a variant looks strange / wrong / off, sees significance flipping or A/A significance, finds PostHog numbers disagreeing with their SQL, reports surprises after mid-run edits, or wants qualitative feedback or a survey for an experiment. DO NOT TRIGGER when: creating an experiment (use creating-experiments), only configuring rollout (use configuring-experiment-rollout) or metrics (use configuring-experiment-analytics), or only asking lifecycle questions (use managing-experiment-lifecycle).
Pinned to revision 4e8964e24a5a, so it is the text this page describes rather than whatever the author pushed since.
Files
- skills/diagnosing-experiment-results/SKILL.md
- skills/diagnosing-experiment-results/references/bias-and-skew.md
- skills/diagnosing-experiment-results/references/diagnostic-snapshot.md
- skills/diagnosing-experiment-results/references/empty-experiment.md
- skills/diagnosing-experiment-results/references/interpretation.md
- skills/diagnosing-experiment-results/references/mid-run-changes.md
- skills/diagnosing-experiment-results/references/numbers-vs-sql.md
- skills/diagnosing-experiment-results/references/qualitative-feedback.md
Every link opens the file at its source, pinned to the revision this page describes.