{"id":"ab-test-readout","name":"ab-test-readout","summary":"完成したA/Bテストを分析し、結果、統計的・実用的に有意かどうか、意味、そして船・非船の呼び出しを書き出します。","body":"# A/B Test Readout Skill\n\nThe hard part of an experiment is the readout: not \"B won\" but \"is this real, is it big enough to matter, and should we ship?\" This skill turns results into an honest decision — and flags the ways A/B results lie.\n\n## Working from a brief\n\nGiven results (even partial), **write the full readout anyway**. If significance isn't provided, reason about it from the numbers and flag what's needed to confirm. Mark assumed figures. Never declare a winner without addressing significance and sample.\n\n## Required Inputs\n\nAsk for (if not already provided):\n- **The hypothesis** and the **primary metric**\n- **Results** — control vs variant: conversions/rate, sample size per arm, duration\n- **Guardrail metrics** (revenue, retention, latency, complaints) that mustn't regress\n- **Pre-registered decision rule** (what would count as a win) if one exists\n\n## Output Format\n\n### 1. Verdict (one line)\n*Ship / Don't ship / Inconclusive — keep running* — with the headline number.\n\n### 2. The result\n\n| Metric | Control | Variant | Relative lift | Significant? |\n|---|---|---|---|---|\n| Primary | | | | p / CI |\n| Guardrail(s) | | | | |\n\nState **statistical** significance (p-value / confidence interval) *and* **practical** significance (is the lift big enough to matter given the cost?).\n\n### 3. Did it really win?\nAddress the ways A/B tests mislead:\n- **Sample / power** — was the test adequately powered, or under-sampled?\n- **Peeking** — was the call made early, inflating false positives?\n- **Novelty / primacy** — could the effect fade?\n- **Segments** — does the win hold across key segments, or is it driven by one?\n\n### 4. Segment cuts\nWhere the effect is strong vs flat vs negative (new vs returning, platform, geography).\n\n### 5. Recommendation & next step\nShip / iterate / re-run, plus what to monitor post-launch or what the follow-up test should isolate.\n\n## Quality Checks\n\n- [ ] Distinguishes statistical from practical significance\n- [ ] Checks guardrail metrics, not just the primary\n- [ ] Flags peeking, power, novelty, and segment-driven wins\n- [ ] Recommendation follows from the evidence, with a monitoring/next-test step\n- [ ] Doesn't declare a winner on an underpowered or peeked result\n\n## Anti-Patterns\n\n- \"B won by 8%!\" with no significance or sample size\n- Calling a result early (peeking) and shipping\n- Ignoring a guardrail regression because the primary went up\n- A statistically significant but practically meaningless lift treated as a win","author":"@mohitagw15856","ownerProfile":null,"authorContacts":null,"sourceUrl":"https://github.com/mohitagw15856/pm-claude-skills/tree/main/exports/openclaw/ab-test-readout","license":"MIT","category":"writing","lang":"en","tokens":597,"stars":0,"calls30d":2,"claimed":false,"visibility":"public","origin":"crawler","version":"0.1.0","createdAt":"2026-08-22","updatedAt":"2026-08-22","files":[],"requires":{"mcp":[],"tools":[]},"safety":{"flags":[],"scannedAt":"2026-08-22","hasScripts":false,"networkEndpoints":[]}}