CHAPTER 23 - PM HERO

A/B Testing untuk PM

Bukan hanya marketer. PM modern wajib bisa run experiment untuk validate hypothesis sebelum full rollout.

TIME: 26MLEVEL: intermediate30 CHAPTERS

1. Mengapa A/B Test

Avoid build wrong thing. Validate sebelum invest big. Reduce subjective opinion (gantinya: data). Build culture experimentation. Skill #1 modern PM untuk decision di high uncertainty.

2. Hypothesis Driven

Setiap test punya hipotesis: "Kalau X, maka Y, karena Z". Contoh: "Kalau ganti CTA dari Sign Up ke Get Started Free, conversion akan naik 15 persen, karena reduce friction perception." Test pakai data.

3. Sample Size Planning

Pakai calculator (Evan Miller, Optimizely). Input: baseline metric, MDE (minimum detectable effect), power (80 persen), significance (95 persen). Output: visitor minimum per varian. Underpowered test = unreliable result.

4. Statistical Significance

p-value kurang dari 0.05 = 95 persen confidence beda bukan kebetulan. Effect size penting (perbedaan signifikan secara statistik tapi 0.1 persen ngga worth implement). Stop test setelah hit sample size, bukan saat 1 varian "menang".

5. Common Mistakes

Test 5 varian sekaligus tanpa correction. Cherry-pick winner. Ignore segment results (overall menang tapi mobile kalah). Implementasi tanpa rollout monitoring. Test trivial change yang dampaknya kecil. Mistake bikin keputusan salah.

YOUR MISSION

  1. Run first A/B test untuk fitur sederhana
  2. Pelajari Evan Miller calculator
  3. Build experiment dashboard
  4. Pelajari case study Booking.com
  5. Setup recurring experiment review

POWER-UP RECAP