Analytics Speak

There are so many questions plaguing analysts, one of the biggest ones is:

Which statistical test do I use?

Disclaimer: The content of this post has been well researched, but if you do find errors big or small, please leave a comment, and we can discuss the point of contention.

Now, back to statistics. The #1 reason analysts use statistics is to test for significance i.e is my answer/data statistically significant?

What you need to do
1. Formulate a hypothesis - Are my CTR's statistically significant?
2.

Having researched this over and over to find the right tests - these are my conclusions.

There are 3 tests as they relate to A/B testing

1. Z test
2. Chi square test
3. Fishers Exact Test

Multivariate Testing
1. Full factorial tests/Taguchi method - reduces the sample size required to get to an answer.

2.

Both omniture SiteCatalyst, Google Website Optimizer use full/fractional factorial tests for MV testing.

For A/B testing however, Google does not mention how or what statistical tests they use to discern a winner.