Statistical significance is one of the most misunderstood concepts in all of marketing research. It sounds so scientific. So precise. So reliable. Shouldn’t I ignore anything that isn’t statistically significant?
We’ll save for another day the different ways you can determine statistical significance, for today we’ll just concentrate on the difference between being significant and being important. Many times, we assume that statistically significant is the same thing as being important. Or, potentially more dangerous, that a finding isn’t important because it isn’t statistically significant.
You can think of decisions as falling into one of these squares:
Boxes “A” and “D” give us few problems. We’ve conducted research and we know that a given finding is important to our business and we can trust that the research is giving us an accurate estimate of what the “real world” answer would be – it’s not just the result of random chance. And “D” is a no brainer. The answer is not important to the business so why would we care if it’s statistically significant?
The troublesome outcomes are boxes are “B” and “C”. Lots of information developed through marketing research may not be statistically significant – it might be qualitative data, social media data, or simply not asked of a large enough sample size to be significant. BUT IT’S STILL IMPORTANT, and it would be very unwise to ignore it just because it doesn’t carry the designation of statistically significant.
And then there’s box “B”. Tons of findings are statistically significant, but do not have the power to move your business at all. This happens all the time when we let our research balloon up to include many “nice to know” questions. Get rid of them. Your research will be cleaner, more precise and more accurate. And significant findings will be important to you.