Equivalence Tests

Reading the Book Statistics Applied to Clinical Trials I learnt about Equivalence tests. This is very interesting indeed. Situations exists where you want to test whether a treatment has the same effect compared to a placebo or compared to an existing treatment. This can be readily extended to situation I face on my daly work with Marketing Research.

For example, suppose the client wishes to change the methodology of their survey from telephone to Online interviews. This will make things much cheaper, but will it make results from the new methodology incomparable with results from the old methodology? To test that we can simply conduct surveys with both methodology at the same time. But then, how do you compare them as to say their results "are the same"?

I soon realized that the usual statistical test of hypothesis is not suitable for this kind of situation. So I quickly thought about something. My approach was to do the usual statistical test, but lowering the confidence level to, say, 70%. I want my test to have more power, so that when I stay with the null hypothesis I am more confident it is true. But to tell the truth I have never even used this because of tremendous difficult to explain people why the usual test is not correct and why this would be a better approach. The problem may be more that 70% is something that seems not as good as 95%, at least from the eyes of non statisticians. Then I came up with a second approach which I used a few times. Ok, we will do the usual stat test, but we will do it several times. We compare the surveys for 3 or 5 months and this will tell us whether there is a methodology effect, even if it is not statistically significant. The approach seemed to work, I could see in one case that the new method produced slight lower averages that were not statistically significant though. Now we have a better idea of how much interview methodology affects survey results and we can even make the decision without relying on statistical tests.

Comparison of interview methodology is just one thing that requires equivalence testing. Many others exists, specially in product testing, when the company wants to launch a new product that is cheaper to produce and we need to test whether the perception of the consumer will be the same.

The approach for Clinical Trials, as showed in this book, is different and easier in a way. You first define what a significant difference would be from practical (not statistical) point of view. One might say, for example, that if differences are lower than 3% they can be considered as equal. Now we can construct a confidence interval for the difference of the means and if this interval is within the range [-3%;3%] we can consider things equivalent. I have to say, though, that this tolerance range is not easy to define in some cases, to the point that I am not sure this could be applied on my type of problems. But I might try.

I searched the internet for equivalence testing and found that there are other ways of doing the test. The book just explained the simplest way of doing it (as I would expect - read the critics I made to the book) and perhaps the oldest way. But the good thing is that it showed me the issue and I can look for it elsewhere...