Weights in Non Probabilistic Surveys

The fact that we really do not have a standard procedure to deal with precision of estimates in Non Probabilistic Samples has bothered me for some time. In the business world there is not a single survey that is true probabilistic and few of them are close enough that we feel comfortable making the assumption of probabilistic samples.

Samples are often weighted to some standard demographic variables, like age, gender and region. And this brings another layer of difficulty since now precision measures are not only calculated in a Non probabilistic sample, but it is also calculated without accounting for the weights.

Recognizing that Probabilistic Samples are not the reality of our world, we need to be pragmatic and use our knowledge of Sampling Theory to get the most we can from whichever sample we manage to get. If we are comfortable enough to make the assumption that the sample is not strongly biased then I believe it makes sense to think about variability and margin of error, even if they have to be labeled at the bottom of the page with an asterisk that states our assumptions. But, assuming fair to calculate variability, how do we proceed on with the variance calculation in presence of weights?

I have not found much literature on this, likely because Non Probabilistic Samples are mostly ignored as being practice of people that does not know Sampling Theory (I might comment more on that in another future post). But I do know that the only weight applied to these surveys is the post stratification weights, were some demographics (and rarely other things) are corrected to mirror some official figures. When weights are based only on Age and Gender, for example, it makes sense to me to analyze the results as a Stratified Random Sample where Age and Gender are interlocked strata. Often, though, they are not weighted jointly, using a two way table, but through their marginal distribution. Still I think it might make sense to define each different weight factor as a different stratum when calculating variability.

More problems arise when we weight marginals of many variables. Defining different weights as different stratum is not doable anymore since we can have hundreds of strata. In this case I have done some tests using SPSS Complex Sample and Stata and declaring the sample as being Simple Random Sample, as if the weights were the simply the inverse of the probability of selection. Variance calculated in this way are for sure much more realistic that the ones SPSS or Stata calculates with what they call frequency weights. Defining post stratification weights as frequency weights in variance calculation is everything we need to get the wrong figures and fool ourselves.

Usually the Simple Random Sample specification will show how weighting data increases the variances. This is a big improvement over not weighting or considering weights as frequency, even though it might not be the best improvement one can have. To me now we are down to a more difficult question which is the question about the validity of the assumptions we need to make to be able to calculate variances and take them as good estimates of population parameters. But this can be the subject of another post...