Voluntary surveys, mandatory surveys and StatsCan expertise

This post was written by Kevin Milligan of the Department of Economics at the University of British Columbia.

I'd like to add something to Stephen's "Blame Stats Canada" post from yesterday about the government's response to the Census imbroglio.

The claim by Minister Clement is that we should trust Statistics Canada on this issue.  Moreover, Dean Del Mastro, MP for Peterborough, said the following during his interview this week on CBC Radio (20:01 of the interview here): 

voluntary questionnaires have never been demonstrated to be less accurate than mandatory questionnaires.

So what we're looking for is (a) scientific evidence on the relative accuracy of voluntary and mandatory surveys and (b) the advice of the experts at Statistics Canada.

Good. Well, look at this. It is a Statistics Canada publication that provides scientific evidence on the relative accuracy of voluntary and mandatory surveys.

Look specifically at page 23, Table 4, column 1. You will note that the (voluntary) Survey of Labour and Income Dynamics drastically under-represents the incomes of those at the bottom of the income distribution when compared to the Census–average incomes are off by more than a factor of 2.

That's not small.

I’m interested to know whether the Minister is curious about this finding. I hope he’ll ask his Statistics Canada advisors about it.

I should note that this is only one, and not the only, example of the more general point that voluntary surveys are often non-representative.  That's why we need the Census so dearly–it allows us to correct for the skewed results in voluntary surveys through re-weighting corrections. You can only do the re-weighting corrections if you have a solid population benchmark like the long-form census.

One final note. This concept is very intuitive. Dean Del Mastro gets it, too. Check out what he says at 16:10: 

polls are generally selected by random groups, that's how you make sure you're not skewing data.

Precisely.  A random sample–when those randomized 'in' all respond–provides non-skewed results. (With sufficiently large samples, of course.  Law of Large Numbers, Central Limit Theorem and all that). I'm glad that Mr. Del Mastro understands. I hope that he'll have a word with the Minister to explain it.

6 comments

  1. Unknown's avatar

    Great post Kevin!

  2. westslope's avatar
    westslope · · Reply

    The Globe and Mail article Tories refuse to reverse census decision has garnered 1,073 comments to date.

  3. Your Blog Doesn't Deserve My Identity's avatar
    Your Blog Doesn't Deserve My Identity · · Reply

    I laugh at your pitiful attempts to use evidence to convince conservatives of anything. Evidence doesn’t work against the powers of rhetoric!
    This government doesn’t care about evidence and it is blatantly clear from the consultations and this census debacle.
    Tony Clement has a history of ignoring evidence, go and see what the doctors told him about safe injection sites.

  4. crf's avatar

    Tony can’t hear you, no matter how loud you shout. He’s travelling with his buds in an awesome new Mach 2 fighter jet.

  5. Lord Kitchener's Own's avatar
    Lord Kitchener's Own · · Reply

    For the record crf, the F-35 can’t actually do Mach 2.
    Technically, the F-35 is slower (though only marginally) than our current F-18s. The F-35’s max speed is Mach 1.7, whereas the F-18 can do Mach 1.8.

  6. Unknown's avatar

    I think our country is finished: we spend billions on fighter aircraft that that don’t go faster than the existing aircraft, we are going to spend millions more on statistical surveys than will then fail to yield the accuracy of existing surveys, we consistently elect minority governments that act as though they are majority governments, and we elect members of opposition parties that are clueless about responsible government and utterly fail to oppose, and in fact support, the aforementioned government, because they gutlessly fear electoral consequences at the hands of that same grossly manipulative minority government. And soon, they won’t even be in a position to judge potential electoral consequences, because accurate riding socio-economic data won’t be available, thanks to the government!

Leave a reply to westslope Cancel reply