Bias: Holes in the Data

There are many types of possible bias in research. This is true for experimental and quasi-experimental designs alike. Ideally, the peer-review process for publishing research catches such methodological flaws in a research program, however, sometimes bad information leaks out.

One type of bias involves sampling. Sampling bias occurs when the sample size selection of participants or subjects for an experiment does not consider possible homogeneity of participants due to ethnicity, age, technological expertise, access to the internet, having access to a telephone, or even having a home.

Case in point, a complex sociological survey: In 2006 the North Carolina Wildlife Resources Commission (NCRWRC) received an exhaustive research report on the attitude of public opinion regarding hunting on Sundays. (Hooper, et al., 2006) As a state with a high proportion of hunters, and one of only a few states that statutorily prevent hunting on Sundays, there was great interest in finding out what the populace in general, both hunters and non-hunters alike, thought of the legislation.

To investigate this, Hooper, et. al. created a multi-modal design. A survey was developed and employed in two distinct modes. A scientifically derived model was used to conduct a telephone survey. This in itself is fraught with potential problems as this inherently precludes the participation of people who are homeless, people with unlisted telephone numbers, people on the national do-not-call registry, or people who don’t have phones. If the model was not designed properly, even the day and time of day when the phone calls were placed would introduce the possibility of bias into the sample as hunters would probably be less likely to answer their phones on a Saturday morning while hunting an elk, etc. In the end, the phone survey of 1000 subjects showed 25% of respondents were in support of hunting on Sunday, 65% were opposed, and 10% had no opinion.

Hooper, et. al. also conducted an online version of the survey. The survey was posted on the NCWRC web site and returned very different responses: 55% in support, 43% opposed, and 2% had no opinion. A completely different type of respondent had participated in the online survey. However, it is also possible, although unlikely, that a small group of respondents repeatedly answered the online survey to skew results in their favor. Either way, there was a clear sampling bias in the online survey.

Another example of bias in research can be seen in a survey on alcohol consumption (Lahaut, Jansen, Van De Mheen, & Garretsen, 2002) that could be seen as being both medical and sociological. In this case, the research paper was investigating non-response bias. It had previously been shown that several factors exist that can cause distortions in response rates leading to potential over-representation of participants who were addressing topics of interest to them, as opposed to people for whom the topic seemed irrelevant. Conversely, there is often under-representation of participants who found admitting to a particular behavior that is considered socially questionable or even unacceptable might cast them in a negative social light.

The authors conducted a mail survey. After the responses were received the authors then did a follow-up study of the non-respondents. They visited the non-respondents individually and again asked for responses. This in itself may introduce bias as the anonymity perceived by initial respondents was not present in the follow-up survey which would lead at least some of the respondents to respond in whatever manner was perceived to be socially desirable. However, methodological issues aside, the follow-up survey found an over-representation of non-respondents who abstained from drinking alcohol. Presumably, this was due to the topic of alcohol consumption being of little salient value to them.

References

Hooper, M., McClafferty, J., McMullin, S., Duda, M., DeMichele, P., Jones, M., . . . and Craun, C. (2006). An Assessment of public and hunter opinions and the costs and benefits to North Carolina of hunting on Sunday. Retrieved April 08, 2014, from http://www.ncwildlife.org/portals/0/News/documents/SundayHuntingStudyNov06.pdf

Lahaut, V., Jansen, H., Van De Mheen, D., & Garretsen, H. (2002). Non-response bias in a sample survey on alcohol consumption. Alcohol & Alcoholism, 37(3), 256-260.