About The ARDA | Tools | FAQs | Contact Us
Administration Methods

There are many ways of administering a questionnaire to respondents, in terms of the physical tools used, the setting, and the procedures. Each study reports how its own data collection was done, but there are also many textbooks offering general principles. Especially well known is the work of Donald A. Dillman, who has long sought to find the best principles for gaining a high response rate on questionnaires administered via mail or online. There is no one best way, however, and both the scientific goals of a project and the practical challenges it faces shape design decisions.

Pencil-and-paper methods have a wide tradition, whether the questionnaires are handed out in group settings or sent through the mails. As anyone who has done a survey of this kind knows, however, considerable work must be done after the questionnaires are in hand, not only entering the data by hand into a computer, but struggling to be certain which of the boxes was really checked when a mark was put far to the side, and what word was really intended in a scrawled write-in item. Since the late 1930s optical scanning of pencil-marked forms has been used in educational settings, but non-students cannot be counted upon to mark the “mark-sense forms” clearly.

Some of the most expensive datasets in the ARDA, such as the General Social Survey were obtained through face-to-face interviews. A trained interviewer visits the home of the respondent, and spends about 90 minutes asking the questions out loud. This form of administration generally requires that the questions be designed with simple answers, so the respondents will remember the options, but for some items the interviewer may hand the respondent a card showing the responses. The costs for each interview can be considerable, given that the interviewer must be paid for all the time and travel, and sometimes multiple visits are required to find the target person at home, so it is impractical to work with a random sample of the entire nation. Cluster samples are generally used, selecting a number of households in each of several geographic areas, often following elaborate plans to compensate for selection biases.

Telephone interviewing is a common compromise. For several decades interviewers have worked from a computer, to avoid a separate data-entry stage to the work, in what is called CATI (computer-assisted telephone interviewing). An example of the refinements that can be accomplished with telephone interviewing is the Lilly Survey of Attitudes and Social Networks. The sampling was done on the basis of phone numbers, as the ARDA description of this project explains: “Using a random list of prefixes from Survey Sampling Inc, and Random Digit Dialing for the last four digits, telephone numbers were called with the goal of speaking to the person 18 years or older who had the next birthday.” Once a phone number had been selected, the interviewer tried as many as eight times to call, and even called back people who initially were reluctant to participate. Some of the Hispanic respondents were interviewed in Spanish. Despite its advantages, CATI has become increasingly difficult, because the world’s communication system is moving away from traditional land-line telephones, and it is no longer reasonable to assume that each household has one phone, whose area code and exchange numbers define its geographic location. Cell phone numbers may not be suitable for rigorous sampling; people using them may be especially unwilling to respond to a long series of questions, and if they are in a noisy public place they may not be able to concentrate on the questions.

In the ARDA data, the American Values Scale, 1985, is an early example of direct computer administration of a questionnaire to respondents, being but a small part of the huge Computer Administered Panel Study which University of North Carolina undergraduates spent an hour a week for twenty weeks answering on computer terminals. In the late 1990s many long questionnaires were administered over Internet, sometimes to tens of thousands of respondents, but the respondents were very far from a random sample, both because many people still did not have Internet access, and because the response rate of people invited to participate was low.

Given the cost of face-to-face interviewing, the declining conditions for CATI, and the biased nature of Internet samples, many researchers have had to become “philosophical” about their methods of questionnaire administration. No one method works best, and a really serious study may need to combine them. Or rather, any topic deserving really serious attention requires multiple studies, using different methods whose advantages and disadvantages balance out. For example, a very large number of items may be administered to a very large number of people online, exploring a vast intellectual territory in the social science of religion, but then a carefully selected subset of the items would be administered to a high-quality sample by more traditional means to anchor the most important parts of the research in something like a random sample. In the existing ARDA databank, major questions are often addressed by similar but different items in questionnaires administered by different means, and the greatest confidence should be felt about the results that are consistent across several studies.

Dillman, Don A., Jolene D. Smyth and Leah Melani Christian. 2009. Internet, Mail and Mixed-Mode Surveys: The Tailored Design Method, 3rd edition. John Wiley: Hoboken, NJ.

QuickSearch The Knowledge-Base

To search the knowledge-base, enter a term below:

Select a Theory below to learn more:

Select a Concept below to learn more:

TCM Contributors

Would you like to be considered for a position on the Theories, Concepts, and Measures contribution team? If so, click on the link below and complete the TCM Online Request Form.

TCM Online Request Form

If you are already a contributor for Theories, Concepts, and Measures and need assistance in managing the content, please click on the link below for instructions.

Site Administration Instructions