Surveys - different methods and instruments

The last section looked at some definitions of surveys, and explored some situations where surveys could be used. Here we will look at some of the techniques used to administer the survey.

The basic definition of surveys is that they involve asking questions and collecting data from people. The commonest method for doing this is by a series of highly structured questions, with responses pre selected. There are two main methods of data collection in surveys:

Structured techniques
Semi-structured techniques

And a number of ways in which surveys can be administered.

Structured techniques

As their name suggests, these arrange for data to be collected in a highly structured way. All the variables will have been identified, and the interview structure, and the way the data will be coded, will be predefined. The advantage of this sort of data is that it is very easy to code and analyse, because the type of answers which the respondent can give is set out in advance.

The instrument (the means whereby the data is collected) is highly structured and specific, and generally comprises either:

  • a written questionnaire
  • a verbal interview, in which case the interviewer becomes part of the instrument along with the interview schedule, although the high degree of structure maintains the objectivity.

The type of questions are generally cloze-ended, meaning there are a set number of responses. Open-ended questions require the user to supply the response.

Questionnaires are said to be self-administered when they are filled in by the respondent on his or her own, and researcher administered when administered by a researcher in the form of an interview. The following are some advantages of both types according to Wikipedia:

Self-administered:

  • Cheaper
  • Don't require interviewers
  • Can be used for large numbers
  • Avoid interviewer bias
  • Quick and easy to code and analyse

Researcher-administered:

  • Can clarify questions
  • Ensure completion of questionnaire
  • Higher response rate
  • Greater control of environment

Questionnaires

These are a highly effective method of data collection, in that they require less time to administer and are therefore less expensive, and permit data collection from a larger sample. The advice generally given is to use cloze-ended questions, which are easier to code, store and analyse.

Structured interviews

Here, the interviewer has a number of structured questions, which he or she goes through with the respondent. The responses are either articulated by the interviewer, or else are present in the coding structure which the interviewer fills in as the respondent talks. The advantage is that the interviewer can develop a rapport with the respondent, which increases the propensity for an honest, and accurate response, and can clarify any misunderstood questions.

In 'Survey research design' by Joseph Janes Library Hi Tech, 2001, 29:4, 419-421, Janes has this to say about interviews:

'Most authors agree that the face-to-face interview method can get you the best, highest-quality data. You can ask more questions, and more specific questions. You can know when people are in trouble and not understanding you, and get a higher response and completion rate. You can also decrease the number of "don't know/no opinion" type responses. With a trained interviewer, you can also get non-verbal behaviors, but that takes work. A face-to-face interview should last about 15-25 minutes; anything over about 45 minutes gets tedious and people can become restive. Interviewers need to be trained to be familiar with the questions, and to ask them as neutrally and consistently as possible, in order to eliminate potential sources of bias and noise in responses. In fact, in some situations you may want to use many interviewers of different ages, genders, races, or backgrounds, to mix up any such possible sources of error. Responses should be recorded exactly (or even tape or video recorded, though this has to be agreed to by subjects in advance). It is an expensive proposition, but sometimes the benefits are worth the effort.'

In 'Scenario planning: strategic interviews and conversations', foresight, 2002, 4:1, 19-30, John Ratcliffe describes both structured and unstructured interviews in his research on the role of conversation in planning strategy. Read the article here.

Semi-structured techniques

These arrange for the collection of data which does not necessarily correspond to a pattern. The advantage is that because data does not need to correspond to preset variables, it is easier to explore dynamic and changing situations.

Semi-structured interviews

Here, the interviewer has a number of questions on the schedule, but can depart from these as appropriate should other relevant issues crop up. There is no set pattern for responses and the respondent is free to respond as he or she sees fit, while the interviewer is at liberty to probe as interesting new issues emerge.

Whereas with structured techniques, all variables are known in advance, with semi-structured techniques it is possible to start with an incomplete knowledge of variables which is a good way of obtaining more in-depth data and further exploring a particular situation.

'Are women better at organisational learning? An SME perspective', Lynn Martin Women in Management Review, 2001, 16:6, 287-297 - Martin uses semi-structured interview techniques. Read the article here.

Ways in which surveys can be administered

By mail - self administered

This is an excellent way of sending the same instrument to a large number of people, but the response rate may be low as you are relying on people having the time and the motivation to fill it in. In addition, you are dependent on having identified all your variables, and it is not the best way of handling data about a complex situation.

'Survey research design' by Joseph Janes Library Hi Tech, 2001, 29:4, 419-421, has a section on self-administered mail surveys and the main issues to be aware of. Read the article here.

The reliability of mail surveys is reported on in 'Mail survey reliability through follow-up mailings: the case of auditor changes' Kimberly A. Dunn, H. Fenwick Huss, in Managerial Auditing Journal, 2004, 19:8, 1048-1054. Read the article here.

Group-administered questionnaire

Here, people are gathered into a group and handed the instrument which they fill in in situ. This allows for collection and administration at one point in time, but it needs to be set up.

An example here is the end-of semester student satisfaction questionnaire, where students are often requested to fill in a questionnaire as part of a lecture, as a way of ensuring response.

Personally administered, in the mall/street or at home

Here interviews are conducted with people who are intercepted in the street or visited at home. This ensures a reasonable response rate, but the cost is high and people are likely to walk quickly past if they are in a hurry! An advantage is the ease with which you are able to obtain a response to something that that requires the senses - sight, touch, smell.

Telephone interviews

The advantage is that these can be cheaper than the mall/street interview, but the big drawback is that most of us resent the interruption of being phoned up for a 'market research' interview when we are having a meal, working, watching television etc., and telemarketing has given telephone interviews a bad name. In addition, it can be difficult to take people through multiple response items over the phone as there is a danger that the first item is forgotten by the time the last one is reached.

Online or email

The use of these has been growing, and they are very quick and cheap to deliver, particularly online interactive forms. The drawbacks are: computer glitches, hostile attitude to spam (arguably we are more likely to delete spam than we are to fail to open unsolicited mail), and (although with the huge growth in internet access this is becoming less of one) you limit your population to those with internet access.

In 'A comparison of online and postal data collection methods in marketing research', Marketing Intelligence & Planning 2003, 21:2, 85 - 95, Heath McDonald and Stewart Adam report the results of their studies into the relative effectiveness of online and mail surveys. Read the article here.

In 'Developing automated e-survey and control tools: an application in industrial management' Industrial Management & Data Systems, 2004, 104:3, 189-200 the authors look at a particular example of an e-survey tool. Read the article here.

The following screenshot shows an example of an online survey, and is a particularly good example of one that is very easy to fill in:

© The Trainline.com

Different ways of administering surveys are explored in 'User survey at Waterford Institute of Technology Libraries: How a traditional approach to surveys can inform library service delivery' in New Library World 2005 106:1, 43-57 by Helen Hayden, Terry O'Brien, and Maoilíosa Ó Rathaille. Read the article here.

Surveys - planning issues