Questionnaire

An important and widely used Research Method that can be very useful when done correctly A quantitative approach aimed at finding information that can be expressed in numbers, charts and graphs A mandatory approach for this course

Definition

The term Questionnaire refers to:

All methods of data collection in which different people are asked to respond to the same set of questions in a predetermined order

Strengths
provide the possibility of surveying large and geographically dispersed groups are relatively low costs (answers may be submitted by post) Provide data that can be calculated and converted into charts and graphs Offer a lot of data in a limited time Technology makes analysis quick and easy There is reasonable anonymity for respondents It can be completed at the time and location of the respondent's choice It provides 'hard' figures which can be used to support various decisions.

Weaknesses
Broad information, but little depth Pre-structured response options and closed questions can lead to a lack of nuance answers must be restricted to options that do not allow detail or grey areas There is no room to adjust the process part way through or to improvise. Even the smallest inaccuracy in the question wording will make the data for the question worthless Some respondents will lie or provide socially desirable answers

Response rates
Different methods have different success rates
Online 10% (Internet) to 30% (within organisations) Postal Around 30% is considered reasonable Delivery and collection (actually picking it up) 30-50% Face to Face / Structured Interview 50-70%

Each delivery method has advantages, but remember:

The success of your questionnaire is dependent upon the will of respondents to participate.

What for?
Questionnaires are usually used for specific types of information about the target group:
Who they are (Personal attributes: age, gender, etc.) What they think (Attitudes, Beliefs, Opinions) What they know about a specific topic What they do (behaviors and routines)

The right mix of questions can yield valuable info

Introduction
start with an introduction or a cover letter explaining:
Who you are

what youre doing and why

what youll do with the information

Questions
Questionnaires have to result in information that can be expressed in numbers. Coding You will need to attach a numerical value (a number) to each answer choice
Yes = 1 No= 0, etc.

This is really important when it comes time to process your data in a spreadsheet

Question Types: Open Questions

Open questions cannot be analysed with math. Keep them to a minimum or avoid them. If you want to ask mostly open questions, do an interview or a focus group Please, tell me how you feel about your new job

List 3 things you like about your job

1 .. 2 .. 3 ..

Question Types: Closed Questions

What do you like most about your job? (Tick one alternative):
the salary the people you work with the opportunities for further schooling other, namely: __________________
(1) (2) (3) (4)

Types of closed questions

Quantity questions Dichotomy questions Category questions List questions Rating scale questions Ranking questions Grid questions

Quantity
Sometimes you may want to ask respondents to write down a number (their age, for example)
Use clear boxes to show where to fill in the answer You can use the numbers as codes
How many Children do you have? In what year were you born?

Dichotomy
only 2 possible answers, which are usually opposites
Used for demographic data
Gender: male/female Marital status: married/single

Sometimes used for attitudes and experiences

Yes/No Agree/Disagree

They are limited in scope Please tick the box indicating your gender
Male Female (1) (2)

Category
Category questions are questions that ask respondent to select a single answer (called a category)
Used to find out about preferences, attitudes and behaviour Only one category can be chosen Theres a limited number of categories The options must be mutually exclusive (no overlapping) The options must cover all possibilities

Which car colour would you prefer?

Blue Red Yellow
Green Other, namely: __________
(1) (2) (3) (4) (5)

Lists (1)
List questions are used whenever multiple answers are possible
Often used to find out about a respondents experience, behaviour or routine You provide a list of possible answers Note the coding in the example - this is actually five questions in one. You are asking people 5 yes/no questions and must be coded as such. Have you cleaned rooms, have you done the shopping, etc.
Please tick the box in the provided column for services you provided as a home care volunteer in the past month. Multiple answers are possible.
Name of service cleaning rooms shopping bed making laundry other Provided?

(1,2) (1,2) (1,2) (1,2) (1,2)

(please describe:)

Lists (grid style)

Because lists involve essentially asking several questions at once, many researchers use a grid style for simplicity
Please tick the box in the provided column for services you provided as a home care volunteer in the past month. Multiple answers are possible. Name of service cleaning rooms shopping bed making laundry other Provided? Yes (1)

No

(2)

(please describe:) .

Rating Scale
Sometimes you want to measure attitudes/opinions on a scale to allow more variety than yes/no, agree/disagree
These are used to find out about peoples knowledge and opinions Note that its a good idea to include neutral opinions. Often the choice is labeled, No opinion/I dont know or Neither agree nor disagree

For the following statement, please tick the box which matches your view most closely. Tourism should always be sustainable.
Strongly disagree Disagree No opinion/I dont know Agree Strongly Agree

(1) (2) (3) (4) (5)

For the following statement, please tick the box which matches your view most closely: Tourism should always be sustainable.
5-point scale

agree

tend to agree

No opinion

tend to disagree

disagree

(1)

(2)

(3)

(4)

(5)

Rating Scale (10 point scale for detailed evaluation)

For the following statement please circle O the number which matches your view most closely

This trip was . 10-point scale

Poor value for money 1 2 3 4 5 6 7 8 9 10 Good value for money

In Rating scales you can decide how much detail you want
Do you agree or disagree with the following statement: CEOs of non-profit organisations shouldnt earn more than the prime minister. Rough measurement

I agree No opinion I disagree

(1) (2) (3)

More detailed measurement

I strongly agree (1) I agree (2) No opinion/I dont

know (3)

I disagree (4) I strongly disagree (5)

Level of detail

When asking about attitudes and opinions, its usually a good idea to have neutral options
Otherwise you risk manipulating respondents This could devalue results
Do you agree or disagree with the following statement: Penalties for driving offences should be much more severe.

Strongly agree Agree Uncertain Disagree Strongly disagree

(1) (2) (3) (4) (5)

Strongly agree (1) Agree (2) Neither agree nor disagree (3) Disagree (4) Strongly disagree (5)

Neutral opinions?

Ranking questions allow you to ask people about their priorities

Used to find out what people think is important by having them rank options. What do you look for in a new car? What political issues concern you most? etc. You can use the ranking numbers as codes

Please, number each of the issues listed below in order of importance to you. Number the most important 1, the next 2 and so on. If a factor has no importance at all, please leave blank.
Factor Health Education Human Rights Child Care Poverty Importance 5 2 4 1 3

Rank

Frequency questions refer to how often people do things. They are used to find out about behaviour and routine. The next questions concern your media habits. Please tick the box which most closely matches your experience.
I read one or more magazines weekly (1) monthly (2) every 3 months (3) every 9 months (4) less often (5) never (6)

Frequency

You can also use a grid to get information about frequency. It enables you to ask multiple questions in a quick format that is easy to fill in For each of the following statements please tick the box which most closely matches your experience
monthly every 3 every 9 months months
(1) (2) (3)

less often
(4)

never
(5)

I read newspapers ..
I read one or more magazines

Frequency (grid)

Things to keep in mind:

When asking closed questions, its important that all options be available. But what if its not possible to include all options?Sometimes, its good to include the option: Other, namely" and allow respondents to fill something in
Keep in mind that you can code other, but will have to record and analyse open answers separately

Other, namely (example)

Which car colour would you prefer?
Blue Red Yellow Green Other, namely:
(1) (2) (3) (4)

(5)

Filtering
If your questionnaire has follow-up questions, you need to provide clear instructions
If one answer determines what question the respondent should answer next, or if questions need to be skipped, make sure the respondent knows what to do

Are you currently a member of an NGO? Yes (1) No (2) If no go to question 24

Checklist for Questions

Each question
Should only contain one question/variable. People often accidentally ask multiple questions at once Must be specific enough Must be clear and unambiguous Must not be too complex Must not be (doubly) negative
If you ask should smoking be allowed you get more trustworthy results than if you ask should smoking be banned because people naturally avoid negative phrasing Starting questions with dont you think implies a desired answer Double negatives (dont you think this is not a good idea? confuse respondents

Must not have an obviously correct or socially desirable answer (biased or leading questions) Must not be too sensitive or unpleasant

Checklist for Answers

Each answer choice:
Must have the right level of detail not too much or too little Must be mutually exclusive you want to use answer choices that rule each other out Must be stated in a logical order Must cover all possibilities Must keep positive and negative options balanced If necessary: include an option that is partly open, such as:
other, namely:

Bad examples * It is not true that smoking doesnt affect your health?

Improved Examples

* Do you believe smoking is

dangerous?

* Do you believe he lied when

he denied his relationship with her?

* Leading/negative

* Do you believe he was honest about

his relationship?

* Do you think the teacher is an

expert in this topic?

* Biased/negative

* Do you think the teacher effectively

answers questions from the class?

* What is your opinion on the

* 2 questions at once

* Is the respondent qualified?

* Make 2 questions:
*

contents and the assessment of the course?

* What is your opinion on the contents

of the course? What is your opinion on the assessment of the course?

Good and Bad Examples

Good and Bad examples

Bad example
Improved example

* What percentage of your

weekly income do you spend on cigarettes?

* Make 2 simple questions

* What is your income per
month?

* Requires calculation and will

take time

* Approximately how much

money do you spend on cigarettes each week?

* It should be easy for

respondents to answer questions

Bad examples

Improved examples

* Do you agree with the Surgeon

* What if they didnt read the
report?

General's conclusion in his latest report on cigarette smoking?

* Do you agree with the Surgeon

General's latest report that cigarette smoking in public places is never acceptable?

* Most doctors believe that

* Implies a desired
* Vague
answer/judgmental

exercise is good for you. Do you agree?

* Do you believe that exercise is

good for you?

* When did you last see a movie? * How fast was car X going when
it smashed into car Y?

* When did you last go to the

cinema? * How fast was car X going when it hit car Y?

* Potential Bias

Good and Bad Examples

Bad examples How often do you use Blackboard?

Improved examples How often do you use Blackboard? Never Up to once a week Twice a week More

Never Sometimes Always

(1) (2) (3) (4)

(1) (2) (3)

* Variables arent realistic

What is your monthly income? Requires thought and could make someone uncomfortable

My income is: Less than 30.000 (1) From 30.000 up to 50.000 (2) More than 50.000 (3)

Good and Bad Examples

Bad examples How many times per month do you play tennis? 1 to 5 times (1) 5 to 10 times (2) more than 10 times (3)
These answers overlap

Improved examples How many times per month do you play tennis? 1 to 5 times (1) 6 to 10 times (2) more than 10 times (3) Which sports do you practise? Soccer (1,2) Tennis (1,2) Field hockey (1,2) Other: (1,2)

Which sports do you practise? Soccer (1,2) Tennis (1,2) Field hockey (1,2)
Not all options are present

Good and Bad Examples

Questionnaire Lay-out
Introduction or cover letter
1. 2. 1. 2. Explain who you are, what youre researching, and what youll do with the answers Is it anonymous? Put the questions into sections (by subject, for example) and make sure theyre in a logical order Use mini-introductions to provide context

Clear structure

It should be attractive and readable (fonts, etc.) It should include a mix of question types It should be inviting it should encourage participation It should be short
1. 2. The longer it is, the less motivated the respondent There should be no more than 20 questions, give or take

Cover Letter
Banner / logo Date, place Salutation Introduce the questionnaire Indicate time Indicates topic and identifies researchers In full Dear <title and name> State what the research is about, why it is useful Say how much time is needed to fill it in (be aware that you want it to seem easy) Stress confidentiality and anonymity

What will you do with information?

Whats it all for?

State how results will be used

Pilot Testing
You write the questionnaire and get a small number of people to fill it in for you This will allow you to find any problems in the questionnaire (mistakes, badly worded questions, etc.) and to refine your questionnaire You can observe or interview the people who fill the pilot test in or you can add a few questions to the questionnaire in which you ask for feedback about the questionnaire itself

Expert testing
Helps you determine representativeness of your sample and suitability of your questions Can help you see if the questions will lead to valid content

Testing with respondents who are in your target group

Enables you to make sure the questionnaire suits that group

Pilot Test
In the pilot test you ask the testers about:
The clarity of the instructions Which, if any, questions were ambiguous Which, if any, questions the respondent felt uneasy about answering Whether in their opinion there were any major topic omissions Whether the layout was clear and attractive Any other comments they might have

Pilot Test Report

A Pilot test report is a brief (1-2 pages) evaluation of your Questionnaire based on the results of your test 1. You evaluate your questionnaire 2. You state what you learned from the test

3. You explain any changes that need to be made / have been made as a result

What now?
Next week you will discuss your Research Plans and get feedback
But you need to get started on your questionnaire already

In Week 6 you will bring a draft of your questionnaire for feedback During the break, you will conduct the pilot test In Week 7 you will hand in the pilot test report