Hw.1

Questionnaire design: the good, the
bad and the pitfalls

Denise Thwaites Bee, Deborah Murdoch-Eaton

Academic Unit of Medical
Education, The Medical School,
University of Sheffield, Sheffield,
UK

Correspondence to
Professor Deborah Murdoch-
Eaton, Academic Unit of Medical
Education, The Medical School,
University of Sheffield, Beech Hill
Road, Sheffield S10 2RX, UK;
[email protected]

Received 21 September 2015
Revised 27 January 2016
Accepted 29 January 2016
Published Online First
24 February 2016

To cite: Thwaites Bee D,
Murdoch-Eaton D. Arch Dis
Child Educ Pract Ed
2016;101:210–212.

You have a question, or want to find out
current perceptions about a subject, and a
comprehensive literature search does not
give the answer. A questionnaire or
survey, if appropriately designed and
administered, can be an easy and efficient
way to collect data. However, a well-
designed tool is essential to provide
meaningful answers.
Guidance on good questionnaire design

is available.1–4 This can be framed
around three simple steps: preparation—
evaluation—delivery. Analysis and inter-
pretation are the final stages of complet-
ing the research.

PREPARATION
Is a survey method the most appropriate
research tool to answer the question?
Questionnaires are useful to investigate
opinions or attitudes of a population. If a
questionnaire is chosen as the research
tool, the next step is to identify whether
a validated instrument already exists. If a
tool needs to be designed, what format
would be of greatest value in answering
the enquiry; a structured interview or a
self-completed written form? The latter
can gather a large amount of rich data,
while the former provides a deeper
understanding through semistructured
questioning.3

Self-completed questionnaires require
careful construction with clear articula-
tion of purpose. Their success depends
strongly on format as well as the
wording; use an attractive, easy to navi-
gate presentation and ensure the length is
kept as short as possible. Consider
whether to include open or closed ques-
tions, or a combination of both.
Questions should only include a single
point, written unambiguously and con-
tained within short sentences. Wording
should be appropriate for your survey
population and avoid jargon to reduce
potential confusion. Closed questions can
provide large amounts of easily handled

(often numerical) data. Open questions,
as in free text responses and larger inter-
view surveys will collect rich information,
but will require considerable resource
time for the analysis, including methods
for sorting and coding of the data.

Sampling
How much data are needed to answer
the question? The intention is that results
from a ‘sample’ can be generalised back
to the whole population. Sampling within
the whole population, or subgroup, may
be the most manageable way to answer
the research question as the amount of
data gathered from a census would be
overwhelming. Recruitment can be by
advertising, although selection should
include random participation to reduce
investigator bias; response bias cannot
always be avoided.5 If a subgroup (eg,
ethnic, geographic, socioeconomic) is the
subject of study, then participant selection
needs to be targeted, systematic and con-
sistent. Despite this general ‘rule’ of
random selection, there is a place for
opportunistic or convenience sampling
(eg, finding participants from among
your colleagues or those attending a
meeting or lecture), provided the poten-
tial limitations are articulated. Identifying
appropriate sample size can be facilitated
using freely available on-line sample-size
calculators. The smaller the population
being studied, the greater the proportion
of this population the sample should be.
And however the sampling is done, it is
essential the limitations this places on
data interpretation are understood. This
is particularly true for convenience sam-
pling when participants may hold very
similar opinions and thus not truly reflect
the study population.

Response formats and scales
Many formats may be used as illustrated
in figure 1, the most common being the
Likert or modified Likert. Additionally,

RESEARCH IN PRACTICE

210 Thwaites Bee D, Murdoch-Eaton D. Arch Dis Child Educ Pract Ed 2016;101:210–212. doi:10.1136/archdischild-2015-309450

http://crossmark.crossref.org/dialog/?doi=10.1136/archdischild-2015-309450&domain=pdf&date_stamp=2016-02-24

http://www.rcpch.ac.uk/

http://ep.bmj.com

simple binary decisions (yes/no) or items with mul-
tiple choice options can be very useful. The scale
choice should be designed to contribute positively to
data collection. Confusion within questionnaires arises
when combining or changing formats; these should be
well signposted to ensure participants continue to
respond appropriately.6 Including some free text items
is valuable in exploring context, broadening the scope
of answers and providing rich data to enhance numer-
ical result interpretation.

EVALUATION
Field testing a new instrument is essential to evaluate
whether it will provide the information needed for
the study, and that final versions will elicit reliable and
valid responses. Questionnaire evaluators should
include researchers, stakeholders and respondents. A
clear brief should cover purpose, clarity and layout,
and whether items are likely to be well understood in
a similar fashion by all respondents. Acceptability and
feasibility are important considerations and should
include both questionnaire size and length of time
required for completion; these should not exceed the
participants’ patience threshold. Negative influences
of length on response rates and quality are frequently
seen in the latter portions of lengthy questionnaires.7

Students are particularly prone to ‘survey fatigue’
from frequent in-course evaluations.

DELIVERY
Even though questionnaires can easily be kept
anonymous, ethical approval is necessary, especially if
publication is desired. Consent may be considered
implicit through return of a completed form, and an

opening statement or a covering letter can make this
explicit. Questionnaire distribution should be tailored
to the study population to enhance return rate. Postal
paper questionnaires or relying on opportunist
encounters can be useful. Electronic means can reach
large numbers of potential respondents and thus
create a large data set. When using email, make sure
the ‘bcc’ and not the main or cc addressee lines are
used to maintain confidentiality. Accompanying
material, attached or accessed via a web link, provides
valuable insight into the importance of the research
and thus enhances participant interest and completion
rate. Consideration of when would be the best time to
undertake the survey will depend on the population
sampled. For example, avoiding exam time for stu-
dents or distributing questionnaires to a ‘captive’ audi-
ence during lectures is opportunistically useful. Small
rewards are often used for surveys, such as entry into
a ‘lottery’ or the gift of a chocolate bar, and enhance
return rate. However, this can be considered ethically
challenging, especially if the inducements are consid-
ered likely to significantly influence compliance and
should be discussed within ethical approval processes.

ANALYSIS AND INTERPRETATION
The chosen questionnaire format should include con-
sideration of how the data will be processed. The
availability of scannable forms or web forms such as
‘Survey Monkey’ or ‘Typeform’ provides automatic
collation, data management and often some statistical
analyses. When presenting results, provide compre-
hensive information on design methodology to facili-
tate a critical review of outcomes. Condensing data to
provide only the central tendencies, that is, means/

Figure 1 For the Likert-type scales, the question becomes a statement that can be agreed or disagreed with. Where subjects are to
choose a position on a line or within a number range, the ends of the scale must be explained and a clear set of statements or
anchors used. There is often a neutral point shown here as neither agree nor disagree, the number 5 and the vertical line on the
Visual Analogue Scale (VAS). Face-Pain Scales are useful for young children to indicate severity, with the assisting adult describing
pain levels (“this one hurts a lot and you want to cry”).

Research in practice

Thwaites Bee D, Murdoch-Eaton D. Arch Dis Child Educ Pract Ed 2016;101:210–212. doi:10.1136/archdischild-2015-309450 211

medians may hide extremes of opinion, thus an indi-
cation of range or SD should be included.
Connections between items or groups of items from
the questionnaire may be important, and factor ana-
lysis can demonstrate these and provide tool validity
assessments. Graphical representations provide valu-
able visual overviews of complex data.
Analysis of qualitative (free text) data requires

coding (or sorting) data into themes, which will form
the basis of an interpretive discussion. There are
helpful programmes that aid sorting (theming) of data
(eg, NVivo) but any systematic means of sorting com-
ments into themes can be used; cards with quotes
written on them are particularly useful for finding

themes. Qualitative data may also be presented graph-
ically (eg, flowcharts) and demonstrate data through a
density analysis. Remember that a single comment is
as valuable in understanding an issue and can be as
powerful as a commonly held belief.

SUMMARY
Preparation, evaluation and delivery of a survey
instrument are crucial. This includes well-researched
background material to confirm the question, ethics
approval, a consideration of validity and whether find-
ings might be generalisable. Surveys gather quantifi-
able data efficiently, but contextual richness and
interpretation often come through the free text.
Quotes can usefully illustrate your interpretations and
conclusions when presenting results (table 1).

Contributors Both authors have contributed equally to this
paper. DTB completed the first draft; both authors have
worked together on the subsequent and final versions.

Competing interests None declared.

Provenance and peer review Not commissioned; externally
peer reviewed.

REFERENCES
1 Woodward CA. Questionnaire construction and question

writing for research in medical education. Med Educ
1988;22:345–63.

2 Artino AR Jr, La Rochelle JS, Dezee KJ, et al. Developing
questionnaires for educational research: AMEE Guide No. 87.
Med Teach 2014;36:463–74.

3 Tavakol M, Sandars J. Quantitative and qualitative methods in
medical education research: AMEE Guide No 90: Part II. Med
Teach 2014;36:838–48.

4 Mathers N, Fox N, Hunn A. Surveys and Questionnaires. The
NIHR Research Design Service for the East Midlands/Yorkshire
& the Humber, 2009. http://www.rds-yh.nihr.ac.uk/
wp-content/uploads/2013/05/12_Surveys_and_Questionnaires_
Revision_2009.pdf (accessed 11 Jul 2015).

5 McFarlane E, Olmsted MG, Murphy J, et al. Nonresponse bias
in a mail survey of physicians. Eval Health Prof.
2007;30:170–85.

6 Jenkins CR, Dillman DA. Towards a theory of self-administered
questionnaire design. In: Lyberg LE, et al., eds. Survey
Measurement and Process Quality. John Wiley and sons,
2012: Chapter 7, p165–96.

7 Galesic M, Bosnjak M. Effects of questionnaire length on
participation and indicators of response quality in a web survey.
Public Opin Q 2009;73:349–60.

Table 1 The do’s (the good) and don’ts (the pitfalls) of
questionnaire design

Questionnaires?

The good ideas The pitfalls

Well-articulated research topic Lack of consideration of the field of
scholarship when starting.

Comprehensive literature
review

Not searching broadly enough. You may
need to research journals or the grey
literature outside your usual reading.

Considered choice of survey
format

Not taking advantage of validated
questionnaires already available. Not
considering what is acceptable and
feasible.

Clear visual design with
signposting

Small font. Poorly organised. Difficult to
navigate. Too long.

Questions with a single point Complex questions, ambiguous, more
than one point, unclear wording.

Field test or pilot your
instrument appropriately

Missing out relevant stakeholder groups
in the review. Inappropriate opportunistic
sampling—it is too easy to think your
colleagues will do!

For sampling, refer back to
your research question,
randomise or purposively
sample

Insufficient consideration.
Sampling can disenfranchise certain
persons or groups of people.

Choose when, where and how
to deliver the questionnaire for
maximum uptake

Not answering the research question
through poor sampling or poor returns.

Enhance quantitative questions
with free text boxes

Can take many hours of analysis. Know
your limits regarding expertise and time.

Research in practice

212 Thwaites Bee D, Murdoch-Eaton D. Arch Dis Child Educ Pract Ed 2016;101:210–212. doi:10.1136/archdischild-2015-309450

http://dx.doi.org/10.1111/j.1365-2923.1988.tb00764.x

http://dx.doi.org/10.3109/0142159X.2014.889814

http://dx.doi.org/10.3109/0142159X.2014.915297

http://dx.doi.org/10.3109/0142159X.2014.915297

http://www.rds-yh.nihr.ac.uk/wp-content/uploads/2013/05/12_Surveys_and_Questionnaires_Revision_2009.pdf

http://www.rds-yh.nihr.ac.uk/wp-content/uploads/2013/05/12_Surveys_and_Questionnaires_Revision_2009.pdf

http://www.rds-yh.nihr.ac.uk/wp-content/uploads/2013/05/12_Surveys_and_Questionnaires_Revision_2009.pdf

http://www.rds-yh.nihr.ac.uk/wp-content/uploads/2013/05/12_Surveys_and_Questionnaires_Revision_2009.pdf

http://www.rds-yh.nihr.ac.uk/wp-content/uploads/2013/05/12_Surveys_and_Questionnaires_Revision_2009.pdf

http://www.rds-yh.nihr.ac.uk/wp-content/uploads/2013/05/12_Surveys_and_Questionnaires_Revision_2009.pdf

http://dx.doi.org/10.1177/0163278707300632

http://dx.doi.org/10.1093/poq/nfp031

edpract-101-210_16264.pdf
Questionnaire design: the good, the bad and the pitfalls
Preparation
Sampling
Response formats and scales

Evaluation
Delivery
Analysis and interpretation
Summary
References