Week 13 Discussion – Exploratory Data Analysis

Strategies and Tactics of Behavioral Research

Third Edition

2

Strategies and Tactics of Behavioral Research

Third Edition

James M. Johnston
Auburn University

Henry S. Pennypacker
University of Florida

3

Published in 2009
by Routledge
270 Madison Avenue
New York, NY 10016
www.psypress.com

Published in Great Britain
by Routledge
27 Church Road
Hove, East Sussex BN3 2FA

Copyright © 2009 by Routledge

Routledge is an imprint of the Taylor & Francis Group, an informa business

This edition published in the Taylor & Francis e-Library, 2010.

To purchase your own copy of this or any of Taylor & Francis or Routledge’s collection of thousands of eBooks
please go to www.eBookstore.tandf.co.uk.

Cover design by Design Deluxe

All rights reserved. No part of this book may be reprinted or reproduced or utilized in any form or by any
electronic, mechanical, or other means, now known or hereafter invented, including photocopying and
recording, or in any information storage or retrieval system, without permission in writing from the publishers.

Library of Congress Cataloging-in-Publication Data
Johnston, James M.

Strategies and tactics of behavioral research / James M. Johnston and
Henry S. Pennypacker, Jr. – 3rd ed.

p. cm.
Includes bibliographical references and index.
1. —Research—Methodology. I. Pennypacker, H. S. (Henry S.) II. Title.
BF76.5.J63 2008
150.72—dc22 2008019278

ISBN 0-203-83790-8 Master e-book ISBN

ISBN: 978–0–8058–5882–2 (hbk)

4

http://www.psypress.com

http://www.eBookstore.tandf.co.uk

To
Ogden Lindsley
and
Murray Sidman

Giants of our field upon whose shoulders we proudly stand

5

Contents

LIST OF BOXES

PREFACE

PART ONE THE NATURAL SCIENCE OF BEHAVIOR

1 SCIENCE AND SCIENTIFIC BEHAVIOR

Introduction

Scientists as Behaving Organisms

Science as the Behavior of Scientists

Control by the Subject Matter

Scientific Method

The Products of Science

Research Methods and Service Delivery

2 BEHAVIOR AS A SCIENTIFIC SUBJECT MATTER

The Evolution of Conceptions of Behavior

Toward a Scientifically Useful Definition of Behavior

A Working Definition of Behavior

Some Implications

3 ASKING EXPERIMENTAL QUESTIONS

The Nature of Experimental Questions

The Functions of Experimental Questions

PART TWO MEASUREMENT

4 SELECTING AND DEFINING RESPONSE CLASSES

Strategies of Selecting and Defining Response Classes

Tactics of Selecting and Defining Response Classes

5 DIMENSIONAL QUANTITIES AND UNITS OF MEASUREMENT

Introduction

Properties, Dimensional Quantities, and Units

Tactical Issues

6 OBSERVING AND RECORDING

6

kindle:embed:0006?mime=image/jpg

Strategic Issues

Tactics of Observing and Recording

7 ASSESSING MEASUREMENT

Strategic Issues

Tactical Options

PART THREE DESIGN

8 BEHAVIORAL VARIABILITY

Strategic Issues

Sources of Behavioral Variability

9 STEADY STATES AND TRANSITIONS

The Steady-State Strategy

Steady States

Transitions

10 STRATEGIC ISSUES IN EXPERIMENTAL DESIGN

Experimental Design and Reasoning

Strategic Issues

Notation of Experimental Designs

11 CREATING EXPERIMENTAL DESIGNS

Introduction

Single Baseline Designs

Multiple Baseline Designs

Turning Designs into Experiments

PART FOUR INTERPRETATION

12 ANALYZING BEHAVIORAL DATA

Data Analysis Strategies

Graphical Analytical Tactics

Statistical Analytical Tactics

13 INTERPRETING EXPERIMENTS

Interpretive Behavior

Sources of Control

Generality

Evaluating Interpretations

GLOSSARY

7

REFERENCES

AUTHOR INDEX

SUBJECT INDEX

8

9

kindle:embed:0006?mime=image/jpg

List of Boxes

BOX 1.1 Are Scientists Different?
BOX 1.2 Rule-Governed versus Contingency-Shaped Behavior
BOX 2.1 Inner “Causes”
BOX 2.2 The Dead Man’s Test
BOX 2.3 Is Thinking a Behavior?
BOX 2.4 Traits and Colloquial Language
BOX 2.5 Theory, Concepts, and Observability
BOX 2.6 Parsimony
BOX 2.7 Pure versus Quasi-Behavioral Research
BOX 3.1 Research Styles
BOX 3.2 Thematic versus Independent Research Styles
BOX 3.3 Advocacy Research
BOX 3.4 Experimental Questions versus Hypotheses
BOX 3.5 The Null Hypothesis Game
BOX 3.6 The Hypothetico-Deductive Method
BOX 3.7 Ethical Considerations in Behavioral Research
BOX 3.8 Serendipity
BOX 4.1 Units of Analysis versus Units of Measurement
BOX 4.2 Behavior, Response Classes, and Responses
BOX 4.3 Another Kind of Response Class?
BOX 4.4 Parent: “What Did You Do Today?” Child: “Nothing”
BOX 4.5 Operational Definitions and Behavioral Measurement
BOX 5.1 Frequency versus Rate
BOX 5.2 A Tale of Two Frequencies
BOX 5.3 How Many Dimensional Quantities Are There?
BOX 5.4 Is Probability a Dimensional Quantity?
BOX 6.1 How Do You Measure Slouching?
BOX 6.2 What About Rating Scales?
BOX 6.3 The Problem of Measurement Reactivity
BOX 7.1 Reliability in the Social Sciences
BOX 7.2 The Relationship Between Accuracy and Reliability
BOX 7.3 Validity in the Social Sciences
BOX 8.1 Free Will versus Determinism
BOX 8.2 What is Inside the Organism?
BOX 9.1 Measuring One Participant Many Times versus Many Participants Once
BOX 9.2 One Data Point at a Time
BOX 9.3 How Long Should Each Phase Last?
BOX 10.1 Levels of Empirical Elegance
BOX 10.2 Why Scientists Do Not Talk About Causes
BOX 10.3 Why Likes Lots of Participants
BOX 11.1 Risks of Between-Subject Comparisons
BOX 11.2 Experimentum Crucis
BOX 11.3 Do Experimental Designs Have to be Perfect?
BOX 12.1 How Science Deals with Subjectivity
BOX 12.2 Do We Need Data Analysis Police?

10

BOX 12.3 The Cumulative Recorder
BOX 12.4 Does a Graphic Approach to Data Analysis Need Defending?
BOX 13.1 When You Cannot Get There From Here
BOX 13.2 Internal and External Validity
BOX 13.3 Inferential as Interpretation
BOX 13.4 Do Attitudes Toward Interpretation Vary Across Disciplines?
BOX 13.5 Are We Preoccupied with Generality Across Individuals?

11

Preface

Our decision to write a third edition of Strategies and Tactics of Behavioral Research arose from our
experiences, as well as those of many colleagues, in helping students to understand this material. We
discovered many ways of improving our discussion of this approach to studying individual behavior, but we
also saw that the audience for our text was changing. We had written the second edition primarily for doctoral
students in psychology, education, and other academic fields specializing in the experimental study of
behavior. However, we also observed considerable growth in Master’s programs, especially those preparing
practitioners working toward newly established credentials in the field of Applied Behavior Analysis.

We have written this third edition, no longer accompanied by a readings volume, to meet these changing
needs. Although the core content and chapter divisions of the second edition remain relatively untouched, we
have discarded many of the secondary issues and digressions that encumbered discussions in the previous
edition. Instead, we have focused on describing and explaining the primary material in a straightforward and
relatively simple narrative. We have composed both sentences and text_indentgraphs with unwavering
attention to the needs of student readers.

Because many students learning this material may be planning careers as practitioners rather than as
researchers, this third edition considers the relevance of methodological procedures and decisions for the
delivery of professional services. This is not a stretch, of course. Many methodological requirements of
professional practice originated in behavioral research methods, and the fact that the Behavior Analyst
Certification Boardfi mandates coursework in this area clarifies the need to address the role of research
methods in service scenarios.

Aside from substantive and literary revisions, we have also added a number of features that will make the
volume more effective as a textbook. New terms are now identified in bold face type and are formally defined
in indented tinted boxes, as well as in the glossary at the end of the book. There are now many tables that
summarize the main points of a discussion, and they are joined by considerably more figures, including figures
adapted from journal articles. Chapter end matter now includes not only study guides, but a chapter summary,
suggested readings, discussion topics, and exercises. This material is also available on an instructor’s website,
www.psypress.com/behavioral-research, which further includes lecture outlines and test items.

In sum, although the chapter topics are unchanged from the second edition, this third edition otherwise
provides a very different experience for student and instructor. Substantial improvements in clarity of
exposition and the addition of new learning aids offer a more appealing learning opportunity, and instructors
will find it easier to take advantage of students’ interests. Incorporating the methodological interests of
practitioners into each chapter extends this appeal to a growing professional discipline, a field partly defined
by its respect for the highest standards of scientific practice.

We would like to thank our many students and colleagues who have offered valuable feedback along the
way. We would especially like to thank Ryan Zayac at Central Washington University, who prepared many of
the supplementary materials, and Wayne Fuqua at Western Michigan University, who served as a reviewer.

Finally, as with the second edition, the first author has assumed primary responsibility for this edition,
although with the active intellectual support and guidance of the second author. Our contributions to the
foundations on which the third edition is based remain thoroughly intertwined.

—James M. Johnston

—Henry S. Pennypacker

12

http://www.psypress.com/behavioral-research

13

Those who fall in love with practice without science are like a sailor who enters a ship without a helm or a compass, and who never can be
certain whither he is going.

—Leonardo da Vinci

14

Part One
The Natural Science of Behavior

15

Chapter One
Science and Scientific Behavior

INTRODUCTION

SCIENTISTS AS BEHAVING ORGANISMS

Are Scientists Different?

The Three-Term Contingency

SCIENCE AS THE BEHAVIOR OF SCIENTISTS

Scientific Behavior

Examples of Methodological Choices

CONTROL BY THE SUBJECT MATTER

SCIENTIFIC METHOD

THE PRODUCTS OF SCIENCE

RESEARCH METHODS AND SERVICE DELIVERY

Research versus Practice

Role of Research Methods in Practice

The chief problem of science is the scientist

—D. L. Watson

16

Introduction

The scientific achievements of the 20th and now the 21st century have changed our lives in profound ways,
and many people have come to revere science as an almost magical endeavor. We have grown confident that,
given enough time and money, science can solve most of life’s problems, and we may be right. Those who
devote their lives to doing research are held in high regard, and scientific careers are now rewarding not just
professionally but financially.

And yet, most people do not understand how science really works. The average citizen does not have contact
with the daily activities of scientists, so it is not surprising that it is hard to appreciate how scientific pursuits
are different from everyday interests. A newspaper article about a scientific discovery inevitably stops short of
explaining exactly how it was accomplished or describing the years of research that made the breakthrough
possible.

Even researchers are likely to “miss the forest for the trees” as they focus on their own areas of interest. Most
scientists are trained in the research literature and methods of their own specialties. They usually do not
appreciate the underlying features of experimental methods common to all disciplines that make science a
special way of learning about the world. There are some writers who specialize in studying science as an
industry or enterprise, and a few others focus on science from a philosophical point of view. However, the
critical essence of science—the features that are fundamental to its effectiveness—often eludes these writers too.

17

Scientists as Behaving Organisms

Are Scientists Different?

Understanding the essential features of a scientific search for nature’s secrets requires looking at the behavior
of scientists is a particular way. This perspective is based on appreciating the fundamental processes
underlying how human behavior actually works, which is itself the product of a field of scientific study. This
point of view is very different from how we are taught by the culture to view human behavior. For instance,
although we learn to talk about what is going on “in the scientist’s head,” this only distracts us from noticing
more important relationships between scientists’ behavior and their daily work environments.

The key to understanding how science works lies in acknowledging that scientists are behaving organisms.
As such, there is no evidence that scientists are generally different from other people. In other words, they are
not any smarter or more logical than others who earn advanced degrees (Mahoney, 1976).

It is also important to recognize that the behavior of scientists, just like the behavior of all human and
nonhuman animals, is as much a part of nature as any other scientific subject matter and can be approached
with the same experimental tools. In fact, the scientific study of behavior over the past 100 years or so has
revealed many now well-established laws about the variables that determine an organism’s behavior. This
research has shown that, in addition to whatever genetic endowment each individual is born with, the major
influence on behavior is each person’s moment by moment experiences as he or she goes through life.

Box 1.1

Are Scientists Different?

In a somewhat humorously disrespectful book, titled Scientist as Subject: The Psychological Imperative
(1976), Michael Mahoney delights in puncturing many illusions about scientists. For instance, he argues
that scientists are not more intelligent than others, often illogical in their work, often selective and biased
in their treatment of data, passionate in their prejudices, frequently dogmatic in their opinions,
sometimes selfish and ambitious in pursuing personal recognition and defending territory, often secretive
about their findings, and fond of spinning “truths” in hypotheses and theories before the data warrant.
His general point is that scientists are not special, but just like the rest of us.

This list of shortcomings should suggest that, however well science usually works, it can go awry.
Although it is relatively uncommon, scientists are sometimes dishonest with themselves (when they
interpret data in a way they know is incorrect) or with their peers (when they publish findings they
know are false). Fortunately, science has some effective self-corrective mechanisms. In brief, scientific
research includes a public component that keeps innocent bias and blatant dishonesty at a minimum.
Scientists must publish complete reports of their methods, data, and analytical procedures before other
scientists will pay any attention to their findings. Some of their colleagues who are interested in the same
topic will repeat the experiments, which will either confirm the original conclusions or cast doubt on
them and lead to still further experimental efforts to see what the truth really is.

Scientific ethics is an important part of graduate training. If a researcher is found to have broken the
cardinal rule of honesty, there are a variety of sanctions that may be applied. These sanctions include
being prevented from being considered for federal grants, being fired, and even being prosecuted under
civil or criminal statutes.

The Three-Term Contingency

18

The interactions between each action or response and its environmental context involve biologically mediated
processes called conditioning or learning. The laws of conditioning describe exactly how the relationship
between an individual’s responses and the environmental events surrounding them affects his or her behavior
in the future. It may be easiest to understand how learning works in terms of what is called the three-term
contingency. In this context, a contingency refers to relationships between instances of behavior (responses)
and their environmental antecedents and consequences. Figure 1.1 identifies the three terms that define the
basic contingencies underlying all behavior. Environmental events that immediately precede responses are
called antecedent events or stimuli, and those that follow responses are called consequent events. (These
terms are often shortened to “antecedents”

Fig. 1.1. Schematic representation of the three-term contingency.

and “consequences.”) The contingencies involving these antecedent events, responses, and consequent events
describe different relationships between a particular behavior or action and those features of the environment
that precede or follow it. These relationships are termed respondents, operants, and discriminated operants.

Conditioning. The process of changing a behavior that involves interactions between responses and
environmental events whose effects depend on the processes of reinforcement and punishment.

Learning. The relatively enduring changes in behavior that result from conditioning processes.

Contingency. A relationship between a class of responses and a class (or classes) of stimuli. Implies
nothing about the nature of the relationship or its effects.

Three-term contingency. A set of functional relationships among distinct classes of antecedent stimuli,
responses, and consequent stimuli that together constitute the model of how behavior is influenced by the
environment.

Antecedent event. An environmental event that occurs immediately before a response. Used generically
when it is not certain what function the event serves.

Consequent event. An environmental event that occurs immediately after a response. Used generically
when it is not certain what function the event serves.

Respondent. A class of responses elicited by a particular unconditioned or conditioned antecedent
stimulus.

Operant. A class of responses defined by a functional relation with a class of consequent events that
immediately follow those responses.

Discriminated operant. A class of responses that are functionally related to classes of both antecedent
and consequent stimuli.

The three-term contingency is a useful way of summarizing how behavior works because the biology of
organisms, together with their unique life experiences, makes their behavior especially sensitive to certain
kinds of environmental events. For example, humans are from birth especially sensitive to sweet-tasting
substances, and their experiences with particular foods (e.g., cookies) can make such stimuli especially
important. When a child’s behavior such as standing on a chair to reach the kitchen counter where the cookies

19

are kept leads to the consequence of eating the cookie, that behavior is more likely to occur again in the future.
This instance of the three-term contingency is called positive reinforcement. If the same behavior resulted in
a different kind of consequence such as falling off the chair or being scolded by a parent, the behavior might be
less likely to occur again. This instance of the three-term contingency is called positive punishment. Of
course, there are other kinds of contingencies as well.

Positive reinforcement. A class of procedures involving the occurrence of a stimulus immediately
following responding that results in an increase in some aspect of the response class over baseline levels.

Positive punishment. A class of procedures involving the occurrence of a stimulus immediately
following responding that results in a decrease in some aspect of the response class over baseline levels.

The relationship between instances of behavior and the antecedent side of the three-term contingency also
influences behavior, although somewhat differently. If a certain behavior occurs when a particular
environmental event is present and the behavior then produces a reinforcing consequence, that antecedent
event comes to serve a sort of signaling function. The behavior (a discriminated operant) is then more likely to
occur when similar antecedent events (called discriminative stimuli) are present than when they are not
present. For instance, if you drive up to a store and see an “Open” sign on the door, you will usually get out of
the car and go in because in the past such behavior has resulted in reinforcing consequences. Not surprisingly,
you would be less likely to get out of the car and try to go in if such behavior has been followed by a punishing
consequence (the door is locked) in the presence of a different antecedent (a “Closed” sign).

Discriminative stimuli. Stimuli that have acquired the function of setting the occasion for a behavior to
occur. A behavior is more likely to occur in the presence of a discriminative stimulus than in its absence.
Abbreviated SD.

Other antecedent stimuli have functions that depend less on the consequences of responding and more on an
organism’s biology. For example, when a stimulus such as a puff of air contacts our eye, we blink, and it is
difficult to avoid doing so. Even innocuous events paired with a puff of air elicit the same kind of blinking
response. This behavior is an example of a respondent. (See Catania, 2007, for a more detailed treatment of the
three-term contingency and the resulting classes of behavior.)

The repertoire of each of us at any point in our lives is largely the result of our history of contingencies like
these. What we do or do not do, our skills, our emotions, and the unique features of our individuality are
largely a function of the laws of conditioning. As with other laws of nature, these relationships are at work
even if we are unaware of what is going on, and no one is exempt from them, even for a moment.

Over the years, scientific study of the relationship between behavior and the environment represented by the
three-term contingency has been very fruitful. Researchers have not only learned a great deal about the basic
components of conditioning, but they have also learned how to apply these fundamental principles to human
behavior, especially under everyday circumstances. As a result, a still developing but powerful technology for
changing behavior has emerged. This technology, called applied behavior analysis, is now used in diverse
areas, including mental retardation, autism, brain injury, education, business, medicine, and sports (Austin &
Carr, 2000).

Applied behavior analysis. A phrase that may refer to (a) the field of research that focuses on applying
the findings of a basic science (the Experimental Analysis of Behavior) concerning fundamental processes
of conditioning to address the need for changing behavior under everyday circumstances (b) the behavior
change technology developed by applied researchers, or (c) the field encompassed by both applied
research and delivery of the resulting technology.

20

21

Science as the Behavior of Scientists

Scientific Behavior

The basic principles of conditioning or learning are relevant in a book on behavioral research methods because
they are at the heart of how science works. The primary activities of scientists are figuring out what
experiments to do, planning and conducting them, and interpreting and communicating their results. When
scientists are doing these kinds of things, they are behaving. In fact, science is really no more than the behavior
of scientists.

Although this may sound simple, the details can become complicated. We certainly do not know everything
we might like to about the behavior of scientists. However, if we examine their activities in light of the three-
term contingency, it can help to explain much of what goes on every day in the scientific workplace. This
approach is also useful because it focuses on what researchers actually do and the environmental
circumstances under which they do it. In other words, it helps us talk about scientific behavior in the same way
we approach any behavior as a subject matter. We can try to identify the antecedent events that may prompt
researchers to choose one course of action over another and the consequences that may reinforce some
practices but not others.

This is a good perspective for a text on research methods because it helps to highlight the many choices that
researchers face in conducting an experiment. Each choice about what to do and how to do it involves some
antecedent circumstances and certain possible consequences. The alternative selected may lead to a reinforcing
outcome or punishing outcome, which will then make similar decisions more likely or less likely in the future.
Examining these scientific contingencies can help to show why some research practices are more effective than
others under certain conditions. If we step back and examine the choices that researchers most often make in a
certain discipline, we can identify the field’s established methodological practices. The reason why those
practices are preferred stems from their effectiveness for the individual researchers who choose them.

Examples of Methodological Choices

One of the first things a behavioral researcher must do is decide what behavior should be measured and how to
measure it. For instance, as chapter 4 will show, there are different ways of defining a particular behavior.
Some definitions are likely to produce more variability in the resulting data than others, and some may
generate less variability. For instance, the behavior of cleaning a kitchen may be defined in terms of specific
actions, such as putting dishes in the dishwasher, scrubbing the counters, and so forth. Alternatively, this
behavior may be defined in terms of a certain outcome, such as clean dishes and counter, regardless of the
actions involved (perhaps the dishes were washed by hand). As we will see, the degree of variability in the data
is an important consequence of a researcher’s methodological choices.

Another set of decisions will be required when choosing the particular features of a behavior that may need
to be measured. For instance, measuring how often the behavior occurs will provide a different picture than
measuring how long it lasts when it occurs. One of these pictures may be more useful than the other in
revealing the effects of an intervention and answering the experimental question.

There are also a series of choices that must be made in designing procedures for observing and recording the
target behavior. For example, the researcher must decide how often observation will take place and how long
each session will last. Not surprisingly, the choices here can again have a big impact on what the investigator
learns about the behavior. For example, measuring a participant’s behavior only once before and once after an
experimental treatment is certainly easier than measuring it repeatedly throughout both control and
experimental conditions. However, the two kinds of data sets provide very different antecedents for the
researcher’s interpretive reactions.

Designing an experiment involves selecting experimental and control conditions and then arranging them in

22

ways that show differences in responding between the two conditions. This aspect of research methods also
faces the investigator with many choices. A key decision concerns how to compare data from control and
experimental conditions.

One approach, which has a long tradition in psychology, is to divide a relatively large number of
participants into two groups. The groups are treated the same except that one experiences the experimental
condition and the other does not. The differences in the aggregate performance of each group are then
analyzed using the rules of inferential statistics.

An alternative approach, which also has a long history, uses fewer participants but separately exposes each
individual to a series of sessions under the control condition and a series of sessions under the experimental
condition. Comparisons are then usually made using graphical techniques examining the performance of each
participant.

These two approaches, which have many variations, are fundamentally different. As later chapters will
show, they face the researcher with choices that lead to importantly different consequences for the nature of
the data and for the kinds of interpretations that can be made.

Finally, once a study is completed and the data are available for analysis, figuring out what the data reveal
about the experimental question provides still more choices. For instance: (a) Are all of the data fully
appropriate for analysis, or should some data points be omitted because they represent the effects of extraneous
factors such as illness? (b) What data analysis techniques are most appropriate? (c) If displaying the data
graphically is planned, what kind of graphs should be used? (d) What type of scale should be used for the
vertical axis? (e) What factors in the experimental procedures should be considered in interpreting the data? (f)
Were there procedural problems that might make the data misleading?

23

Control by the Subject Matter

This chapter has introduced three themes that continue throughout this volume:

1. The essence of science lies in the behavior of individual researchers.
2. This behavior results from the same kinds of environmental contingencies that determine all other

behavior.
3. …