This brief manual is prepared to enable its user to apply the methods of OPERATION-ANALYSIS for the summary and evaluation of reports of systematic observations and of experimental research in both psychology and behavior science. Users can summarize their analyses by entering pertinent information on previously prepared forms, from which pertinent information can be readily abstracted.
General Purposes
The analysis has been developed to further several purposes:
When the analysis of a paper is complete, the summary should enable precise replication of the experiment. The analyst will learn with experience that many papers omit information on the procedures carried out, sometimes because the variable is not considered pertinent, sometimes for brevity, and sometimes for inscrutable reasons. Missing information may be critical for the results.
Analysis may show that an effect presumed to be due to one variable is in fact attributable to another. More importantly, a single operation may prove to have been used in the "operational definition" of a variety of concepts.
A GUIDE TO AID IN THE OPERATION-ANALYSIS
OF EXPERIMENTS
This guide is designed to be used in filling out the "Operation-Analysis Sheets." These instruments are designed to give authors and readers a complete and concise statement of the experimental procedures described as carried out.
Information on each should be stated completely enough to permit evaluation and to make replication possible. The analysis should permit evaluation of a paper from the following points of view:
The analysis sheet leaves its users to their own devices in evaluating (a) the relationship of the experiment and its results to the problem as it was stated by the experimenter, (b) the appropriateness of the experiment in providing a solution to a problem or as test of a theory, and (c) the adequacy of the presentation of previous findings and discussion.
The following are general rules to be followed throughout:
The entries for the first three are "n.r." -- not reported. The analyst should leave to the reader the decision whether the information is needed or appropriate.
For the fourth entry: "not clear".
Each sheet is designed for analysis of a single experiment. Papers reporting on multi-experiments will require as many sheets as there are experiments (cross referencing where applicable).
Special Sheets
One size fits all:
Some of the categoric operation statements on which are required are not appropriate in particular fields of research; only a minority of research reports need descriptions of surgical interventions.
Note: The numbering system in this guide corresponds with the analysis sheet. The completed analysis should leave no section blank.
1. REFERENCES
1.1 Title
What is the full APA style citation? (Authors should choose a title that could serve as an abbreviated abstract of the paper). What are the key words under which this paper would be found indexed in the subject index of abstracts?
1.2 Traditional references
What are the traditional theoretical reference headings relating the problems this experiment investigates to similar attempts in the literature? How should it be indexed?
1.3 Who designed the experiment
Who carried out data analysis and reporting?
Who ran the subjects? (author? research assistant? graduate student? undergraduate? staff members?) Cases ranging from carelessness, incompetence, through assistants eager to get data that will please the boss and on to downright fraud have filled the literature with "unreproducible results." Hence, responsibility must be identified.... can you trust your assistants?
1.4 Exploratory research:
2. SUBJECT SELECTION
2.1 Age and Cohort
Some of the most egregious errors in psychology have arisen from the failure to consider cohort, and on the identification of people by their cohort occurs more frequently in everyday divisions (e.g., "slacker," "babyboom," "yuppy.")
What was the age of the subject? (report as accurately as possible; for groups, mean and variablitiy)
Year of Birth (for groups, means and variability)
2.2 Categorization
Species-Genetic identification of nonhuman subjects, including breed as appropriate (e.g., Cebus capucinus); Social or other available categorization of humans subjects (e.g., College Sophomores taking Introductory Psychology)
2.3 Number
How many subjects started in the experiment? How many completed it? Describe any procedures related to incomplete data. (In research related to medical status or surgical procedures report on recovery or death.)
2.4 Recruitment of Subjects
How, when, and where were the subjects obtained? (e.g., Patients on ward, clients, volunteers from class? Course requirement? Extra credit? Paid? Purchased from dealer? Bred in dept. colony?)
2.5 Groups
How many groups were there? How many subjects per group? Males and Females in each? What was the number of subjects per group on whom data are reported?
2.6 Sex
How many member of each sex per group? Treatment? Did these totals change during the experiment?
2.7 Behavioral Testing
What tests or behavioral criteria were used to select or to assign subjects to groups? What were the criterion scores or the reliability/validity data on such tests?
2.8 Group-Assignment
How were subjects assigned to groups? Random? Matched?
2.9 Experimental history:
How many subjects, received what treatment, have previously served as subjects in similar (or other) research?
3. OPERATIONS BEFORE THE EXPERIMENT PROPER BEGINS
3.1 Bio-cyclical Events
What were the chemical, anatomical, imposed or natural environmental (e.g., diurnal, oestral, seasonal) cycles under which the subjects existed before and during the experiment? At what point in such cycles were data collected?
[N.B. Season of the year]
3.2 Feeding
For non-human subjects:
What was the dietary and feeding routine of the subjects before the experiment began?
3.3 Ambience
In what milieu did the subjects live before the experiment began?
Humans: home; dormitory; hospital; detention centers, etc.
Other animals: single cages? group cages?
3.4 Subject History
What specific background did the individual subjects bring to the experiment that might affect results? (Include problems or other pertinent features. e.g. institutionalization)
3.5 Specific Experiment-Related History
What prior experience did the subjects have with the specific materials, apparatus, procedures, and conditions of the experiment? Include all experience given to the subjects by the experimenter prior to the commencement of the experiment proper such as warm-ups, "adaptation", practice trials, shaping, etc.
3.6 Pretraining/Instructions
How were the subjects enabled to carry out the specific procedures of the experiment? What said or done to make it likely that the subjects would perform according to the experiments plan or expectations? (shaping, encouragement, deprivation, outcome contingencies). When instructions were stated, clearly indicate their completeness, their accuracy, and their truth or falsity, if they are misleading, deliberately or otherwise.
4. MAINTENANCE UNDER EXPERIMENTAL CONDITIONS
4.1 Time
When exactly was the experiment conducted? (dates, hours)
4.2 Bio-cyclical Events
What changes in cyclical relationships (3.1) were made or occurred once the experiment had begun?
4.3 Feeding and Maintenance
What changes were made in the dietary and feeding routine of the subjects at the beginning of or during the experiment? What changes were made in the living milieu of the subjects at the beginning of the experiment and continued while the experiment was carried out?
4.4 Experimental Space
What were all the physical characteristics of the microclimate and microsituation in which the experiment was carried out? (This relates to the immediate space where the experiment was carried out be it a hospital ward, an office, an experimental cubicle, or a Skinner box).
4.5 Apparatus
What "things" or "gear" were used to conduct the experiment? Who made them? Were they standard, modified, or home-brew? (give the complete details of all pieces of equipment used and how they were with one another).
5. PHYSICAL STATUS, INTERVENTION AND EXAMINATION
5.1 Physical Description
What were the pertinent physical characteristics of the subjects? (Information as to size, weight, eye color, etc.; especially any specific anatomical of physiological characteristics which distinguish the subjects not already recorded in 2.1 or 2.2).
5.2 Nutritional Considerations
What manipulation of nutritional variables occurred? (If any: e.g., dietary composition).
5.3 Experimental Surgery
What were the precise details of all surgical procedures performed? (If any: histological data).
5.4 Biochemical (Hormonal-Pharmacological) Interferences
What usage of biochemical substances occurred? Who was the supplier? What were the dosages, rate, and method of administration? What was the routine of administration? (blind, double-blind, "this will...")
5.5 Examination
What were the precise details of all diagnostics, pathological or histological analysis, X-ray examinations, or other detailed examinations of the subjects?
6. EXPERIMENTER CONTRIBUTIONS
6.1 Pilot Experimentation
What were the details of previously unreported (or unsuccessful) attempts to obtain these experimental findings? What was done to find the appropriate method that would give these clear-cut results?
6.2 Non-Subject Involvement
Who was involved in the experiment besides the subjects? What did they do? (Who ran the subjects, did the statistics, interacted with the subjects and wrote their report?)
6.3 Experimental Design
How does the report describe the overall scheme, plan, or blue-print of the experiment? (Latin squares, split plot, case history). Diagram if possible, a la Campbell and Stanley, or Sidman using only key descriptors (details to be furnished in later sections).
7. IDENTIFICATION OF STIMULI, RESPONSES AND DEPENDENT EVENTS
7.1 Stimuli
Identify the presumed behavior-controlling objects or events empirically controlled or manipulated by fully describing their nature (physical, psychological, physiological) and specification? (Include manipulanda and variable events; e.g., programmed presentation of lights, lowering of a platform, opening of a door, distribution of forms, etc.).
7.2 Responses
What are the identities of all the events in the activity of the subjects presumed to result from the experimental manipulation? (Include all criteria used to identify, report, and measure manifestations of the same). Include all that is measure to be subsequently analyzed, or that could be analyzed, whether or not data on it is later reported in the paper.
7.3 Dependent Events
What are the consequences and means of delivery of the consequences as a result of a subjects activity on a trial or during the experiment? (payoffs, reinforcers, punishers, avoidings, all contingencies).
8. PROCEDURES
8.1 Pretreatment or Baseline Measures
What were the techniques and procedures used to obtain these (pretreatment or baseline) measure? (Identify what they were by groups and subgroups. These measures are the basis for within subject or within group analysis).
8.2 Treatments
What are the orderly relationships (conjuncture and sequential) of stimuli, responses and dependent events as they were presented to the subjects (by individuals, by groups, group by group)? (This is the usual "procedure" on each trial or operant cycle. Parametric values will be reported on in Category 10).
9. MEASUREMENT AND ANALYSIS
9.1 Techniques used to Obtain Data (Probes)
What were the techniques (probes) used to enable the experimenter to obtain data that yielded measures. (questions, stimulus presentations, manipulanda, puzzles, observers, etc.)
9.2 Measurement
What measures, and measured behaviors did the probes of subjects behavior produce? (counts of presses, counts of answers, counts of eyeblinks) What did the data measure?
9.3 Data Selection
Which available measures were made and analyzed? Which measures were transformed? How? Why? Which measures were taken but not analyzed? Why? Which available measures were not taken? Why?
9.4 Data Analysis Techniques
What were the names (and references) of all data analysis techniques used?
9.5 Tabular and Graphic Displays
Which measures or results of analysis were displayed? How were they displayed?
10. SPECIFIC VALUES OF ALL VARIABLES
10.1 Independent Variables
What are the independent variables (and their values) for each group and treatment?
10.2 Experimenter Determined Probabilities of Occurrence
What were the probabilities (percentages) of presentations on activities, trial by trial, cycle by cycle, session by session, for groups and treatments? (With what regularity did events controlled by the experimenter occur sequentially or in just a position?)
10.3 Sequencing Rules
What were the rules the experimenter used to determine which particular events, procedures, or placements occurred when they did? When were they used? (Randomizing, "staircase" ordering, haphazard placement, counter-balancing, etc.)
10.4 Procedure Parameters of Trials or Operant Cycles
What were the number and duration of: trial per block, blocks per session, intertrial intervals, interblock intervals, intersession intervals, cycles per session, and sessions?
11. OUTCOMES
11.1 Intra Subject or Group Treatment
What were the results when subjects or groups were compared the subject's or group's own baseline data or other reference base? (within subjects or groups results)
11.2 Comparative
What were the outcomes of between group or between treatment level comparisons? (between subjects, groups or treatment result) Specify.
11.3 Terminological
Did outcomes permit (allow) new statements (qualitative results as distinct from quantitative) or new terminological usage? what new statements or terminologies did the experimenter make or introduce?
11.4 Side Effects
What was observed to have occurred (even if not measured) that the experimenter did not expect? what was different (and noticed) from pre-experimental states of activity that did not fall within the scope of the experimenter's plans? (These are the serendipitous results of an open-eyed observer on what was happening).
12. PURPOSE
12.1 Introduction
What was the hypothesis (or hypotheses) stated for the experiment? Against what theoretical background was it conceived and designed?
12.2 Discussion
How did the outcomes support (deny) the hypothesis(es)? What new questions were raised?
12.3 Summary
Briefly encapsulate the qualitative findings, how they were obtained, and how they relate to theory.
13. EVALUATIVE COMMENT
13.1 Critique of the Experiment
Was the report complete? What was missing? what was reported or missing that could effect the experimenter's explanations of the results? (Use this guide sheet as your instrument of critique).
13.2 Reviewer's Choice
What are your constructive criticisms? What are your suggestions? What is your overall evaluation?
Footnotes
1 The earliest antecedent to this 1996 Manual is dated 1970. With accruing experience, the manual has been progressively revised to facilitate the precision of analyses.
2 This manual and the analysis sheets have been developed through several years of experience in analyzing reports of experiments published in a wide variety of journals and fields, and using various methods. The analysis sheet permits problems to be identified that often have eluded the writers, editors, or readers of a paper who sometimes fail to note the omission of significant information on details.