Internal and External Validity

Internal validity is the extent of the ability to attribute the observed effect  to the experimental variable, and not to other factors. External validity is the extent to which the effect can be generalized.  Poor experimental design can affect both types of validities.
Experimental Design:  for The notations used in representing a design are as follows.

  • “X” represents exposure to treatment
  • “O” represents Observation
  • Time is represented by horizontal arrangement  of  Xs and Os. e.g.  X   O1  O2  represents a test unit is exposed to treatment and then observed at two different points in time
  • Different time units are arranged vertically e.g.

                                     X1    O1

                                     X2   O2
   indicates two different groups of test units (one per row) and each group exposed to different experimental treatment (at the same time) and were subsequently observed (again at the same time)
Extraneous Variables:  These variables threaten the internal validity and are categorized into following categories.
  • History(H): These are specific events external to the experiment, but occurring at the same time and may affect the criterion variable. Sometimes this effect may be subtle or hidden. e.g. In finding the effect of teaching in the class on the performance of student, the effect of any external coaching will cause a history effect.
  • Seasonality (Se): Seasonal cycles my affect the criterion in either way. e.g. Increase in sales of fans due toe technical enhancements can be affected by seasonality like in summers the sales will anyways increase.
  • Maturation(M): These are changes in test units which are not due to the experimental variable, but result from passage of time. e.e people get older, bored, tired, hungry or distracted.
  • Testing: This is related to the fact that experiment itself may affect the responses. These are of two types
    • Main testing effect (MT): This comes as a result of respondent’s desire to be consistent
    • Interactive testing effect (IT):  In this case a prior observation effects the perceptions of the experimental variable.

The results of two testing effects are different. Main testing effect comes when the process of measurement O1 affects O1 or later observation O2 .The interactive testing effect comes when the process of measurement O1 results in a change in the reaction to the experimental stimulus.

  •  Instrumentation(I): This refers to any or all changes in measuring instruments that might  account for differences.
  • Statistical Regression (SR): This refers to the tendency of “extreme” cases to move closer to the avaerage during the course of the experiment.
  •  Selection (S):  This arises from the way in which tests units are selected and assigned in an experiment. The equivalence of various groups in an experiment can be established in two ways: Matching  and Randomization. Matching isn’t perfect. It’s difficult to match test units on any but a few characteristics and that too on those which are important determinants of the response. Randomization doesn’t have these problems and is generally a preferred approach.
  • Mortality (Mo): This refers to loss of test units during the duration of the experiment.
Specific Designs: 
(1) Pre Experimental Designs: The researcher has little control over the experimental setup.
    1. The one shot case study: The design is denoted as   X  O i.e. a single group is exposed to experimental variable and the responses are observed. The group is arbitrarily selected and no random assignment. This type of design does not well establish the validity of the causal relationship as it is marred by little control on extraneous variables.
    2. One group pre-test post-test: The design is denoted as O1 X  O2. It adds pretest to above design. The sampling is still convenience and the design fails to control for history, maturation, both the testing effects, statistical regression and mortality etc.
    3. Static group comparison:  The design is denoted as

                                         EG: X   O1

                                         CG:      O2
                    Various types of threats to validity in this case are selection, maturation, testing effects etc 
(2) True Experimental Designs:

  • Before after with control group design:  The design is denoted as

                                                    EG: (R)  O1   X   O2

                                                    CG: (R)  O3        O4

Two groups are used with random assignment (R signifies that). The difference  O4 – O3 design  reflects effects of extraneous variables. History effect is nullified since we have used two groups. If E represents the effect of experimental variable and and U is due to uncontrolled sources . we see that
                                                    O2 – O1 = E + U + IT
                                                    O4 – O3 = U
                                                    ———————————–
                                                    (O2 – O1) – (O4 -O3)  = E + IT
So interactive testing effect is still a problem  for this type of design.

  •  Four group six study design: This Solomon’s design is denoted as follows

                                        EG#1: (R)  O1   X  O2

                                        CG#1: (R)  O3        O4
                                        EG#2: (R)          X  O5

                                        CG#2: (R)               O6

                        The sample here is divided into four groups and total six observations were recorded. The analysis is shown as follows. Since there is random assignment in all the groups so we assume here that except for sampling variations, the four groups are equal a priori and the best estimate of the “before measurement” for EG#2, and CG#2, is average of two before measurement observations.

                                        O2 – O1 = E + U + IT
                                        O4 – O3 = U

                                        O5 – 1/2(O1 + O3) = E + U

                                        O6 – 1/2(O1 + O3) = U

                                        —————————————————
                                        [O5 – 1/2(O1 + O3)] – [O6 – 1/2(O1 + O3)]  = E

                                        [O2 – O1]- [O5 – 1/2(O1 + O3)]  = IT
This way the experimental effect and the interactive testing effect can be isolated.

  • After only with control group: After all why are the two groups needed in the previous design. Anyways this is getting canceled out. We can straight away use the later two groups and calculate the experimental effect.

                                        EG#2: (R)           X   O5
                                        CG#2: (R)                 O6

                            The difference  O5 – O6 straight away gives the experimental effects. But this design has two problems. First, the selection bias. The prior equality of the groups is assumed due to sheer random assignment and no pre measure is taken. Second, this design is highly sensitive to experimental mortality.

 (3) Quasi Experimental Designs: In these type of designs the stimuli can’t be scheduled appropriately or random assignment in to groups does not just become possible.

  1. Time Series experiment: This design is denoted as follows.
                                                             O1  O2   O3   O4   X   O5  O6   O7   O8
                    A group of test units is observed over time, and then treatment is given and again several observations are made. The maturation, Instrumentation, Statistical Regression can be avoided. Selection is reduced by random assignment. Mortality can be controlled. The main problems here are the history effect and interactive testing effect. The data type used for this design is the panel data and involves a totally different type of analysis.