3A – Research Design: Experimental and Quasi-Experimental – Captain Linnea Axman

good afternoon I’m captain Linnea excellen and I’m very happy to be presenting research designs and the next slide we’re going to talk about objectives at the end of this presentation you should be able to list at least three research designs that may be useful in clinical research and that are considered experimental and we’ll talk about what experimental means provide one example of a research question best answered using one of the experimental designs and describe the processes from research aims purposes to design an analysis first I’d like to talk to you a little bit about the types of research designs and I recognize that you’ve already had many lectures and you probably feel that you know this but just in review you need to write or think about specific aims appropriate for your research design and those aims may be qualitative quantitative or in many cases mixed the specific aims of a project are statements of what you intend to accomplish with your research and below you’ll see two examples one aim is to compare the long-term effectiveness of the daily low-dose aspirin therapy in prevention of breast cancer in postmenopausal women another specific aim would be in an in another study of course to examine the effect of deployment on soldiers health-related quality of life some more specific aims and again these are statements of which you intend to accomplish with your research we may want to continue the development of an interview tool to guide patient practitioner encounters or in the case of another study describe the lived experience of providing end-of-life care to mortally wounded Marines and in the second case there that would be a specific aim for our qualitative research study specific games will then guide your research questions your research hypothesis your design and eventually your analysis here’s some examples of some research questions and the seeds that follow from the very specific aim the aim of the study is to examine the effect of deployment on soldiers health-related quality of life the question that would proceed from that aim is do a specific set of external stressors predict a higher level of stress and a lower level of health-related quality of life over time and then from that question the following hypothesis the combination of a history of domestic violence tobacco use and deployment to the Southwest Asia theater of operations is predictive of a higher level of stress and a lower level of health-related quality of life over time as you can see these are very very specific and the more specific that you can be the tighter your research project will be next I’d like to talk a little bit about the number of quantitative designs that there are and how they are categorized you probably know that descriptive and correlational studies are observational studies you don’t manipulate the variables you literally observe them the next category is quasi experimental and this involves interventional research which many of you may be involved in experimental research is preceded by something we call pre experimental and you may know this as post-test only study design and I strongly discourage you from using this design and we’ll talk a little bit more about why following experimental designs we have something called secondary analysis of data which is which uses pre-existing databases or previously collected data we do quite a bit of that in healthcare related research following that is meta-analysis and a less rigorous but very useful sister or brother to meta-analysis is synthesis of literature or systematic review and finally we have something called methodological research which is the development of questionnaires or Survey Research just going to review experimental research designs with you I realize you’ve had this in a previous class but I think it’s always good to go back and see where we’ve come from so as you’ll recall experimental research designs include pretest post-test control group designs post-test only with a control group design randomized block designs factorial designs nested designs and crossover designs pretest post-test control group design is the most commonly used experimental design and and I would say most new investigators

like this this type of study design this is a design in which you randomly select the sample an experimental and and then the experimental control group are randomly assigned and then the treatment is under the control of the researcher and in order for it to be a truly experimental design all three of these things need to be there random selection random assignment and control over the experimental group and this is what it looks like pretest post-test control group where the e is the experimental group C is the control group the zero is for observation and that would be where you give a pretest or post-test and the X is the actual intervention here’s an example of a pretest post-test control group design what is the effect of brief therapy on adolescents aged 10 to 13 years with mood disturbance the experimental group gets usual care plus brief therapy the control group gets usual care and the instruments used to measure the effect are the brief profile of mood states or what we call brief PMS and one of the things I’d like to point out here is that for ethical reasons we don’t withhold care in healthcare we always give usual care and then we add the additional intervention post-test only control group design is an improvement on the post-test only design because for obvious reasons you have a control group with which to compare the endpoint or the effect the dependent various design because the dependent variable cannot be measured before the treatment or intervention or when there is concern that subjects may learn from pretest I know everyone in the audience has learned from testing situations and that’s called pretest sensitization I actually look forward to pretest because I learn from them so much here’s what a post-test only control group design looks like and you see that there is an intervention and all that is the only observations are after the fact in both the experimental and the control group so when would you use a post-test only with control group design and I think this is a very timely example when you want to evaluate the response to pain medication in the field of battle after traumatic amputation obviously it’s difficult to know who’s going to have a traumatic amputation so you really can’t give a pretest or a pre evaluation when do we use randomized block designs and what in fact are they they sound kind of scary and complicated the statistics for randomized block design can be somewhat complicated but it is a very nice design to use it’s it uses two group pretest post-test or two-crew post-test pattern with the addition of a blocking variable and that’s used to control for confounding for example if you wanted to do an intervention to relieve pain and the intervention is dependent on the severity of pain you would want to know pre intervention where the individual was with pain to lump everybody together including high low amounts of pain would confound the findings so to prevent this the subject are ranked ordered according to that compounding variable which is level of pain and then randomly assigned at each level either to the experimental group or the control group until all samples are matched by rank pairs this can however be done by using analysis of covariance after the fact I don’t recommend that I think you need to be able to rank order your subjects by level of pain and one example of this would be the zero to ten scale and so you would need to know though at least qualitatively where you’d put people do you put people zero to three in one group four to seven and that may where you be where you want to use your analysis of covariance so do rank order and then also do an analysis covariance or your statistical analysis here’s an example of a randomized block design and I think you can see why there could be some compounding here if you didn’t use this design women were randomly assigned to the experimental or control group and then stratified by the level of breast self-exam performance in the last three months to control for pre intervention BSE differences and to test for interactions between prior performance and the intervention now we’re going to talk a little bit about quasi experimental designs and how they differ from true experimental designs we’ll also touch on Survey Research at the end of this presentation you should be able

to list at least three research designs that may be useful in clinical and community research and that may not be considered experimental verbalize at least two reasons why quasi experimental research may be indicated and discuss the major differences between the types of quasi experimental research designs quantitative studies that do not use experimental research designs and I’ve touched on them before include descriptive observational or time dimensional designs and we call those epidemiological studies correlational pre experimental quasi experimental which will we’ll talk about next survey methodological design secondary analysis of data and meta or made analysis depends on what part of the country it come from how you pronounce that a little bit about observational studies or epidemiological studies observational studies that we use frequently in healthcare and in medicine nursing cohort studies case control studies cross-sectional studies mortality studies and ecological or aggregate study why do you we use observational designs well if we want to obtain measures of disease frequency naturally-occurring or exposure frequency or if we want to measure the effect in our measure of Association so here’s some quasi experimental designs but why do we use them I mean the designs are going to look very similar to experimental designs and so I’m sure if I was sitting out in the audience I’d say well why not just call it an experimental design and they will tell you that you will often read in the literature something that’s described as an experimental design and in fact it is a quasi experimental design you use a quasi experimental design when you have insufficient control in one of the following areas either you have insufficient control of the treatment variable you’re unable to manipulate it you can’t manipulate the setting and this is quite common both in the clinical setting as well as in the community and organizational research or if you’re unable to randomly select or assign so quasi experimental design is done for a variety of purposes to explain relationships to clarify why certain events happen it’s we can use it as one means of showing causal relationships or we can also use it to predict a specific phenomenon or outcome what’s the current thinking about quasi experimental designs well quasi experimental designs provide a logical framework for research you can tailor your designs when using quasi experimental in other words if you need to use mixed methods you can you can use multiple theoretical perspectives you can also demonstrate the role of theory as a basis for assessment and you can incorporate quality control or what we call a process implementation index how much of your therapeutic intervention did the patient or client actually receive and how that affects the outcome and then you can also use complex or what we call realistic statistical models in order to analyze this data you may have heard of structural equation modeling hierarchical modeling multi-level modeling all of these are possible in quasi-experimental designs so here’s some types of quasi experimental designs again you may recognize them as somewhat similar to experimental studies one group pretest post-test which again may be considered pre experimental if the post-test only is done one group pretest post-test is not experimental it’s only if the post-test only is done now an equivalent comparison group design that’s where say for example you compared the effect of an intervention on two wards but the words were slightly different perhaps one is med certain one is medical and one a surgical time series designs you may know that type of study as a repeated-measures study and multiple time-series designs and the only difference there is you add the addition of a comparison group so here’s some more types of quasi experimental designs here’s an example of a one group pretest post-test and you’ll be able to tell from this that often these are done in the community a maternal child health program conducted a smoking behavior assessment of patients moking status and measured variables before and after receiving a health education program which included the measures actually included a health belief score and the addition of a physiologic measure or that measured smoking prevalence which was self-report and saliva thiocyanate I’d like to you to notice that in this study they did use a physiologic measure which actually is the strength of the study as you know there’s always

problems with self-report particularly in smoking studies because the individual knows what you want them to say so I’ve a sign it is good within 72 hours of smoking so if the individual has smoked within the last three days you’ll be able to tell if they are being socially desirable in their respondents rather than perhaps forthcoming and biasing your results this is what a one group pretest post-test looks like again looks very similar to the experimental although you see there is an absence of a comparison group here so here’s an example of a non equivalent comparison group design and I think you’ll see why and we call it non equivalent we don’t call it a control group all sixth-graders in two communities experimental in comparison were indirectly exposed to the Minnesota Heart Health Program through multimedia marketing the experimental community also was exposed to targeted screening and education and this data was collected over a five-year period and here’s what it looked like and as you can see there’s an addition of a comparison group and how the comparison group is identified is with a see as if it were a control group however it’s underscored when do we use time-series designs and why do we want to do that we use repeated measures or time series designs to establish the periodicity or the pattern of an outcome variable being examined to establish the reliability of an instrument or an outcome measure and to apply an intervention or treatment over a specified period of time and then withdraw abruptly we only do this if it’s ethical to do so of course and here’s what a time series design looks like you have your experimental group and you can see you do multiple observations I would like you to note that she do multiple observations before the intervention as well as after the intervention and you would say for example you were doing an intervention where you collected data every month after the intervention in the pre intervention time period you would also collect data every month here’s an example of a time series design the NCI or National Cancer Institute supported cancer communication program was evaluated for its ability to affect cervical cancer screening in rural communities in Alabama the measurement of interest was the number of women identified as new users attending the cancer screening program at Hale County Alabama on a quarterly basis over a three-year period data were collected one year before and multiple time points during and for one year after the intervention the seasonal variation in usage of the clinic was of interest let’s talk a little bit now about what multiple time series designs look like multiple time-series designs build on the simple time series design that we saw before however using the word simple in time series design in the same sentence probably shouldn’t be done because time-series designs are not simple but multiple time series designs improve on the control over major threats to validity such as history particularly appropriate when an organization can frequently observe rates for program participants so if your patient or client is going to be coming back for frequent visits this is really a good design to use outcomes are studied at different times for both the experimental and comparison group and here’s what this looks like and you certainly can have more than one experimental and more than one comparison group you’re not limited however the more groups you have of course the more costly the project some of the statistical models that deal with the variance and measurement error inherent in any design that is not truly experimental our corrected analysis of variance models probit and logit models log linear models factor analysis and structural equation modeling in hierarchical or multi-level modeling one of the best examples that you would use hierarchical multi-level modeling would be students in classrooms in schools in school systems in factor analysis and structural equation modeling are very good when you have multiple variables that don’t have measurements that the deal with issues like anxiety or empowerment that are very very difficult to to measure without some type of proxy measure so what haven’t we covered and when we’ve been talking about quasi experimental designs we haven’t talked about quite a few things we haven’t talked about threats to internal validity or bias

generalizability or external validity mixed and multiple multi method designs in other words those that use both qualitative and quantitative and the data analysis that is often inherent in interventional studies which includes cost-benefit cost-effectiveness analysis and quality control here’s some good web sources trocol actually provides a lot of good information free of charge and and this is public access on the web you