PSTAT 122
Project Description
Your final project is due on Wednesday 6/11/25 at midnight.
Preliminary notes:
• All projects are to be done individually
– Please do not discuss your work on your projects with any other students in the class at all. As these final projects are in lieu of a final exam, you should essentially treat this as a take-home final exam.
• Below are a number of suggested options that you might do for your final project. You are welcome to choose one of these, or propose your own idea (as also described below).
– If you propose your own idea, you must submit a brief description of your plan to Canvas, by Wednesday May 14th at midnight.
– Unless you are doing a simulation study (Option 3 below), all projects must involve the collection of real data.
– If proposing your own real data idea:
* It must fit either a Block Design or Full Factorial Design (with or without blocking) as described below
* The data must be COLLECTED BY YOU and it must be a designed experiment. Obser- vational data are not in the realm of this course so be sure that what you are proposing is indeed an experiment and not just observational.
• The AI policy for projects is the same as it has been for this entire course: you are welcome to use it, but please do so carefully, and with appropriate crediting for anything that you have AI help you with.
Please choose from one of the following options:
1) Randomized Complete Block Design (RCBD) or Generalized Randomized Block Design (GRBD)
In this option, you will design and implement an experimental study with one primary factor of interest and one blocking variable. Here is one possible suggestion:
i) Online Survey GRBD
Create an online survey via any platform of your choosing such as Google Forms that you will distribute to fellow UCSB undergraduate students. Your survey will primarily consist of ONE statement on a potentially controversial topic (i.e. political, something about campus, etc).
Specifically:
• Your factor of interest will be the wording of your statement.
— The different wordings should not change the overall message of the statement; that is, the core message of the statement should remain the same, but your versions of the statement may differ in ways such as tone, emphasis, or peripheral information surrounding the statement.
• Your outcome of interest will be a score from 0 to 10 indicating the respondent’s degree of agreement with the statement.
• Your blocking variable will be the class standing of the student. You may choose to keep it only to traditional Freshmen/Sophomores/Juniors/Seniors, or you may make it more flexible. Either way, the only other question in your survey will be the class standing of the student.
You should either create a separate survey for each version of the statement and distribute the links to these at random, or create one survey that is set up such that it will provide a version of the statement at random. You should have at least 3 versions of the statement to compare.
How you distribute your survey is up to you; for example, you may email it to individuals or to a group email list, or you may post it on your social media account, or a combination of these two; if you are distributing different links for each version of the statement, you will need to have a way to do this randomly. Your report should discuss the implications of how you distributed your survey in terms of whether you have truly obtained a random sample or not, and how this might impact your results.
Unlike in the RCBD, you will potentially have many replicates within each block. You just need to make sure that you have each treatment group represented at least once in each block, but potentially many more times depending on your sample size calculations.
ii) Your own design
You may also design your own RCBD or GRBD experiment. It must have one factor of interest with at least 3 levels, and one blocking variable with at least 3 blocks. You may also choose the above option but modify it in any way to your liking as long as it is still a RCBD or GRBD with the required number of levels and blocks. If you choose to design your own RCBD or GRBD or make modifications to the above suggestion, you must submit a brief description of your plan to Canvas by Wednesday May 14 (as noted above) so that we can ensure that it is a proper design and that it will be feasible in the time that you have before the deadline.
2) Full Factorial Design
In this option, you will design and implement a study with at least two primary factors of interest. Here are two possible suggestions:
i) Paper Airplanes Factorial Design
Similar to Lab 5, but instead of an RCBD, you will have multiple factors of interest to do a full factorial experiment. The factors are:
• Paper clip on the nose (yes/no)
• Paper clip on the middle (yes/no)
• Paper clip on the rear (yes/no)
That is, you could potentially have 2 or 3 paper clips on the airplane in any given replicate, unlike in the Lab 5 RCBD where you always had exactly 1 paper clip or 0 paper clips on the airplane.
The outcome of interest is again how far the airplane flies when you throw it. Your experiment should have multiple replicates, with the number of replicates determined by a sample size calculation (to be discussed further in class).
ii) Soda Taste Test Factorial Design with Blocking
Obtain a sample of multiple cans or bottles of any brand of soda (to be more economical, you might want to try to find a large pack of the smallest individual size possible). The number of cans/bottles that you obtain may be guided by your sample size calculations. The question at hand here is whether two factors of interest have any impact on the enjoyment (taste or otherwise) of drinking the soda.
The factors of interest are:
• Method of pouring:
— Down the side of the glass
— Straight onto the center of a glass
— Leave it in the can/bottle
• Method of consumption:
— Via a straw
— Straight from the container
Find participants to taste the soda according to each of the conditions. As each participant will try every combination of conditions, the participants are thus each a block. Participants should rate each taste on a scale of 0 to 10.
iii) Your own design
You may also design your own factorial experiment. It must have either:
• 2 factors with at least one of the factors having at least 3 levels
• 3 or more factors, in which case each factor may have only 2 levels (but more is also just fine)
You may also choose one of the above options but modify it in any way to your liking as long as it is still factorial design with the required number of factors and levels. If you choose to design your own factorial experiment, you must submit a brief description of your plan to Canvas by Wednesday May 14th so that we can ensure that it is a proper design and that it will be feasible in the time that you have before the deadline.
3) Simulation Study of the TukeyHSD test
i) Comparing TukeyHSD to ANOVA
In this option, you will design and implement a simulation study in which you will investigate the TukeyHSD test. Your aims of this option are two-fold (your project must investigate both):
• Aim 1: Demonstrate that, if the ANOVA conditions are met, the TukeyHSD test maintains an overall Type I Error rate of approximately α = 0.05, for the probability of making at least one Type I Error
• Aim 2: Compare the power of the TukeyHSD test to the ANOVA (again you may assume that the ANOVA conditions are met).
— That is: our usual ANOVA workflow is to perform the ANOVA first to see if there are ANY differences between the groups, and then follow that with a TukeyHSD test.
* One natural question to ask is, why do we do this? In other words, why don’t we just do TukeyHSD and forego the initial ANOVA analysis?
* The standard answer is that the TukeyHSD test is more prone to miss any significant dif- ferences, because its adjustment for the overall Type I Error rate costs it some statistical power.
* Here, Aim 2 is to investigate this claim via simulation.
For each of the above aims, your simulation study should investigate a large variety of scenarios. Specifically,
• For Aim 1:
— You should investigate how your results vary (or not) across varying numbers of treatment groups (e.g. 4, 5, or 6 groups). You should have at least 3 such scenarios (it can indeed just be 4 through 6, or you may choose other numbers).
— You may otherwise keep the following constant across your simulations (but you should make sure to clearly and carefully state what they are in your project write-up):
* Group mean
* Group variance
* Sample Size per group
— Here, a Type I Error has occurred for the TukeyHSD test if at least one of the pairwise p-values is less than 0.05 (equivalently, if the smallest pairwise p-value is less than 0.05).
• For Aim 2:
— You should investigate how your results vary (or not) across the following two factors:
* number of treatment groups
· Again you should have at least 3 different levels (e.g. 4, 5, or 6 groups)
* effect size (i.e. the group means and how different they are from each other)
· You should have at least 3 different levels here as well.
* Your simulation study should investigate every combination of these two factors (so for Aim
2 alone, you would have a total of 3 × 3 = 9 scenarios to simulate).
— You may otherwise keep the following constant across your simulations (but you should make sure to clearly and carefully state what they are in your project write-up):
* Group variance
* Sample Size per group
— For the TukeyHSD test: for simplification, you may count each replicate as a correctly identified significant result if at least any one of the pairwise p-values is less than 0.05.
While you can run initial simulations at lower repetitions/iterations as you investigate, in your final report your Type I Error probabilities and power should be estimated with at least 10,000 iterations per scenario. You can expect it to take a long time to run all of your required simulations at this level (likely 30 minutes or more, after you have it coded correctly). It will be best if your simulations are written in a script file, which then saves your output to an RData file, and then your Rmd file reads in the RData file; it will be overly cumbersome to write simulations of this magnitude in an Rmd file. If you do not know what all of this means, this project option is probably not a wise choice.
ii) Your own designed simulation study
You are welcome to modify the above suggestion and/or propose your own simulation study entirely. For example, you might be interested in investigating the impact of model misspecification on the performance of the TukeyHSD test, and compare it to its relevant permutation test (which you would have to write as part of your project). If you choose to design your own simulation study, you must submit a brief description of your plan to Canvas by Wednesday May 14th so that we can ensure that it is a proper design and that it will be feasible in the time that you have before the deadline.
Specific Expectations
A grading rubric and a skeleton project example that highlights the expectations of your report will be made available by Friday May 9th.