Name: ______
AFDA: Surveys and Experiments Unit Test
Part I. Matching:
Write the letter of the appropriate term in the blank. Terms may be used more than once. (1 point each)
____ 1. Empirical tests of theories____ 2. No interference or influence on behavior
____ 3. Difficult to assess influence of potential bias
____ 4. Observe effects of varying one factor at a time
____ 5. Random assignment to a treated group / a. controlled experiment
b. observational study
Part II. True/False:
Indicate whether the statement is True or False by writing the word “True” or the word “False” in the blank provided. (1 point each)
______6.A test is statistically significant if it has been predicted as likely to have occurred by
chance alone.
______7. The sampling error usually decreases as the sample size N increases.
______8.An experimental factor is a controlled dependent variable set by the experimenter.
Part III. Multiple Choice A.
Directions: Using the survey context provided, circle the letter of the sampling technique that is most appropriate for the given context. (3 points each)
9. A teacher is working to survey 132 high school students to determine which type of lunch sandwich should be served at the school’s field day. She decides that every student should have an equally likely chance of being picked. Which sampling technique is the bestchoice?
a) Cluster Sampling: Each of the 11th grade math teachers survey the entirety of their own
class of thirty-six students, thirty-three students, twenty-eight students, or thirty-five
students.
b) Simple Random Sampling: Each student is assigned a number by a random number
generator, and the 132 students whose number is generated are surveyed.
c) Stratified Sampling: Each of the four 9th, 10th, 11th, and 12thgrade art elective teachers
randomlysurveythirty-three students from their own classes.
10. Miranda makes a bet with Antoine that no matter how many times her fifteen classmates flip a coin, 50% of the flips will be a tail. Which of the following theories and sampling techniques should Miranda use to best prove her point?
a) Using the law of large numbers, Miranda choses to have each student flip the coin
ten times and record their data in a class chart.
b) Using random assignment, Miranda choseseight of her classmates to flip the coin ten
times and record their data in a class chart.
c) Using convenience sampling, Miranda asks the first five classmates that arrive to each
flip the coin thirty times and record their data in a class chart.
11. Jayden is researching the types of singing that students most enjoy. There are three distinct singing groups in his school, and Jayden wants each group to be equally represented in the sample. Which sampling technique should Jayden use?
a) Simple Random Sampling
b) Stratified Random Sampling
c) Systematic Sampling
Part IV. Multiple Choice B.
Directions: Using the survey context provided, circle the letter of the method that is most appropriate for data collection. (3 points each)
12. Jared is preparing for a debate in his Government class. He wants to find out what other students thing about his topic, but he is afraid that his peers will not feel comfortable answering the questions in front of each other. Which type of data collection method should Jared use?
a) Jared should form focus groups.
b) Jared should conduct a survey.
c) Jared should use transcript analysis.
13. Brianna wants to know how adults age 18 – 25 feel about retirement options. She realizes that adults of this age category may need the advice of older adults or research to be informed and be able to give thoughtful responses. Which type of data collection method should Brianna use?
a) Brianna should conduct a paper-to-pencil questionnaire.
b) Brianna should conduct a computer assisted personal interview.
c) Brianna should conduct a discussion within focus groups.
14. Malik is writing a five page paper on mathematicians for his career exploration class. He decides to find out more about a mathematician’s daily life by contacting four math professors at William and Mary and asking them about their own college education, their current research, their life goals, and their interests outside of math. Which type of data collection method should Malik attempt to learn more about the lives of these four professors?
a) Malik should perform a case study on each of the four professors.
b) Malik should read a literature review written by each of the four professors.
c) Malik should conduct in-person interviews with each of the four professors.
Part V. Multiple Choice C.
Directions: Circle the letter of the correct response for each question. (3 points each)
15. In a popular teen magazine, Emily read the results of a text-in survey regarding texting messages and emails. The article stated that 77% of teens prefer texting messages instead of emailing messages. Which best reflects the bias in this survey?
a) As a result of response bias, teens that prefer to email were equally likely to choose to
take part in the text-in poll.
b) As a result of self-selection bias, teens that prefer to text were more likely to choose to
take part in the text-in poll.
c) As a result of undercoverage bias, teens that prefer to email were less likely to choose
to take part in the text-in poll.
16. For her research class, Ally has created an experiment to judge the effects of caffeine on middle school students (with the permission of their parents). She conducted the experiment in four different schools. She separated each school’s students into two groups. To one group, she gave caffeine pills and to the other group she gave sugar pills. Ally firmly believes that the students with caffeine pills will act more hyper within the following hours, and she shares this belief with the students after they are placed in groups. Which is the best choice for Ally to reduce bias?
a)Haveone of her peers in the research class conduct the experiment so that Ally will
not know which group was the control group and which was the treatment group until
after data has been analyzed.
b) Give the students in the treatment group the caffeine pill, but get rid of the
placebo in the control group so that those students do not take any pill.
c)Repeat the experiment exactly as stated in a series of trials to compare the data that she
collects from the first trial’s control and treatment groups.
Part VI. Multiple Choice D.
Directions: Using the given context, circle the letter of the best answer. (2 points each)
17. Dominic’s proposal to use intervention research in order to understand the effect of grading student papers in a certain color of ink has been accepted. Knowing that he wants to use random assignment, which type of grouping should Dominic use?
a) Group in experimental/control groups
b) Group in focus/opinion groups
c) Group in pre-existing/natural groups
18. Raheem is assigned to complete an observational study on first graders and their eating habits at lunch. He hopes to collect information on how many first graders eat the recommended servings of fruits and vegetables for a meal during their school lunch. Which type of grouping should Raheem use?
a) Group in experimental/control groups
b) Group in focus/opinion groups
c) Group in pre-existing/natural groups
19. Natalie is conducting a randomized controlled trial. Why is it important for Natalie to use
random assignment?
a) to balance individual characteristics between groups
b)to save money and time throughout the experiment
c) to reveal causation from the data results
.
Part VII. Short Answer
20.Directions:In the answer space provided state three examples of possible bias from the following survey plan. For each example, state the type of bias and describe a way to reduce the bias. Predict how changing the survey plan to reduce the bias will affect the survey plan’s results. Use complete sentences. (12 points)
Survey Plan:
Josiah is surveying households in Hampton on their preference of a year-round school year versus the traditional plan. He decides to call households to ask his questions. In order to choose who he calls, he opens last year’s phone book to the first page. He then calls the first fifteen numbers in the phone book. Josiah asks the following question of whoever answers the phone: Our county is moving towards a year-round school year in a research-based effort to increase student learning; do you agree with this change? He records the answers as either “Yes, I support year-round school” or “No, I do not support year-round school”. If the person is not certain, he asks them to choose. If the phone call is not answered, he does not record any data.
Answer Space:
Answer Key: AFDA Surveys and Experiments Unit Test
1. a. controlled experiment
A controlled experiment is an empirical test of theory.
2. b. observational study
An observational study has no interference or influence on behavior.
3. b. observational study
In an observational study, it is difficult to assess the influence of potential bias because
there are not controls set to remove bias.
4. a. controlled experiment
In a controlled experiment, the experimenter can observe the effects of varying one factor
at a time.
5. a. controlled experiment.
In a controlled experiment, subjects are randomly assigned to a treated group.
6. False
A test is statistically significant if it has been predicted as unlikely to have occurred by
chance alone.
7. True
The sample error usually decreases as the sample size N increases.
8.False
An experimental factor is a controlled independent variable set by the experimenter.
9.b) Simple Random Sampling
Each student is assigned a number by a random number generator, and the 132 students
whose number is generated are surveyed.
10.a)Using the law of large numbers, Miranda choses to have each student flip the coin
ten times and record their data in a class chart.
11.b) Stratified Random Sampling
Each strata, or grouping by age level, will have the same representation within the sample.
12.b) Jared should conduct a survey.
Surveys are good to use when the topic may be uncomfortable or controversial and
accurate answers may not be given if questions are asked in front of others.
13.c) Brianna should conduct a discussion within focus groups.
Focus groups are good to use when people may need more information, benefit from discussions with others, or need more time than direct question/answering to develop a thoughtful response.
14.c) Malik should conduct in-person interviews with each of the four professors.
In-person interviews are good to ask extended questions and provide room for follow-up questions for an adequate gain of information but in-depth research does not need to occur.
15.b) As a result of self-selection bias, teens that prefer to text were more likely to choose to
take part in the text-in poll.
Teens who prefer to text are more likely to take part in a text-in survey. This is an example of self-selection bias.
16.a)Haveone of her peers in the research class conduct the experiment so that Ally will
not know which group was the control group and which was the treatment group until
after data has been analyzed.
The main source of bias in this case is that Ally has a preconceived notion of the experiment’s result and has shared this idea. This bias will change the way that she looks at results if she knows which group was the control group and which was the treatment group.
17.a) Group in experimental/control groups
Dominic wants to use random assignment, which is an aspect of true experimental research and experimental research is a type of intervention research, so experimental/control groups should be used.
18.c) Group in pre-existing/natural groups
Raheem is doing an observational study, so he should group in pre-existing/natural groups so as not to interfere with the behavior.
19. a) to balance individual characteristics between groups
An advantage of randomization of assignments for randomized controlled trials is that randomization balances the individual characteristics between groups.
Short Answer #20 Grading Criteria: Analytical Rubric
Category: / Inadequate(0 points) / Adequate
(1 point) / Very Good
(2 points) / Excellent
(3 points) / Points Rewarded:
Examples of Bias / Fails to correctly identify an example of bias. / Correctly identifies one example of bias. / Correctly identifies two examples of bias. / Correctly identifies three examples of bias. / /3
Type of Bias / Fails to correctly identify the type of an example of bias. / Correctly identifies the type of one example of bias. / Correctly identifies the type of two examples of bias. / Correctly identifies the type for each of the three examples of bias. / /3
Reducing Bias / Fails to describe a way that will reduce bias for an example. / Describes a way that will reduce bias for one of the examples of bias. / Describes ways that will reduce bias for two of the examples of bias. / Describes ways that will reduce bias for each of the three examples of bias. / /3
Prediction / No predictions are made. / Predictions of results are not based on how the survey plan would change with the reduction of bias. / Predictions of results are based on how the survey plan would change with the reduction of bias but these changes are not logical. / Predictions of results are logical and are based on how the survey plan would logically change with the reduction of bias. / /3
Total:
/12
Intended Learning Outcome:
The student will identify possible sources of bias, identify the type of bias, and describe ways to reduce bias when given a plan for a survey.
Analysis synthesis, evaluation
Add to table of specifications
Validity and Reliability:
With a high degree of validity, the test item measures both the content and the level of cognitive demand of the intended learning outcome. The directions are written based on the ILO, so asking the student to identify possible sources of bias, identify the type of each bias, and then describe ways to reduce bias when given a plan for a survey fits the curriculum and instruction. The levels of cognitive demand are also met as the student will use evaluation in order to identify sources of bias by critiquing and predicting which parts of the survey plan will create a bias that impacts the survey plan, comprehension to identify the type of bias, and synthesis to adapt the survey plan in order to reduce the bias.ANALYSIS NOT COMPREHENSION – go through this and check it for allIn order to further assess these higher level cognitive demands, the question introduces novelty by having students predict how their changes to the survey plan will affect results (an idea that was not discussed in class but that they will have, albeit unknowingly, observed in their previous project). This tests the ability of students to analyze how their changes will not only affect biasness but also how they will affect the way that the survey is conducted and the results that occur. The validity is also maintained as both the directions and the item prompt are at a 6th grade Flesch-Kincaid reading level. Thus the reading comprehension and ability of the students is not being tested or changing their ability to complete the math aspects. In these ways the item assesses what I intend for it to assess, and the intended assessment aligns to the intended learning objective in both content and levels of cognitive demand.
When I was constructing the short answer item, I worked to make the directions, the prompt, and the nature of the response as clear as possible. After understanding what I expected or hoped to see in a response, I built the directions and the prompt around these factors. For reliability, I avoided systematic error by giving clear expectations in stating the number of examples that I expected as well as giving the directions to write in complete sentences. I also, especially in this question regarding bias, avoided personal or cultural bias. For example, at first I wrote that the school was changing to a year-round school to increase the low-ranking education ofstudents in the United States. Then I realized that this could be a cultural bias. Instead, I simply said “in a research-based effort to increase student learning”. Having Josiah leading with this before asking the question was, of course, an example of bias but that was purposefully included in the survey plan (not the prompt writing itself). In adhering to the rules of short answer question writing, I also worked to ensure that the student’s response would truly demonstrate what they have or have not learned without having the prompt provide the correct answer. For this reason, in building the prompt I added in a few statements that are questionable to whether they would introduce bias into the survey plan, but they do not fit any of the types of bias that we have learned. For example: “If the person is not certain, he asks them to choose”. This statement seems to show Josiah improperly affecting their answers in some way, but it is not a type of bias that we discussed (especially with clearer examples in the prompt). Thus the student should not choose this statement as one of their three examples, but should instead choose a statement that they do know the type of bias. In this same cause, the response space is adequate in that it provides enough space for students to fully describe how they would reduce the bias without encouraging paragraph-length answers. Although it takes up the rest of the page, this was a chosen length of space and not just a choice of convenience. The blank space, as opposed to lines, gives the students the option of using the space more fully with margins and “font” size if they feel like they need more room.