ConcepTest Questions for Statistics

Bruce Atwood

    Coming up with good concept test questions that fit with the content being covered on a particular day is time-consuming and difficult. Mazur (Peer Instruction: A User's Manual, Prentice-Hall Inc., Upper Saddle River, NJ, 1997) describes the basic principles behind writing conceptual questions (ConcepTests). Below are some examples of different types of questions.

Simple questions

    I ask quite a few questions that are very straightforward. These can serve many purposes. At the beginning of class they quiz students on the assigned reading. During class I often stop and use some questions as a quick check to see whether the students are with me by asking them to use a new definition or do a short calculation. Students usually perform very well on these questions and the positive feedback helps keep class pleasant for all. These types of questions are also good for exam review.

The distribution of math SAT scores for male students in 2005 was approximately normal with a mean of 538 and a standard deviation of 116. To calculate the probability that the average score of an SRS of 20 male students is 550 or higher we calculate the z-score as
    1)
    2)
    3) Neither formula is correct

Answer (2)

These questions are very easy to make up or find. Most textbook test banks and web sites have many questions of this type.

Self discovery

    "Leading" questions can be asked to move students toward a needed definition or formula.

A golf ball manufacture wants to test the effect of two different golf ball designs (A and B) on driving distance. Which of the following two experimental designs is best?
    1) For 100 golfers randomly assign 50 to hit drives with ball A and have the other 50 hit the other ball. Compare the difference in average distance for each group.
    2) Have each golfer hit both balls. Use random assignment to determine which type of ball they hit first. Calculate the difference in distance (ball A minus ball B) for each golfer and average these differences.

Answer (2): This is a matched-pair design. Why is it better?

I often have students work out a step (a key or difficult step if possible) of a derivation of a formula. This helps students think about the derivation where otherwise they might just be copying off the board. You could even ask a sequence of questions taking them through the steps of the derivation.

For the standard normal distribution P(-1.96<Z<1.96)=0.95 so approximately 95% of the observed values of Z are in the interval (-1.96,1.96) for a large sample. For a sample mean, , there is a 95% probability that
    -1.96<<1.96
Multiplying this expression by gives
    -1.96 <-μ<1.96
True or False: -1.96 <μ-<1.96
    1) True
    2) False

Answer (1): This result is obtained either by multiplying by -1 or recognizing that |-μ| is the same as |μ-|. Continue by adding to the expression in the question to show that the interval (-1.96 ,+1.96 ) has a 95% chance of capturing μ.

Finally, students can discover some ideas for themselves.

Let X have a binomial distribution with n=30 and p=0.4. This distribution along with a normal distribution with μ=n p and σ= are shown below. If we want to approximate the probability that X≤10 using the normal distribution we should calculate the z-score using
    1) X=9
    2) X=9.5
    3) X=10
    4) X=10.5
    5) X=11

Answer (4): (Continuity correction) Using X=10.5 will most closely approximate the area of the yellow bars. The exact solution is 0.291 while the approximations corresponding to the five choices are {0.132, 0.176, 0.228, 0.288, 0.355}.

Address misconceptions

    There are "classical" misconceptions:

True or false: To reliably estimate a population parameter larger populations require larger samples. (Assume that the population is much larger than the sample size.)
    1) True
    2) False

I ask this question early in the semester (in the context of sampling) and most get it wrong. I ask it again when we are calculating confidence intervals for proportions and almost everybody gets it right. I remind them about the previous time I asked them.
    I grade selected homework problems but don't go over them or exam questions in class unless more than half the class has done it wrong. In cases where they have done poorly, I often rephrase the part of the question where they went astray and ask about it in class. Their wrong answers make up some of the multiple choice responses.

Is the mean height of Beloit College female students less than male students? Let = the mean height of female Beloit College students and = the mean height for male students. What are the appropriate null and alternative hypotheses?
    1) : -=0 and : -≠0
    2) : -≠0  and : -=0
    3) : -=0 and : -<0
    4) : -=0 and : ->0    

Answer (3)

The p-value for the problem in the previous question was <0.004. Pick the best concluding sentence:
    1) The data provide very strong evidence that the difference in heights between female and male Beloit College students is less than zero.
    2) The data provide very strong evidence that female students are shorter than male students at Beloit College.
    3) The data provide very strong evidence that the average height of female students is less than the height of male students.
    4) The data provide very strong evidence that the average Beloit College woman is shorter than the average Beloit College man.
    5) The data provide very strong evidence that the average height of female Beloit College students is less than the average height of male Beloit College students.

Answer (5): The answer needs to specifically contain the word average or mean and must indicate clearly the population. The logical next question would be "How much shorter?" Thus a test like this one should be accompanied by a confidence interval.

    Discussion of common misconceptions (oversights) are a feature of Workshop Statistics Student Toolkit (Beth Chance and Allan Rossman, Key Publishing Corporation, Emeryville, CA, 2005). Another source of common student errors is the Student Performance Q&A accompanying the solutions to the AP Statistics exams at AP Central. These have been helpful sources for me to develop questions to give my classes extra practice on areas of difficulty.

Harder questions

    At least once a class I like to ask a question about an interpretation or concept that makes the students think hard. These are the questions that I'm hoping about half the class gets right so that I can use the peer persuasion technique or else have a springboard for some class discussion. These questions are harder to create or find. Most test-bank or web-site questions from publishers tend to be simpler, computational, or otherwise less conceptual than the type of questions I want. An exception is some of the test-bank questions accompanying the text Mind on Statistics (Jessica Utts & Robert Heckard, Thomson Brooks Cole, Belmont, CA, 2004). See also Assessment Resource Tools for Improving Statistical Thinking (ARTIST). Many of their "Forced-choice, Check List" problems can be used or modified to make good questions.

True or False: The following statements about 95% confidence intervals are equivalent.
a) There is a 95% chance that μ is in the confidence interval.
b) There is a 95% chance that the confidence interval contains μ.
    1) True
    2) False    

Answer (2): Two different samples will likely give two different confidence intervals. How can there be a 95% chance that μ is in each?

Fun questions

    These could be about current events that lead into the topic of the day. The main site for ideas is Chance. Statistic jokes are another possibility. A resource is CAUSEweb.org. There are also many classical probability questions.

There are 26 people in our class. Do you think the probability that two of us share the same birthday is about
    1) 1/10000
    2) 1/1000
    3) 1/100
    4) 1/10
    5) 1/2

Answer (5): Surprisingly, there only needs to be 23 people in the class for the probability to be greater than 0.5. One minus the probability that everyone has different birthdays is 1-= 0.598.

    A PRS can also be used to survey the class. The software displays the results as a count or proportion on a bar graph. Collecting data for a quantitative variable and transferring it to a statistical package for analysis is too cumbersome to do in class with the system I am using. Undoubtedly this will be easier with future generations of PRS hardware and software.

Graphical questions

    This category is included in those above. However, I have put it separately because these questions, although not terribly difficult to think up, are time-consuming to create. Some of my questions have been inspired by those in Activity-Based Statistics (Richard L. Scheaffer et al., Key Publishing Company, Emeryville, CA, 2004).

True or false: The following histograms could all be from the same dataset.
    1) True
    2) False

Answer (1): All the histograms are from the same sample consisting of 50 observations from a normal population with mean 37 and standard deviation 10 and 50 observations from a normal population with mean 63 and standard deviation 10. The point is that the shape (modality) of the distribution can change with interval size.

    I have included quite a few graphical questions in a small collection of all types of questions organized by topic:

HTML, pdf, Mathematica

    I hope these questions and remarks help you develop questions for your classes. Many other types of questions are also possible. Examples: ask for a prediction of the result of an experiment, or what should the first step be to solve a particular problem. Please email me with any questions, comments, or suggestions you might have.

Created by Mathematica  (August 14, 2006)