Writing Test Questions is Hard
I’ve known this for a while. I first learned it when I was writing a test for my TEFL students. I had to figure out questions that were hard enough to demonstrate that they understood the material, but not so hard that they would be completely demoralized. I also had to make sure the questions were on the material the students legitimately should know, and not just on English language in general. When I started teaching HTML, I created a course, complete with an exam at the end, to demonstrate that students had learned something. The first iteration of the exam was fill in the blank. This made the test easier to write, but a lot harder to grade. So the second version was multiple guess. But multiple guess problems are hard to write because often the questions can be way too easy. For instance, how difficult would this be to guess the correct answer?
What does HTML stand for? a) Human Testing Meat Lovers b) Harsh Toboggans Must Leave c) HyperText Markup Language d) How To Make Lunch
If you guessed anything other than c) you were clearly not paying attention in the class. And many of the questions were like that, especially when I added in True/False questions like: True or False: HTML is a markup language. That question might be difficult, but it would almost always immediately follow the “What does HTML stand for?” question. Even if you guessed on the first question, you would see the second question and be able to figure out both. (Unless you were taking the quiz by randomly stabbing at answers, in which case, why are you bothering?) That isn’t to say that True/False questions are automatically easy, but that I find them difficult to write so that they aren’t either impossibly hard or ridiculously easy. So my first multiple guess quiz was not very good, mostly because it was too easy. So I wrote a second version.
Making a Multiple Guess Quiz Hard
I decided that my goal was to make the test extremely hard. So I modeled my quiz on multiple guess tests like the MCAT which I had just taken in order to get into business school. Those computerized tests are written with a fancy programming back end to give you harder questions for each one you get right and easier ones when you get questions wrong. I didn’t feel like programming something that complex, so I did something that annoyed many of my students: I set up questions to have 1 or more possible correct answers. And in order to get a point for the answer, you had to get all the correct answers, but none of the incorrect ones. In other words, my test did not give partial credit. And, since you were allowed to take the test multiple times until you passed, I did not show the answers. I was surprised (even if you are not) to learn that this made many students very angry. Once I put the test live I got many emails demanding I explain their scores, demanding the answers, arguing with the test questions and answers, and so on. Here’s a sample question from that test:
Which are valid ways to create bold text in XHTML? a) … b) … c) … d) … e) None of the above
Note that I didn’t include “All of the above” as a valid answer, as you could select all four values if you wanted to chose that. The correct answer to that question was A, C, and D. If you left out any of those choices, your answer was wrong. However, this introduces another way that writing tests is difficult.
Writing Unambiguous Questions
In the sample XHTML question above, I had several students argue that because answer C was actually CSS, not XHTML, that that was not a correct answer to the question. I disagree, and stand by my answer, but I can see what they are getting at. I had other questions on the test that were more ambiguous than that one, and I changed those questions to make them clearer. Writing unambiguous questions can be very difficult. I learned this writing my HTML test, but I lately had it driven into my head through the homework I’ve been correcting for my son’s school. Every week, the third graders get six pages of math and language arts homework. And nearly every week there is typically one question that is ambiguous. This week, the ambiguous question was:
Which is the best estimate for the liquid volume of a bottle of water? a) 500 milliliters b) 500 liters c) 5 milliliters d) 5 liters
The answer is A. But I would argue that it could have been D as well. A 5 liter bottle is the size of a water cooler bottle. And I wasn’t the only one who thought that because at least half of the tests that I graded had D as an answer. This question would have been less ambiguous if it had shown a picture of a personal water bottle or given some indication of the size water bottle they were talking about. But as it was written, I felt that it was unfair. Another set of questions that I found ambiguous or unclear were two very similar ones:
Is this sentence fact or opinion: It was raining all day. Is this sentence fact or opinion: I can never find a black crayon when I need one.
According to the teacher, the answers were:
It was raining all day. Opinion I can never find a black crayon when I need one. Fact
I disagree with both these answers. According to DifferenceBetween.net, a fact is something that is verifiable by evidence, while an opinion has no backing evidence. I would argue that “It was raining all day.” could be considered hyperbole, especially the way most people would use it in context, but without context it could just as easily be verifiable, evidence-based fact. While “I can never find a black crayon when I need one.” must be opinion because there are numerous black crayons in the world. While it is possible that someone might always struggle to find one, unless they deliberately set out to divest themselves of all black crayons, there would be at least one occasion where they needed one and one was available.
A simple quiz for you
As you can well imagine, teachers do not find me to be the most agreeable parent in the school. I have learned to not email the teacher every time I find a typo in the answer sheet or homework instructions. And I no longer bring up my objections to the questions on homework that is graded on effort rather than accuracy. But it’s hard. I have to write quizzes all the time. The book I’m writing right now has a quiz at the end of every chapter. It’s the part of writing each chapter that I hate the most. Mostly because I know that if I were a student reading the questions, I’d be objecting to the ambiguous ones, so I want to make them not too hard and not too easy.
True or False: Jennifer thinks way too much about third grade homework assignments.
This is most definitely a verifiable fact.