Author Topic: Writing Test Questions  (Read 269 times)

zzzptm

  • Noble Ape
  • Administrator
  • District Champ
  • *****
  • Posts: 121
  • Country: us
  • Awesomeness: 1
  • Ceci n'est pas une pipe
    • View Profile
Writing Test Questions
« on: August 26, 2019, 09:42:33 AM »
If this is good, we'll sticky it.  :smug:

OK, so let's say you're tasked with writing some multiple-choice questions based upon material in a reading selection. How do you do it best? That's what this discussion will be about.

Part of what's best is avoiding what's bad. So, what's bad?

BAD 1. Do not have questions with "shock" value in them. That includes charged language, harsh words, clever jokes, graphic scenes, and even adverse scenarios. For example, I wrote some questions for a TOEFL study guide that involved a conversation in which a professor was informing a student that he was in danger of failing his course. That got rejected because the test-takers are under enough stress already, they don't need to be thinking about how *they* might be failing something.

Keep the questions to the facts at hand. Don't express opinions through the question. Students can evaluate conditions in the questions in answering them, but not in reading them.

BAD 2. Don't write questions or answers that are overly wordy. Your test-takers have reading levels, and we must not exceed those levels. If there are repetitive words in every answer, build them into the question and cut them from the answers. If the question goes over two lines of text, tighten it up. If an answer goes over one line of text, tighten it up.

BAD 3. Don't use words like not, except, or best. Use NOT, EXCEPT, and BEST. See the difference, there? You do not want a student to miss seeing one of those key words and then be tested on a faulty question.

With those out of the way, we're ready for the GOOD rules:

GOOD 1. The test question must correctly convey the evaluation to be performed on the information that is in scope for the test.

GOOD 2. Answers must be free of patterns. In other words, do not fall into a pattern where a student could surmise, "If all answers are short, take the longest one. If all answers are long, take the shortest one." I bring that one up because it is the most frequent patterns I've seen in answers. Keep the answers mixed in terms of length and which ones are correct. Be random - sometimes, the longest short answer is right and sometimes, it's not. If you need to keep a set of 20-sided dice handy in order to generate proper randomness, so be it.

GOOD 3. Keep the right answers random. Don't favor one letter over the others and don't exclude a letter relative to the others. Keep that 20-sided die handy and roll with every question: 1-4, the answer is A. 5-8, it's B, and so on up to 17-20 being E.

I'll add more later...
"Yeah, well... you know... that's just, like, uh... your opinion, man." - The Dude

zzzptm

  • Noble Ape
  • Administrator
  • District Champ
  • *****
  • Posts: 121
  • Country: us
  • Awesomeness: 1
  • Ceci n'est pas une pipe
    • View Profile
Re: Writing Test Questions
« Reply #1 on: August 30, 2019, 08:27:12 AM »
GOOD 4. Don't give away an answer with "best 3 out of 5" clues. Let me explain:

37. This question is complete gibberish to the test-taker, but the answers all have common elements. Based upon the combination of common elements, which of the following answers would be correct?

a. 3 + 2x
b. 3 - 2x
c. 2 + 2x
d. 3 + 3x
e. 2 - 2x

Answer: A
Reason: 3 of the 5 answers start with a 3, 4 of 5 have a 2x term, and 3 of 5 have a +. A has all three of those common elements, so it's the best shot at being the right answer.

So how to fix that question? I would get rid of answer (d), move (e) up to that slot and then make the new (e) "None of the above". Now it's a 50-50 proposition on any one element being key to the "right" answer, other than 2x, which is 100% in the right answer... unless that "None of the above" is right. And to keep that answer in play, it's important to have at least one other question have a "None of the above" be the right answer, so the test-taker will know it's not an empty threat.

With proper answers, then the test properly assesses whether or not the student knows the material, not if the student can spot patterns in the answers. I know this works because I've used it myself to get perfect scores on tests where I had no clue about what was being asked, but had a great list of answers that telegraphed the correct response.
"Yeah, well... you know... that's just, like, uh... your opinion, man." - The Dude

zzzptm

  • Noble Ape
  • Administrator
  • District Champ
  • *****
  • Posts: 121
  • Country: us
  • Awesomeness: 1
  • Ceci n'est pas une pipe
    • View Profile
Re: Writing Test Questions
« Reply #2 on: August 30, 2019, 08:42:52 AM »
BAD 4. Don't play mental games with the test-taker. I know I mentioned above about having "None of the above" be used as an answer to make it a credible deterrent on later questions, but that's not a game. It's more of a negotiating position, along with random distribution of answers. That's a good thing that tells the test-taker that *any* answer is fair game, so it's best to pay attention to the question, not the pattern in the answers.

If you *do* choose to play a game, then you're not assessing the material accurately. The test is instead an exercise in behavioral psychology. This in and of itself can be useful, but not as a valid assessment!

Let me explain... consider a test with 20 questions, all true-false. 1-19 are trivial and are all "true". 20 is very difficult, due to excessive wordiness and more than one conditional clause. Whether or not the answer is true or false, it will mess with the test-takers' minds. This is especially true if 1-19 are worth only 1 or 2 points each, and 20 is worth the other 81 or 62... that makes it high-stakes, high-pressure, and unfair as an assessment, but valid as an object lesson: "Always answer each question one at a time, and trust yourself!"

I gave two such tests, one after the other, with 20 being false the first time and 20 being true the second time. For those who confessed on the second try that they guessed I would go all true and that's why they did that on the last question, I said they got it wrong: the reason was part of the test!

They cried foul, and rightly so, so I said it wasn't a test for the gradebook, but for their minds. And also a warning that the reason for the right answer was more important than a good guess. Object lesson complete, I went back to normal tests and the students focused on learning the material carefully, so that they could get a good grade for explaining a question in the class participation session after the quiz.
"Yeah, well... you know... that's just, like, uh... your opinion, man." - The Dude

LAB

  • Coach Class
  • Packet Reader
  • *****
  • Posts: 34
  • Country: cl
  • Awesomeness: 0
  • Bevillathlon
    • View Profile
Re: Writing Test Questions
« Reply #3 on: August 30, 2019, 09:21:25 PM »
GOOD 4. Don't give away an answer with "best 3 out of 5" clues. Let me explain:

37. This question is complete gibberish to the test-taker, but the answers all have common elements. Based upon the combination of common elements, which of the following answers would be correct?

a. 3 + 2x
b. 3 - 2x
c. 2 + 2x
d. 3 + 3x
e. 2 - 2x

Answer: A
Reason: 3 of the 5 answers start with a 3, 4 of 5 have a 2x term, and 3 of 5 have a +. A has all three of those common elements, so it's the best shot at being the right answer.

So how to fix that question? I would get rid of answer (d), move (e) up to that slot and then make the new (e) "None of the above". Now it's a 50-50 proposition on any one element being key to the "right" answer, other than 2x, which is 100% in the right answer... unless that "None of the above" is right. And to keep that answer in play, it's important to have at least one other question have a "None of the above" be the right answer, so the test-taker will know it's not an empty threat.

With proper answers, then the test properly assesses whether or not the student knows the material, not if the student can spot patterns in the answers. I know this works because I've used it myself to get perfect scores on tests where I had no clue about what was being asked, but had a great list of answers that telegraphed the correct response.

This is so true. Especially with math. I teach math and make some great math tests for ACDEC practice every year (see: Bevillathlon). And that is so true with math and how you can just look at the answers and choose based on common elements.

When I make my math tests, I use excel to randomize question types and answer choices but usually because I get tired of randomizing answers (especially on financial questions) I just do

$2300
$2400
$2500
$2600
$2700

These makes them at least have to actually work the problem and not look for common elements.
Rockwall HS Coach, 2015 - Present