So now I want to talk about answer options and distractors. That's the choices and one correct answer. The best type A multiple choice questions have the following answer choice characteristics. First it's got five answer choices and we'll talk about why five answer choices is better than four or less in a minute. Number two, all answer choices are plausible. Ideally when you look at your question statistics, if your test is given over a large population of your target test takers, you would ideally want each answer to be chosen by at least some subset of your test takers. So every answer that you put in there, including all the distractors, ideally would be chosen by someone. Now that doesn't mean that every answer is chosen equally. That's not what we're looking for or not what we're expecting either, but you would like some bell-shaped curve of the answer selections. You want to avoid all of the above and you want to avoid none of the above type answers. I'll go through that in a minute also. We also want to avoid complex answer combinations. You've all seen these A and C or B or C but not A. These combinations where you feel like you actually need some kind of logic class or logic background to be able to understand what am I even answering here? I'm very confused. You don't want someone to miss a question on the exam because they're confused about the way that the answer is pieced together. You also want to avoid multiple true false questions, which I'll also go through further in a second. You want to make sure that one answer is generally accepted as being more correct than the other answer choices and not just the opinion of a question author. That was one of my problems when I first started writing questions. I thought that I had just the greatest mentorship in the world in dermatology and I did. There were many things that my teachers taught me that I wanted to impart to the world of dermatology through my testing. The problem is when you really start trying to put your finger on where does that information and dogma even come from, you actually find out the dogma often comes from that person's teacher and that person's teacher's teacher and it actually isn't written anywhere. Never was published. It becomes very hard to say this is the truth that everyone should know whenever no one has actually ever shown that this is the truth. It's really just their opinion as to what's the best practice. You need to be able to justify your answers somewhere in the literature. They need to be vetted in the literature and not just in the lore. That's one of the hard lessons I learned early as a test writer. Why does the American Board of Dermatology stress five different choices? Why not two? Why not three? Why not four? So the reason really comes down to the validity of the test and the ease of the psychometric evaluation of that test. When you have five choices you have a decreased ability to guess the correct answer by random guessing. That helps increase the reliability and validity of the test psychometrics. It minimizes the ability to randomly guess correctly. Four answer choices are technically acceptable on a board style test and can be psychometrically validated for testing, but five are really preferred in dermatology test writing, especially for the American Board of Dermatology. So although four is possible, five is best. Now when we talk about the answer choices, we have one clearly correct answer. The other four are what we call distractors. Those are the alternative choices to the correct answer. I want to spend a little bit more time talking about distractors. The best distractors have the following characteristics. First they reflect the common misconceptions on the topic. So if there's a topic that's frequently missed or has some other sort of common mistake that happens regarding the topic, that's a great distractor to include in a question. Distractors need to be clear and concise. They do not clue or cue the correct answer. So what does that mean, to clue or cue? Sometimes different kinds of distractors, although they're wrong, have certain qualities about them that actually point back to a different answer choice that's actually correct based on the way that the options are listed. A good distractor is grammatically consistent with the stem. So if it says the best is, then you don't want a plural form of a distractor because that wouldn't ever be correct to have a plural that went with a singular verb. So pay attention and be structurally sound. You also want it to be in a parallel form and by that I mean that if nearly every choice is a drug, you don't want one of the extra choices to be a lab test. More drugs in a lab test is not parallel in form. Choices should be similar in length. The American Board of Dermatology is particularly picky on the length of choices. It doesn't want the right answer to stand out simply because of its size relative to the other choices. All answers should be relatively similar in length. You need to use similar language. It needs to be arranged in alphabetical or numerical order so as not to queue. This is something I didn't realize until I was writing for the American Board of Dermatology who at the time when I started was getting their assessments critiqued by test writing experts at the National Board of Medical Examiners. But it can be very useful to just put everything in alphabetical order in your choices. This helps us not queue most answers being the third one or the letter C more often than whatever people naturally do if they're writing a question. Test writers will rarely put the right answer second, for instance, if you're doing it all by yourself. It's usually either the first choice or the third choice or occasionally the fourth. But if you put it in alphabetical order, it's a little bit more likely to be randomly located. Similarly, if you're dealing with numbers, put them in numerical order or if there's some other kind of logic to the order, then keep that logic throughout. That helps to not queue the correct answer. Avoid distractors that use absolutes. Things that say never or always, those are easy to eliminate because absolutes are almost never true. So that's not a good quality of a distractor to say always or never. Absolutes are not good in question writing. So likewise, realize that the opposite can be true too. So correct answers shouldn't have maybe or sometimes have or things that are non-absolute in the presence of other absolutes. You need to just avoid the absolutes and non-absolute combinations as best you can in your question writing. Now let's take a moment to practice. I want you to read through this question. And as you finish reading the question, I want you to assess it as either good or this one needs further edits. And if edits are needed, what edits would you make? Now for me, there is some serious testmanship possible here. The question asks, which of the following serologic tests? Well right away, two of the options are not serologic tests, so automatically you can eliminate them. I would get rid of those and replace them with serologic tests if that's what the question is asking for. Now we also have some other blood tests here that have a lot of abbreviations. And this is a good lesson as well to just tangentially talk about. You should not use abbreviations in your answers as the sole answer choice. You can have abbreviations in your answer choices, but they really ought to be parenthetical or the spelled out should be parenthetical one way or the other. So you might spell it out what each of these stands for and put the abbreviations in parentheses in these choices here or vice versa. You might start with the abbreviation and then spell it out. And how do you pick which one goes where? Whichever way that it is most known to the test-taking population, that's what should be listed first. So if VDRL is more well known than its lengthier non-abbreviated form, then put VDRL first and put the spelled out in parentheses. That's a good habit to have. Don't use abbreviations alone. Not everybody speaks the same way you cleverly abbreviated everything in your training or perhaps even now after your training. Don't make someone's knowledge of your abbreviation style to be the reason why they get a question right or wrong. You want it to be based on their knowledge, so give them all the information. all the answers must be possible and plausible. Imaginary choices or choices meant to be funny are not best practice and should be avoided. So when it's not plausible, true choices are less, so the question becomes less reliable. If you are funny with your unicorns and your bizarre fake animals, you're going to have instantly eliminated a couple of choices and now instead of 20% chance of guessing it right, you're more like at 33% chance of guessing it right. It's not as reliable an assessment and if you're making funny things that no one will pick, but just to get a kick out of the imaginary things that don't exist, that's not really a good question. Everything needs to be possible and plausible. That's what keeps your assessment reliable.

answer options

distractors

multiple choice questions

plausible choices

test reliability