Now we're going to talk about common errors in board style question construction. I mentioned these briefly earlier, but I want to go over these four items, and we're going to start with all of the above and none of the above questions. So why should we not do all of the above and none of the above questions and the various combinations of these answers, like we talked about with the logic A and B, A not B but C or D, etc. So why are those no-no's? All of these allow a test taker to use partial knowledge to arrive at a correct answer, often without knowing all of the information the question is testing. So I'll give you an example right here. So which of the following is considered deep fungal pathogens? And you're reading the information. As you're reading the information, I know this one causes blastomycosis. I have no idea what this is. I feel like I know this one, but I'm not sure if that's deep fungal. Oh, sporothorax chankii, that causes sporotrichosis. So I have no idea about B and C, but I know A and D are deep fungal pathogens. So without having the knowledge of the question, I can get the question right. So I don't know all of the answers, but I can still get the question right. And that's really not okay. That's not okay because I had less knowledge than the other people that got it right that knew all four things. So it's not a valid question. So that's basically what this says here. The all of the above and the none of the above work the exact same way. It allows partial knowledge to get the questions correct, and that's not really what the best test question should be. It's not fair to the people who had more knowledge. The next kind of problem is super common, and it's called a multiple true-false question. A multiple true-false question is a very common, poorly written question in type A multiple choice question writing at the American Board of Dermatology, super common to see out of the novice first or second year question writers. So a multiple true-false question is where each distractor could stand alone as its own true-false statement. So test writers who use the multiple true-false question style are usually testing multiple concepts in one question instead of testing one concept per question. And that's a real cornerstone part of test writing. Every question needs to be testing only one topic to be fair. The reader is unable to answer without reading and considering each answer in a multiple true-false question. So you've all seen these kinds of questions. You just didn't know they were called multiple true-false questions, and you've probably written some multiple true-false questions if you've written these, because often these are the easiest kind of question to answer. And you just have to understand that these are not okay, and they're not going to pass the muster of the American Board style question writing. So here would be an example of a very common style of multiple true-false question. So all of the following statements about the treatment of pustular psoriasis are true except, now you can't, think about how we talked at the very beginning of what you should do in the ideal question. You should be able to cover up the answers and get the question correct without having to read the answers. You can't do that with a multiple true-false style question. You can't cover up these answers and be able to tell what the right answer is. You have to read each question. And when you read each question clinically, pustular psoriasis and the acute generalized exanthematous pustulosis can be indistinguishable in the early stages of disease outbreak. Is that true or false? And fliximab is effective first-line choice for severe pustular psoriasis. Is that true or false? And so you see, as you go down the line, that is what happens in multiple true-false questions, and that's not okay. It's testing multiple things in one question. It's not parallel in format. Every statement is totally unrelated to the statement before it. They're not related. That's something you have to try and avoid. And usually these are all things that are true except, or all are false except, or which of the following is true, which of the following is not true. Those are all multiple true-false style statements. They can get even harder to detect than just that, but that's the most common way that you're going to find this. So be on the lookout for them and don't allow them to move forward. Another example, which of the following statements are true about pediatric patients with acne? Well, that's the most classic introduction to a multiple true-false as ever there has been. So if that's the way you're writing test questions, you've got to change. But you can't cover up the answers and know what the right answer is because it's not really asking you anything. It's asking you to say true or false to five separate statements. And so that's not what we're looking for in type A questions. We don't want true-false line items in that kind of question. We talked last time in the original about multiple true-false and how we're trying very hard to eliminate multiple true-false as a type of question. These are the things that are, which of the following are true? All of the following are true except? Those are things that are not good style of questions. And I gave this as an example of a very classic multiple true-false question. And I just want to add a couple more. So when determining the appropriateness of Mohs surgery for a patient with skin cancer, the AAD Mohs Appropriate Use Criteria app, and then it makes a whole bunch of statements that you can see there below. Now this isn't saying which of the following is true, but that's what it's implying. It's sort of a hidden multiple true-false question, a little bit more tough to detect. So which of the following statements is true? If I cover up the answers with my hand, I can't tell what the right answer is. I can't answer the question by covering up the choices because you have to read each line to say what's true and what's false. Is that true? Is it false? So you want to avoid these kinds of questions. These statements are all independent of one another. They're not related. And so it's not good test form. So why do these occur and why are they the most common kinds of questions, writing errors made by dermatologists who write questions? Because I'm going to tell you, having critiqued thousands of questions now, I would say this is the most common kind of wrong style of question writing that we encounter. To me, it's because it's an easy type of question to write when you're assigned a topic. Somebody says, hey, I want you to write on psoriasis. I need a multiple choice question on psoriasis. Well, first of all, those can be very hard. The common diseases are actually very hard sometimes to write questions. What makes it easier is if you can make a multiple true-false question statement because you can figure out five true things about psoriasis and then take one of those true things and reverse it in its statement and make it false, or change one of the facts to be untrue. You know, something is greater than instead of less than or whatever. So then that makes your job easy as a test writer. But you have to work hard to avoid this type of question, I'm telling you. It's not fair on the test taker. It's testing multiple concepts in one question. Doesn't discriminate well. It's not fair, and each choice is actually its own true-false test question. So it's testing five things in one question. That's not fair if you happen to know four things right and one thing wrong. Now the next kind of don't-do-it kind of question is the so-what question. The so-what question, you know, this is really clinically important. You ask, is this really clinically important? So the answer may be true and the question that is asked may be valid, but is it really clinically relevant? You need to take your test writing and your question writing skills to the next level and begin by asking, what is a clinically relevant concept that I want to test knowledge of? That's really what you're trying to do. Is it clinically relevant? Is it written in a way that will realistically present in a clinical setting? Are you asking questions that are expected to be walking knowledge by the clinician or will everyone in practice go look up the answer before acting? Think about that kind of thing. Is this really the knowledge I'm trying to test? Are the differences in choices significant enough to be clinically meaningful? Is the question framed in the right clinical perspective? And this last one is really hard. This is really an advanced skill of question writers and this, I'll tell you, that took me a long time to master and I'll give you some examples as we go and hopefully you'll understand what that means as we go. So here's an example of a question to assess. So according to the 2020 U.S. Census, what percentage of U.S. population is comprised of those who self-identify as black or African American? What do you think the right answer is? You got your choice? Well, it doesn't really matter, does it? So while it may be true that there are 12.4 percent of the U.S. who identify as black during the 2020 Census, is the difference in that spread of choices here really an important thing to test? So it's true, it is valid, you know, but what are we trying to test here? And does it really matter if someone said it was an 11.4 and it's actually 12.4? Is that really that critical of a difference to discriminate? I would say this is a factoid. This is not something that gives us the right kind of question about black or African American percentage of the U.S. population. So here's the next one. So after reading this one, I want you to answer to yourself, why would I have a problem with this question? And the answer has to do with that clinical presentation. A little trickier, while it may be true that all of the answer choices cause skin necrosis, you are never going to assess a patient in the isolation of the context of how or why they arrived in the emergency department. You're doing, you're going to know more of the associated information. Whether they come in from cold exposure, whether they have a history of recent vaccine, whether they have renal disease in their past, et cetera. So none of these things are really a good thing to test in isolation. So because it's not clinically the way anything would ever present. So how does a question such as this test something useful in clinical practice? Design your question to be clinically useful and clinically relevant. That's what I'm going to encourage you to do. Here's another example. Herpetic Whitlow is most commonly found on which body part? Now you say, well, that doesn't seem that unreasonable a thing to ask. Well, you sure may know the answer, but it's, it's a kind of who cares? How is this clinically relevant? I would argue it isn't the kind of thing that will come up in clinical practice. This is an example of a test item with little clinical value. It's true that herpetic Whitlow is most commonly found on the finger, but how is discriminating this fact important clinically? You can get herpetic lesions on all these areas. I would argue that it's not clinically relevant in this manner. It would be far better to show a picture of a finger with a herpetic Whitlow and ask for the most likely diagnosis with realistic options as distractors. Now that is a clinically relevant herpetic Whitlow question. So you see, we're getting a little bit more challenging and technical in what's appropriate and what's not. It's all in the clinical presentation and the framing of your question. This one, I would argue, the way that it is, is not relevant. So when I first started writing questions at the American Board of Dermatology, you know, at the beginning of a test writing session, they would gather all the new test writers and they would give a little test writing orientation. Now a lot of that is done online in a very excellent way. But in 2005, I remember Lila Lee, who's still at the American Board of Dermatology in an assistant executive role, stating that it takes about one to three hours to construct one well-written test question. And at the time, this blew my mind. I'm like, oh my gosh, who's got time for that? How many of these am I supposed to write? But over time, I can tell you, having done that for over 15 years, it's really true. And not only does one need to review the literature to make sure that their answer is correct and that the best answer is currently still justified in the literature, but you also need to research all those distractors because, you know, you've come up with distractors, but do you actually know that those haven't been shown to be an answer sometimes? And so you have to be very careful about that as well. You want to not dirty your question by giving a sometimes right distractor. When you look deeply enough at the literature, sometimes you find your distractors have been shown to be true. So maybe it's in smaller reports or international literature or whatever, and you just want to try and keep your questions as clean as possible and avoid controversy that'll get them tossed later or appealed, you know, or argued or whatever. Whenever you can, you want to make sure that your distractors are legit. You also want to avoid brand new knowledge. You never want to be the newest kid on the block and test a question that's like straight out of the last, you know, JID because you want the knowledge to be, you know, withstanding the test of time before it becomes a mainstream question. And I'm not saying that, you know, there isn't great stuff that's worth knowing in the JID. I just am saying you want to make sure that some brand new discovery withstands the test of time as being true and, you know, minimally disputable. And this brings me to a tangential point, and that is you always want to test things that are fairly mainstream and available in clinical medicine. If this test is only available in one university's research lab, even though there's a paper on it that's very good, you probably aren't ready to use that as a test question subject because that's not really a clinically relevant test yet because nobody can order it unless you work at this institution. You want to let it be a little more mainstream and available to everyone before you're like, what's the best test for this or that? But the point of this is that we want to research the distractors as well. And that's actually what takes a lot of time is looking at the literature to say, well, this drug that I use as a distractor has been useful in this disease that I'm saying it's not the right answer. And so it's the kind of thing that you need to take a little bit more time to find the best kind of wrong answers to. This is another great quote that will come out of my time in the ABD test writing orientation. And that is, don't stay married to any one of your questions. Don't defend it to the death. It's not your baby. It's a question. That was a memorable quote for my first year of ABD test writing. And I must say that I had to hear that again and again before my defensive nature softened to the point where it really is all about making the best question possible. And what makes a question the best possible? It's for as many sets of eyes to review it and to think about it and dispute it and discuss it and to provide feedback on it and to raise questions and et cetera. So that's what makes a great test item in the end. And I can promise you that every one of the questions that ends up on the American Board of Dermatology examinations has gone through countless numbers of reviews by multiple, multiple different people, sometimes as many as 20 or more people have reviewed the question before it makes its final stand on the exam. And even then, sometimes it'll get thrown out because of some kind of controversy that arises that all those people still didn't catch. So it really is scrutinized. So final point, edits are natural. So those involved simply want to make it discriminate well and perform well on a test. So take the feedback or the opportunity to rewrite or to fix the item. Every test writer for the American Board of Dermatology test has gone through years of this process. Even the most seasoned item writers still get questions tossed. I get questions tossed every time I bring them forward. And you know what? It's something that's not such a big deal because it gives me another question for next year to work on and to fix. So it's all good. So whether it's your peer review process at AAD or any other organization you may be writing for, embrace the opportunity to improve your question writing through the peer review process. You will get better and better with time the more you allow that to happen.

board-style exam questions

partial knowledge

multiple true-false questions

so-what questions

peer review process