Thank you. My name is Daniel Schlesinger. I'm currently a Mohs surgery and cosmetic surgery fellow at Northwestern University in Chicago and I'm going to be talking about kind of interesting topic of prompt engineering and large language models. And here are my disclosures. It's mostly been discussed but just some basic vocabulary. LLM stands for large language model, a la chat GPT or bar Gemini. Natural language processing is NLP. Generative AI is this newer field that we've been talking about a lot which is able to create new content. And there's been a huge explosion in generative AI and LLMs over the last two years with lots of potential applications for our field and many others. And I think broadly speaking when you're thinking about this a lot of times the question that you're asking depends a lot more than the answer that you get. And so prompt engineering is all about optimizing those questions to get what you want. Just thinking about what kinds of things generative AI can give us. These are some broad categories. We're most familiar with text but you can generate images and videos and even code and documents. And we'll even talk about other ways to think about it. But there's lots of different categories of things that generative AI can give us. Briefly timeline of LLM evolution. Although it started many decades ago things really turned a corner in 2017 when there was a transformative paper about transformers which revolutionized the way that LLMs work because it essentially allows them to have this property of self-attention. To remember what happens in the beginning of a sentence or a paragraph or document even as it's processing the end of it. And being able to pay attention to the entire thing all at once gives it the ability to give you much better answers to things. Okay so a few lessons. So let it help you get past writer's block. When I was thinking about this presentation for example I turned to Chachi B.T. and asked what I should talk about. I thought this was kind of interesting. It immediately knew I was talking about the American Academy of Dermatology. I wondered how it knew that. I asked it. It said it used context and common associations. But when asked what context it had it didn't have anything specific. And I actually asked Gemini which is which used to be known as BARD, Google's AI assistant, the same exact question and it thought I was talking about a Microsoft Azure meeting. So you get really different answers. I thought maybe I'd use my previous conversations when considering context but it said it I had started in a brand new conversation so it wasn't using any of that. But just just to think you might be surprised but know your GPT. So a lot of people know Chachi B.T. That's the one that that has been the most in the news. Here's the basic the free version of Chachi B.T. Correct this paragraph. Psoriasis with lots of misspelling psoriasis is a common skin condition dry itchy skin first-line treatments are long-term oral steroids and antibiotics. It corrects the spelling mistakes but it leaves in this weird statement about the first-line treatments. I upgrade to chat GPT for it corrects the spelling mistakes and it corrects the first-line treatments. So just to go to show you get drastically different outputs depending on the model you're using. So Chachi B.T. is freely publicly available as 3.5 for everyone. If you pay some money every month you'll get Chachi B.T. 4 which gives you a lot more nuance and complexity. You can attach files which lends multimodal applications. It also gives you access to the Chachi B.T. store which is kind of like an app store. Basically you or anyone else can build custom GPT. So you can take Chachi B.T. and modify it for your own purposes. I just heard of a large beauty company even making their own internal GPT things so that all their employees have internal consistency essentially. It's all done with a no code so you just type in words and you feed it images and you train it to do whatever you want. On the same note diversifier sources. So we've talked a lot about Chachi B.T. I wondered if Chachi B.T. could give me some answers about this article I read about acne creams causing cancer yesterday and it gave me kind of a rundown of the article that was published a lot. It just kind of summarized it. But if I ask Google's Gemini the same exact thing it goes super in-depth and it not just summarizes the article but it thinks about important considerations and where I can find more information and it goes way more and it gives me so much more nuance than I got from Chachi B.T. There's other advantages that I found with Gemini over Chachi B.T. It has access to real-time information. So Chachi B.T. even the newer version of 4 is date locked at April 2023 so it doesn't have any knowledge past then. It can sometimes kind of sort of browse the web like it did with the acne causing cancer article but its knowledge is still locked. Gemini is not. You can also modify the prompts to kind of tweak it a bit more a bit more real-time and it has the ability to accept videos. Perplexity is another well-known large language model. Like the others there's a free version and an advanced version. The advanced version is CLAWD2 and its main strength is that it is able to give you these direct source citations and that is able to lend it some credibility that the others are not always able to do. So here's an example it basically gives you this is perplexity it gives you these direct citations including from medical journals right up top. This is basically summarizing which ones have access to real-time data versus not and which ones have web browsing capabilities. It's all a bit muddy and and vague and even though they sometimes say they don't they sometimes can and vice versa. But beware of hallucinations. So I asked Chachi B.T. on the left to generate a paragraph bio for Daniel Schlesinger neurosurgeon and it came up with a great bio for me. Unfortunately it's not part of my training and Gemini was able to recognize that it said they simply did not have enough information about me. So hallucinations are a common problem. This was in the news a lot several months ago a lawyer used Chachi B.T. to generate legal arguments and it basically made up everything and then when he asked it to substantiate the case law it made up fake cases to document those. When you're engineering the prompts I find it's very helpful just like when speaking to a student or any other human ask it to show its work don't just take it at face value. So I showed it an example of this rash I just said what's going on here and it gave me this kind of vague response there's these clustered red spots right near the armpit it gave this kind of differential I said can you be more specific with regards to diagnosing it gave me this long differential diagnosis but still not honing in on it I said just pick your most likely diagnosis it's herpes zoster it gets there but it wasn't it wasn't willing to get there initially even though when I asked it to explain how picked herpes zoster out of all these other diagnoses and it was right it had all this stuff it wasn't willing to go there initially but as I was able to lead it through with prompts it was able to finally get there. Think beyond text so a lot of what we've discussed and what people have used this for is feeding text and generating text but there's so many mother so many other applications beyond that you can feed it a data file and ask it to do research analysis you can feed it code or feed it text and have you give it give you back code or images or PDFs so here's an example help me perform data analysis on my data set that's an okay prompt but it's better if you say this is my RCT comparing these two populations describe the contents provide descriptive statistics perform and suggest a test of significance for this I asked it to just I asked Gemini just build me a website and give me the code for it have various features it gave me HTML CSS and JavaScript on the very first prompt and I'm sure if I had gone through it in more depth it could have built it out fully. So Dr. Lee sent me this great presentation from Stanford that was that that we watched on YouTube and it discussed this interesting thing called the history of future illness so we're thinking about natural language processing as dealing with words and it taking words in a sentence and then predicting the next one but it language doesn't have to work with human words what if it works with ICD-10 codes and what if it took every ICD-10 code and predicted the next one so I think that it's not been as fleshed out yet but this this is a possibility and I think thinking more broadly about the kind of data you're feeding in and the data you're getting back it's kind it's about pattern recognition and global situational awareness rather than necessarily human language try it for all the stuff you hate doing I convinced Gemini to write a fake employment contract for a dermatologist and asked it to write it with lots of legalese and heavily favor the employer I then gave it to chat GPT to analyze it and give me its feedback it gave me a lot of good feedback it picked up on a lot of it but I didn't trust it so I gave it back to Gemini and I said double check chat GPT's work and it was Gemini was actually impressed by chat GPT but there were a few things that it missed like the general principle of ambiguity as a weapon in contracts something you've always got to look out for but it's totally fine you can pit one LLM against one another you can also use it to help you with other prompting for other LLMs give me a complex mid-journey prompt which is an image generator and it gave me the prompt and it generated the images so just just thinking about if you're not getting the right answers from the one AI asking another LLM to help you this is another topic unto itself but just there's so much that you can do to get great prompts and image generators and all these LLMs can help you do it there's a bajillion different use cases you could think about in practice management thinking about profitability and social media and patient education and diagnostics and daily practice and billing and coding and LLMs can help you with all of those things and lastly beware of privacy and confidentiality concerns this is a big disclaimer on Gemini I think came up as soon as I started asking it to generate human images of skin diseases which it was unwilling to do it's a complex topic and it varies from one LLM to another and there aren't really great answers quite yet but some people are working on it in the same presentation that dr. Lee sent me there was I really loved this statement no aggregation without representation but just like your data that you type into Google and search and browse is being aggregated your data that you type into LLMs is also likely being aggregated it's it's pretty disturbing but it probably either is happening or will be happening soon to for you to be remarketed to this organization the light collective is working on trying to distill this especially with regards to health health data because getting a bad diagnosis is already traumatizing enough and then being followed and remarketed against it as you're trying to get new information is even worse and on that note your patient portals by the way at over a hundred hospital major medical systems have been doing this and they they published this in a big investigation but they found that even in places where you allegedly have protected health information there's remarketing happening so the same thing will likely happen in LLM thank you so much

large language models

GPT

Gemini

self-attention capabilities

text generation