What should I say when people do this? I don't want to be too harsh.?

First, you have the disorder, not the personality, right? I have a touch of the personality, and it is a bit of a struggle to keep my mouth shut too, but I don't really know what the disorder is like, so I can empathize some and sympathize some also. It sounds to me like you're handling it well. Yes, word misuse is annoying, and people make stupid jokes, but it's mostly through ignorance, so I try to say little about it. You could point out that research shows that it's people of higher intelligence who seem more susceptible to disorders, so thank them for complimenting the intelligence of the person they're talking about. Ignorance can be a wonderful thing: it keeps the stupid happy. Ignorance is bliss. Good luck. ,