I am comfortable with who I am, but still fall into the ‘not good enough’ camp occasionally. I think the best thing, not always easy, is to reframe it to read: We’re not a good fit.
It sounds like she’s more concerned with what others will think, judging you based on the clothes you wear rather than the person you are. And, if her style is upper crust and yours is not… not a good fit.
I’m a laid back tomboy who wears jeans and tennies 24/7, swears like a truck driver, (was a truck driver) and comes without advanced degrees. My biggest complaint is that I wish they knew I didn’t want to be like them. 😉