The vast majority never notice it.
For me, the few that have either were positive, assumed I was a LEO, or made a joke about it. I am always polite and answer whatever questions people have, but I don't play along with any jokes about using it on someone. Had that happen yesterday at the grocery store. Cashier joked to the bagger about me using on him. I simply said that was not what it was for.
When anyone tries to joke about guns with me, be they friends, family or strangers, i get my sternest look on my face ans tell them "I never joke about guns."