This is a thought that's always bugged me a little since I open carry nearly all the time. Though I may be deterring violence directed specifically at me, I'm also depriving others (who may not be armed) of potential intervention in a violent act committed against them. I'm not really sure how to square that circle.
They aren't your responsibility. Their safety and defense is their responsibility. If they haven't armed themselves, that's on them. You can't be everywhere.