The game goes like this: That group has some people who are jerks, therefore I don’t want to be associated with that group and refuse the label even if by every reasonable definition I am a part of that group.
This isn’t a plea to ignore the existence of hate groups or bigotry, rather this is about the unwarranted but self-aided branding which happens when a cruel or foolish minority within a group causes other members of that group to refrain from identifying with it so they won’t be seen as associated with this cruelty. Of course how we choose to label ourselves is our choice to make, as is how large a minority of idiots we are willing to put up with while retaining the label, but abandoning labels when you are already a member of a disliked minority group has consequences. Besides the obvious linguistic problems of communication, both within group and to outsiders, you quite possibly face the further alienation and could in fact set yourself back.
Refusing to identify as a member of a group because some people within it are jerks or bigots is a sure way to make these individuals seem more prevalent or speak for the group as a whole. This brings with it the distinct chance that this group, and its associated label, will get even more hate which could end up setting back all associated causes, including whatever new label the escapees chose to adopt.
If I wasn’t laying it on thick enough by now, and though this applies to other groups as well, I’m talking about atheists who refuse the label because some people who are atheists are also terrible people. I’ve had atheists flatly tell me that they don’t identify as an atheist because some prominent member, or sub-group, of those communities is dishonest, bigoted, etc. Besides being linguistically bizarre (think “I’m not liberal because Bill Maher is an asshole”), I think this is poor strategy and, ironically, sets the stage for the types of endless rebranding and splitting you see in religions. It’s unfortunate but there are popular sexist atheists who will say horrible things about women and yes there are atheists who are just about as bad at reasoning as religious apologists but I don’t think that’s a good reason to abandon the label to the dark side.
Labels capture very little of who we are, and we shouldn’t expect them to do much more, however I just don’t think letting atheists become more associated with hate and stupidity is going to help secular people as a group overturn the death grip religion has on society.