If 'pussy' is as derogatory as calling someone a 'dick', where's the misogyny? Sounds like gender equality to me. Is 'asshole' fair game? As a dude, I feel no way personally, or as a member of the male constituency, about dick being a common insult. I don't see how 'pussy' reflects negatively on women as a whole, and I don't think most people who use it colloquially think misogynistically when they refer to someone that way.
Genetalia need not be gendered anyway. The trans and gender non-binary community doesn't take very kindly to it, I've heard.