I can sympathise with your frustration.
You can still express your contempt w/o using certain words. If, for example, skaters (of non-specific gender/sexuality) were constantly getting in your way, you can moan about that specific behaviour w/o attributing it to their gender/sexuality. Them getting in your way is nothing to do with their gender/sexuality, right?
I guess it goes both ways too. Imagine if someone was going on about male privilege and discredit you simply because you are a straight male. It's just too broad and simplistic. It's not fair.
Someone called me a racist the other day just because I asked something reasonable but they didn't like being asked. In this instance the notion of 'race' was weaponised simply to undermine my point.
I guess you don't want to do the same like this deranged pathetic person (it was almost funny) randomly yelling at me 'racist!' just to have their way.
If your grievance is a legit one you should be able to find the right words to express it w/o resulting in attacking the entire group.