I don't worry about it but the reasoning I've heard is kind off goofy.
"It's not racist, it's because bikers used to wear Iron Crosses and bikers are gnarly!"
Ok, then why did bikers wear it?
It reminds me of this sandwich place that used to be around here called Chink's Steaks.
"It's not racist, it's just the owner's nickname because he had really slants eyes,"