I would LOVE a planet money episode about this, it really is mind boggling that it's lasted this long.
I would guess that back in the day skateboards were overpriced, then became market corrected (around 2003-2009), and now with the competition becoming more spread out and budget options still being available (shop decks, remive, etc.) it's remained stagnant. That's really all just speculation though.
As far as other "specialty stores" there are a few more I can think of. Bicycles and ski/snowboard shops are big ones and often times these can bleed over to include skateboards in them. Golf and tennis are pretty big too.