Nah. Would you skate someone's car without permission?
If I needed to do repairs on my house because some brat with a bmx bike decided it was the perfect spot for a clip I'd be pissed. If it was someone on a skateboard, I'd probably be OK with it, but I'm a different person than most. To the average person a skateboard, bmx, scooter, rollerblades, unicycle, etc., are all exactly the same if they're causing some kind of damage or even making unusual noise.
Then again, I'll likely never own a house because what even is real estate these days, but that's beside the point.