For the worse? Its just that the board will sound dead 
I actually had some spit tablets that had pretty good tolerances. Worked very well with spacers.
And a proper mounted bearings, with spacers and axle nut tightned down, will make the bearings last longer.
Yes, skating a dead board compared to a snappy one is "for the worse," not sure how you could say otherwise. You realize how important audible feedback is while skating once it goes away. I'd rather have that than my bearings lasting a little longer. If you guys can skate like this then more power to you, but as I said there is a reason that no pro skater (and barely any hobby skaters) on the planet does this.
I think the main benefit of spacers is that they decrease the amount the inner race can flex towards the inside of the wheel when sideways forces are applied (powerslides etc.). This should make the bearings last longer.
Of course it would be nice to have wheels with proper tolerances but alas that is not the case with F4s at least and I will not skate anything else. Still not a reason to stop using them IMO.
Think about it though, if you don't have the axle nuts tightened, you are not getting this benefit, you are just creating sideways force on both inner races as opposed to just one. It only works if you are transferring that force to the hanger via the washers, by having no play. It absolutely is a reason to stop using them.
IF by some miracle you can find wheels with proper tolerance, I recommend everyone try it at least once. It's a great way to have a laugh at the park with your buds.