I have not yet had wheels that would have too wide inner spacing for any spacers I've used. Too narrow, yes.
I personally use spacers, and my argumentation for them is that when they're properly in place, they half the impact and load sideways, by contacting the both bearings of the wheel. That happens even when there's play in wheel. I usually tighten the wheel up (and with spacer and both washers, it should still rotate smoothly, but with "dead" boardfeel), then loosen the nut 1/4 turn.