Spacers definitely have a purpose, when the bearing has lateral load, it keeps it on track/in line.
Here is a brand new spacer vs. a 5.year old spacer.
You can obviously see the damage from lateral load on the worn spacer. If you don't have spacers, the bearing can become misaligned which can cause damage to the bearing, or the bearing popping out of the seat (picture jaws' wheel popping off after the apt stair gap/drop). Sure, not having isn't the end of the world in most case but if you're spending good money on bearings, not using the included spacers is counter productive