If the ball retainers are broken, the bearings can't "work" like they're supposed to. Just because they fill space in your wheel and still allow the wheel to spin doesn't mean doesn't mean they "work perfectly".
What you're saying reminds me of the little kids I see skating around on setups that are completely jacked. Like a kid that rides a deck for months until it's a mushy, flimsy POS with stress cracks coming through the top of the board and a squared off nose and tail. Then they say, "Brand X decks are the best, I've been riding mine for 8 months and it still has pop and it never snapped!"
I'm sure they'll completely fall apart of lock up eventually, only a matter of time.