I don’t have any smooth spots on my balls, but I wonder, do you wear tighter underwear that may consistently rub in those spots? I have places on the outside of both my thighs where my pockets have rubbed against them for decades and hair no longer grows. Same on my ankles from the short, tight socks I used to wear when I played sports. I assume the friction just destroyed the follicles.
It’s possible? I’m not a dermatologist or anything, it’s just what I’ve observed on my own body, specifically from my pockets is the biggest one. If you ride very frequently, maybe next time pay attention to where you’re body is making contact and causing friction. if it’s right where your balls rest on the seat and they move back and forth constantly there, it’s a good bet.