I've always had the opposite problem. Due a combination of likely genetics as mentioned above and the fact I used to walk almost everywhere when I was younger, I've always had naturally large calves; while I've also always been a hard-gainer when it comes to developing the upper body. Now at my present age and body type, I've come to the conclusion my calves have pretty reached max growth no matter how dedicated I train them but they will still always be bigger than my arms no matter how many bicep curls or tricep pushdowns I do.
But I do agree about how ridiculous the image is of these guys strutting around with hulking ape torsos supported by chicken legs as seems to be the majority at my gym. I have always been one of the few male members I've seen actually doing lower body work, let alone have a dedicated leg day, and it all comes down to both the image most guys think men should resemble and the will to rise above the discomfort threshold leg work entails.
I have had several guys comment on my calves (a lot of them wear sweatpants to hide theirs) and tell me they wish their's looked like that, but still don't seem to want to put in the work towards that goal. While I may have had a natural leg up on them (no pun intended), anyone, especially if starting young and with a Mesomorph type, can build impressive legs; but the average guy either for lack of priority or moxy, never bothers.