Women wear bras for more reasons than just to cover up their nipples. The support is necessary for a lot of us. Also, I've noticed a man's VDL and been intrigued and I've also noticed a guy is trying to get me to notice his VDL and it's weird. Like with everything else, women don't have the same reaction to men to things--men rarely are uncomfortable/unhappy when they get an unsolicited/unexpected view of a woman's body (downshirt, upskirt, braless, etc.), but women are often disturbed, weirded out by getting an unsolicited/unexpected view of a man's body. Partly because we're just built differently and partly because we get a lot of them and they're often threatening.
Speaking personally, I'm far more turned on seeing a man acting super-confident even if he's wearing fucking snowpants than just some random dude popping out of now where with an obvious boner in his pants.
And, in my experience, it's not the sight of braless nipples that seems to get most guys, it's the undisguised shape of the breast and the jiggling and swinging when I go braless that seems to be most intriguing to them. Bras really hide the shape and natural motion of a breast, but plenty of them fail to hide nipples.