You inherit the X chromosome from your mother. The X chromosome plays a big part of the genetics in penis size.I never heard that. Mine is nearly identical to what I remember of my father but I never saw any naked guys on my mom's side of the family so...
And of course like someone mentioned evolution is imperceptible.