How short should men's shorts be?

The only time I wear shorts is when I workout and they are usually around a few inches above the knee and they are a little baggy because I don't want them to be tight. I'm pretty sure that wearing "short-shorts" or tight shorts will get you branded as a creep if you are a man.