How short should men's shorts be?

The only time I wear shorts is when I workout and they are usually around a few inches above the knee and they are a little baggy because I don't want them to be tight. I'm pretty sure that wearing "short-shorts" or tight shorts will get you branded as a creep if you are a man.
Thankfully, short shorts are part of Aus/NZ culture, and we can and do wear them everywhere. Part of the short shorts acceptance comes from the popularity of NRL and AFL shorts worn as everyday clothing. I recently had a visiting Canadian expressing surprise that "they let him in" - "him" being me, having lunch at a pub wearing shorts.