You could ask the diminutive dude what he'd prefer to be called. If he finds whatever term you pick to be insulting or offensive, then you have some trouble to fix. So find out what he prefers and use that.
Also, keep in mind "petite" is the feminine version of this term; "petit" is the masculine one.