Whether arriving very early is actually less stressful than arriving just on time

I arrive twenty minutes early for an appointment and spend the time sitting awkwardly outside. A friend arrives exactly on time and appears completely relaxed. I say I'm avoiding stress, but I've just moved the waiting into a different part of the day. How do you decide what buffer is genuinely useful versus needless extra time spent anticipating something?