Deciding when a day trip deserves an overnight stay

A town we're considering as a day trip is two hours from our base by train. That gives us perhaps five hours there once we factor in breakfast and the return journey. It's enough for the main sights but not much time to wander or enjoy dinner. An overnight stay costs more and means moving luggage. Where's your threshold for turning a day trip into a one-night stop?