Deciding whether to slow easy runs further when they already feel very slow

Every easy-running guide says to slow down until you can talk comfortably. I already feel slow enough to be passed by enthusiastic walkers on hills. But when I push the pace, the next day's legs feel heavy and I don't recover well for the following run. Do you accept a very slow easy pace as part of building endurance, or use walk breaks to keep the effort genuinely low? The watch's pace number keeps messing with my head.