My Python scraper skips a few items and I can't see why. I've been adding `print()` calls for URLs, selectors and loop counters, then deleting them when I think I understand the bug. I want to learn to set a breakpoint, step through iterations and inspect variables while the script is paused. For a small project, what debugger workflow is worth practicing first so I'm not just moving the print statements into a prettier window?
Moving from print-debugging to a debugger on a small web scraper
- programming-help
- moving
- debugging
29
4 replies
Start by breaking at the point where the bug should already exist, not at the top of the script. For a scraper, I’d pause after fetching a p...
Start by breaking at the point where the bug should already exist, not at the top of the script. For a scraper, I’d pause after fetching a page and inspect the list of parsed items before the loop touches them. That tells you whether the problem is in retrieval, parsing, or iteration.
And watch the data shape, not just the values. I used to step line by line and still miss that I had a list where I expected a dict. A breakpoint on the first bad item is way more useful than ten print statements.
That makes sense. I think I’ve been staring at the whole loop and not the shape of the thing coming into it.