by David | Jan 18, 2023 | Automation
I was trying to pull out a big description block for an item in a recent scraping project. Of course, this contains all kinds of weird and wonderful HTML formatting as it is probably built in a WYSIWYG editor. I found that Scrapy doesn’t have a good way to...
by David | Apr 25, 2021 | Automation
Scraping tables The python Scrapy library is an excellent helper to build simple but powerful scrapers. It’s common to want to scrape HTML tables when we scrape text of pages and as I’m going to show it really doesn’t need to be difficult. The rough...
by David | Apr 25, 2021 | Automation
Scrapy is great, debugging Scrapy less so Are you adding print statements and then rerunning your scraper time and time again to get that one selector right? Do you have chrome open in the background and using jQuery to test those selectors live on the website you are...
by David | Aug 11, 2020 | Automation, Testing
I’m convinced that any time we want to build a large application we need automated testing. If we can’t run our tests automatically, we can’t scale development. The testing time will increase for each feature we add and eventually our testing cycle...
by David | Jul 29, 2019 | Automation
High test coverage is essential to safe refactoring and to ensure high test coverage you need to either work using TDD or make sure your coverage doesn’t drop. How will you make sure coverage doesn’t drop if you don’t know what it is? Here’s my...
by David | Nov 7, 2018 | Automation, Tools
Today I will show how I went about automating google translate to speed up the translation of a large set of terms I needed for one of my apps, TravelRates. The actual translation is very simple, however, since it is based on the paid version of...
