by David | Jan 18, 2023 | Automation
I was trying to pull out a big description block for an item in a recent scraping project. Of course, this contains all kinds of weird and wonderful HTML formatting as it is probably built in a WYSIWYG editor. I found that Scrapy doesn’t have a good way to...
by David | Oct 14, 2021 | General
Firebase Analytics is an incredibly powerful tool to understand how users use your mobile or web app. The past year I have been using it to draw insights and help prioritise and improve various functions in our team. Firebase Analytics is mostly good for collecting...
by David | Aug 28, 2021 | Tools
Shell (or Bash as it is sometimes called), is the command line prompt of Unix-based systems such as Ubuntu or MacOS and, as we will see today, it can be a huge friend of yours. I have had the pleasure to work with some kings of bash and I was truly amazed bu what they...
by David | Aug 23, 2021 | General
Taking a source data set, say a CSV with rows of data, and generating a list of pictures or emails inserting these rows of data into a template is commonly called mail merge or data merge. Every so often I find myself wanting to do this. The first time was years ago,...
by David | Jul 3, 2021 | Testing
In this article, I want to show a practical example of how to set up Gitlab continuous integration for Flutter with test reports. Tests need to be run all the time to be valuable. When they fail just after you introduce an error, you will know right away that what...
by David | May 8, 2021 | Tooling
I tend to automate the most basic tasks because it’s more fun than copy-pasting. If I need to do something more than ~5x I’m like to write a script for it. This time I was writing a blog post comparing some Flutter packages and had to generate badges for...
by David | Apr 30, 2021 | Development
Why this? I use Firebase for many of my projects and a big reason is that it takes away the complexity of handling an authentication system for no cost. It integrates with google login for one thing immediately. That being said, writing a backend using this works...
by David | Apr 25, 2021 | Automation
Scraping tables The python Scrapy library is an excellent helper to build simple but powerful scrapers. It’s common to want to scrape HTML tables when we scrape text of pages and as I’m going to show it really doesn’t need to be difficult. The rough...
by David | Apr 25, 2021 | Automation
Scrapy is great, debugging Scrapy less so Are you adding print statements and then rerunning your scraper time and time again to get that one selector right? Do you have chrome open in the background and using jQuery to test those selectors live on the website you are...
by David | Apr 7, 2021 | Development
That’s a mouthful of a title indeed but I found today that the plain Javascript SDKs bundled with modern browsers are capable of a lot. To be specific, to take a file, parse it as XML then put it through an XSLT parser and spit out HTML. Why David, do you want...
