Shell (or Bash as it is sometimes called), is the command line prompt of Unix-based systems such as Ubuntu or MacOS and, as we will see today, it can be a huge friend of yours.
I have had the pleasure to work with some kings of bash and I was truly amazed bu what they were able to achieve using it.
Why so amazed? Because with plain bash, just like with plain javascript, the entire tooling is available on almost every single platform.
This means that you can copy this to almost any Unix environment, server or client, and it will run.
How to download URLs
Today, I had a list of URLs I had scraped from an online site. A bunch of pictures I needed to upload.
There has got to be a super simple quick way to iterate these and just download them right? And yes, there is.
Just to save you the google and copy/paste I’ve put it together for you in a simple gist file here.
Drop this somewhere and give it executable permissions with chmod +x and you can pass it the contents of a list of strings, either using pbpaste or you can cat the output of a file and pipe it.
You can refer to the commented-out examples in the file.
How does the download work?
There’s a while loop that puts each line in the standard input stream into the variable full_url.
I print this using echo for each line, just for debugging. Then the curl command. An incredibly powerful, useful and standard application for doing URL actions, takes the URL and downloads it.
I use the basename command to get the name of the file I’m downloading, removing the URL path and leaving only the name and extension. The $() part runs the basename command as a sub-shell, executing it separately and putting the output of that command in place of the $(). Magic ey!
And that’s all there is to it.
Happy downloading!