Shell (or Bash as it is sometimes called), is the command line prompt of Unix-based systems such as Ubuntu or MacOS and, as we will see today, it can be a huge friend of yours.
I have had the pleasure to work with some kings of bash and I was truly amazed bu what they were able to achieve using it.
This means that you can copy this to almost any Unix environment, server or client, and it will run.
How to download URLs
Today, I had a list of URLs I had scraped from an online site. A bunch of pictures I needed to upload.
There has got to be a super simple quick way to iterate these and just download them right? And yes, there is.
Just to save you the google and copy/paste I’ve put it together for you in a simple gist file here.
You can refer to the commented-out examples in the file.
How does the download work?
There’s a while loop that puts each line in the standard input stream into the variable
I print this using
echo for each line, just for debugging. Then the curl command. An incredibly powerful, useful and standard application for doing URL actions, takes the URL and downloads it.
I use the basename command to get the name of the file I’m downloading, removing the URL path and leaving only the name and extension. The
$() part runs the
basename command as a sub-shell, executing it separately and putting the output of that command in place of the
$(). Magic ey!
And that’s all there is to it.