1

@slhck provided a useful answer to this similar question.

Downloading multiple files with wget and handling parameters

This code was provided to read urls from a text file filelist, then run a loop to download each.

    #!/usr/bin/env bash 
    while read line 
      do   
        wget -c --load-cookies cookies.txt $line -O ${line##*/} 
    done < filelist

 

As I'm not familiar with shell scripts, I am wondering if you could download multiple xml files by:

  • creating a simple comma seperated text file filelist
  • read from:

filelist

renamedfile.xml, url-01.php
renamedFileDifferent.xml, url-02.php
specificFileRename.xml, "url-03"
newFilename.xml, "url-04"

  • read through each line
  • split the line into newfile, remoteFile

  • And run:  

    wget -O newfile remoteFile >/dev/null 2>&1

 

Is it possible to write this in a shell script? Can you provide an example?

Community
  • 1
rrrfusco
  • 145
  • 8

1 Answers1

1

To get the text from before the comma you use ${line/,*}.

(What this actually does is replace ",*" or all the text from the comma onwards with empty string - leaving only the part of the string before the comma)

To get the text from after the comma you use ${line/*,}.

So the full command would be:

while read line
  do
    wget -O ${line/,*} ${line/*,} >/dev/null 2>&1 
done < filelist

or, on one line:

while read line; do echo wget -O ${line/,*} ${line/*,} >/dev/null 2>&1; done < filelist

In Windows (assuming you've installed wget, from http://gnuwin32.sourceforge.net/packages/wget.htm, and set your path correctly), it would be:

for /f "delims=, tokens=1*" %a in (filelist) do wget -O %a %b
Luke
  • 1,145
  • 1
  • 9
  • 17
  • What directory is a good place to put the text file, and shell file typically? This will be run as a cron job. – rrrfusco Sep 25 '12 at 00:59
  • Wherever you want to run it from (and save the files to). eg. if you put `filelist` and `test.sh` in sub directory of your home, and then run it from there, that's where it will save the files. – Luke Sep 25 '12 at 01:03
  • I'm not sure it will run as expected if run as a cron job. You will want to specify absolute paths in the filelist (`~\downloaddir\renamedfile.xml`) and probably in the crontab as well (`46 8 * * cd ~\downloaddir; test.sh` or maybe just `46 8 * * ~\downloaddir\test.sh`). Refer http://stackoverflow.com/questions/13204/why-doesnt-my-cron-job-work-properly and http://www.unix.com/shell-programming-scripting/129329-crontab-working-directory.html – Luke Sep 25 '12 at 11:09
  • another option is to harcode the paths into the script, eg. `wget -O ~/downloaddir/${line/,*} ${line/*,} >/dev/null 2>&1` and `done < ~/downloaddir/filelist` – Luke Sep 25 '12 at 11:12