|Samer Afach 17f29a4ac4|
This program simply takes a list of files as argument and a directory to download the files, and it downloads them sequentially, or in parallel. The program gives the option to load the whole list, or do parts of the list at a time. An option also is provided for how many threads/processes to be used.
(Installation was prepared for and tested with Debian Jessie.)
You can install the package with pip using
# pip install listdownloader
OR you can use the scripts that are provided to do that (
run_install), which are available in the repository.
OR you can create the source installation package yourself using
python3 setup.py sdist
and then use pip to install the package that will be built in the directory
# pip3 install listdownloader-x.y.z.tar.gz
where x.y.z is the current version of the program.
The program installs the package listdownloader and a script file for usage.
Running the script and using the package
The script can be executed (globally) using:
$ downloadlist.py -f file.txt -d destination -t threads -l lines
file.txt is the file name/path with the list of URLs to be downloaded (line by line)
destination is the path, to which the files should be downloaded
threads is the number of processes to be used to download the URLs simultaneously
lines is the number of lines to read from the files and read simultaneously. 0 leads to reading the whole file.
You may use the package in your own scripts by importing it:
then you can download a list of files using:
listdownloader.download_files(URLs, destination, num_threads)
URLs is a list of the URLs to be downloaded
destination is a string with the path, at which the files have to be saved
num_threads is the number of threads/processes to use for the download.
You can also download a single file using the function:
This script was written by Samer Afach for test purposes.