ListDownloader

About

This program simply takes a list of files as argument and a directory to download the files, and it downloads them sequentially, or in parallel. The program gives the option to load the whole list, or do parts of the list at a time. An option also is provided for how many threads/processes to be used.

Installation

(Installation was prepared for and tested with Debian Jessie.)

You can install the package with pip using

# pip install listdownloader

OR you can use the scripts that are provided to do that (run_build, and run_install), which are available in the repository.

OR you can create the source installation package yourself using

python3 setup.py sdist

and then use pip to install the package that will be built in the directory dist:

# pip3 install listdownloader-x.y.z.tar.gz

where x.y.z is the current version of the program.

The program installs the package listdownloader and a script file for usage.

Running the script and using the package

The script can be executed (globally) using:

$ downloadlist.py -f file.txt -d destination -t threads -l lines

where:

file.txt is the file name/path with the list of URLs to be downloaded (line by line)

destination is the path, to which the files should be downloaded

threads is the number of processes to be used to download the URLs simultaneously

lines is the number of lines to read from the files and read simultaneously. 0 leads to reading the whole file.

You may use the package in your own scripts by importing it:

import listdownloader

then you can download a list of files using:

listdownloader.download_files(URLs, destination, num_threads)

where:

URLs is a list of the URLs to be downloaded

destination is a string with the path, at which the files have to be saved

num_threads is the number of threads/processes to use for the download.

You can also download a single file using the function:

listdownloader.download_file(URL, destination)

License

MPL

About

This script was written by Samer Afach for test purposes.