Here’s a quick note on how to download the contents of an http directory via command line.
Using lftp to download an HTTP directory
Forget wget, lftp is what you covet!
What is lftp?
LFTP is a sophisticated ftp/http client, and a file transfer program supporting a number of network protocols. Like BASH, it has job control and uses the readline library for input. It has bookmarks, a built-in mirror command, and can transfer several files in parallel. It was designed with reliability in mind.
How do I use it to download the contents of an http directory?
sytax example #1:
$ lftp lftp :~> mirror http: //example.org/somedirectory
sytax example #2:
$ lftp lftp :~> mirror http://cans.cdm.depaul.edu/~badawi/courses/TDC368/codes/HW5/ Total: 1 directory, 6 files, 0 symlinks New: 6 files, 0 symlinks 62538 bytes transferred lftp :~> quit